Tóm tắt công việc
Define and implement quality assurance strategies for AI agents built on our bbv AI Hub product, covering functional correctness, robustness, reliability, safety, and regression behavior.
Validate complete agent workflows, from user input and tool calls to integrations and final responses.
Design and maintain versioned, traceable test datasets that turn business requirements and real user scenarios into test cases, including edge cases, multi-step and tool-use tasks, and safety scenarios.
Use Langfuse to benchmark agent versions, prompts, models, and configurations against standardized datasets, tracking metrics such as task success, tool-call correctness, hallucination rate, latency, and cost.
Design evaluation rubrics and LLM-as-a-judge prompts, combining deterministic, LLM-based, and human evaluation, and validate the reliability of automated evaluators.
Establish baselines, quality thresholds, and regression criteria for agent releases, and support automated quality gates in CI/CD pipelines.
Analyze failed agent traces to find root causes and turn production failures and user feedback into permanent regression test cases.
Work closely with
AI Engineers,
Software Engineers, and
Product Owners to define measurable acceptance criteria, resolve issues, and verify fixes.
Report evaluation results clearly and contribute to AI agent quality standards on the bbv AI Hub.
Competitive salary with full insurance coverage (medical, social, unemployment) based on net salary.
13th-month salary (paid semi-annually, with partial payment made mid-year rather than waiting until year-end).
Performance review twice a year with bonus opportunities.
Annual loyalty bonus (50% of monthly net salary from the 2nd year).
Premium health insurance and annual health check-up.
14 days of leave, plus additional leave based on tenure.
Daily perks: lunch allowance, free snacks/coffee, and sports activities.
Learning & development support: training programs and certification sponsorship.
English-speaking environment with tuition support.
Team activities and events: company trips, team building, and holiday celebrations.
Opportunity to work with international teams and join internal tech communities.
Several years of experience in Software Quality Assurance, Test Engineering, or Quality Engineering.
Strong understanding of testing methodologies and test design techniques, with experience designing and maintaining structured test datasets.
Experience with API and/or web application testing, automated and regression testing, and Git-based collaborative workflows.
Strong analytical skills and the ability to trace complex failures to their root cause.
Good understanding of AI/LLM concepts and their implications for software quality.
Experience testing LLM-based applications, AI agents, or LLM evaluation is a strong advantage.
Experience with some of the following is a plus: Langfuse or similar evaluation platforms, LLM-as-a-judge, LangChain/LangGraph, Python or TypeScript, CI/CD quality gates, prompt engineering, and hallucination/adversarial testing.
Ability to communicate clearly with both technical and non-technical stakeholders.
Good English communication skills (both speaking and writing).