Senior QA Engineer (Internal AI Products)

We are seeking an experienced Senior QA Engineer to ensure the quality, reliability, and performance of AI products used by Virtuos employees. These may include AI assistants, knowledge search, content generation tools, and automated workflows that support creative, development, and business teams.
You will lead testing from early prototypes through release and ongoing use. Working with R&D engineers, product managers, and internal users, you will assess both software behavior and AI output quality. You will build automated checks, investigate user issues, and mentor junior QA engineers.
• Define test strategies, plans, and release criteria for internal AI products. Prioritize coverage based on user impact, data sensitivity, and product risk.
• Test complete user workflows across web, cloud, and desktop applications, as applicable. Verify APIs, integrations, access permissions, and failure recovery.
• Build and maintain evaluation datasets from approved internal use cases. Agree clear scoring criteria with product owners and subject experts for accuracy, relevance, completeness, and task success.
• Evaluate AI outputs through repeatable tests and human review. Cover incorrect or unsupported answers, inconsistent results, ambiguous requests, and unusual inputs.
• For AI knowledge search, verify source accuracy, answer references, and access restrictions. For AI agents, test tool use, approval steps, and protection against unintended actions.
• Test safeguards for confidential company and client information. Cover unauthorized access, data leakage, and attempts to make the AI ignore instructions or reveal restricted content.
• Build automated functional, API, regression, and AI evaluation tests. Add reliable checks to continuous integration and delivery pipelines and compare results after model, prompt, data, or code changes.
• Assess response times, concurrent usage, service limits, and cost per task. Verify timeouts, retries, fallback behavior, and clear error messages when AI services fail.
• Coordinate pilot testing and user acceptance testing with internal teams. Turn employee feedback and production issues into test cases and improvements.
• Investigate defects with engineers. Capture relevant inputs, model and prompt versions, logs, and steps to reproduce issues while protecting sensitive data.
• Report test coverage, AI quality trends, and release risks. Recommend release readiness, verify fixes, and track recurring quality issues after launch.
• Mentor junior QA engineers and improve shared testing methods, documentation, and automation practices.