QC Engineer - AI Conversation

1. Conversational AI Test Strategy
Develop test strategies for conversation flows, intents, and entities of the chatbot/voicebot.
Design test case sets covering a wide range of conversational scenarios: happy path, edge cases, multi-turn conversations, flow interruptions, and context switching.
Establish evaluation criteria for AI response quality (accuracy, relevance, tone, naturalness).
2. Functional & Response Quality Testing
Test the accuracy of NLU (Natural Language Understanding): intent recognition and entity extraction.
Evaluate the quality of AI responses: correctness, coherence, contextual relevance, and avoidance of repetition or rambling.
Test the system's ability to handle ambiguous cases, out-of-scope questions, and multiple languages (if applicable).
Test the integration between the AI conversation engine and backend systems (API, CRM, database).
3. AI Safety & Ethics Testing
Assess the risk of the AI producing incorrect, biased, or inappropriate (harmful/toxic) content.
Test the AI's ability to appropriately decline sensitive or unsafe requests.
Work with the AI/ML team to report and improve cases of AI hallucination.
4. Automated Testing & Large-Scale Evaluation
Build automated test suites for conversation flows.
Design and operate large-scale response quality evaluation processes (batch evaluation, sampling review).
Collaborate on building a golden dataset/benchmark to measure quality over time.
5. Defect Management & Reporting
Log and classify conversation defects (incorrect intent, incorrect context, inappropriate responses, safety issues).
Collaborate with AI/ML Engineers, Prompt Engineers, and Conversation Designers to improve quality.
Provide periodic reports on AI system quality (accuracy rate, CSAT, escalation rate to human agents).
6. Process Improvement
Propose conversational AI quality evaluation processes tailored to the product's specific needs.
Train and mentor junior/middle QC staff on AI conversation testing methods.