AI QA Engineer – Senior

Work Experience4+ Years

Work TimingsGeneral Shift

About the RoleTraditional QA doesn’t work for AI.We’re looking for a Senior AI QA Engineer who understands that AI systems are probabilistic, non-deterministic, and failure-prone by nature — and knows how to test them anyway.You will own AI quality, safety, reliability, and regression testing across agentic systems and AI-powered SaaS products.What You’ll Be DoingAI-Specific Testing & Validation

• Design and execute AI-specific test strategies, including:
• Prompt robustness testing
• Hallucination detection
• Output consistency checks
• Edge-case and adversarial testing
• Validate RAG pipelines:
• Retrieval accuracy
• Context relevance
• Response grounding
• Test multi-agent workflows and tool integrations

Automation & Tooling

• Build automated AI test harnesses
• Create evaluation pipelines for AI responses
• Define regression tests for prompt and agent changes
• Integrate AI testing into CI/CD pipelines

Collaboration & Quality Ownership

• Work closely with AI developers during design, not just after implementation
• Help define acceptance criteria for AI features
• Monitor AI behavior in production and flag drift or degradation
• Document AI failure modes and mitigations

RequirementsMust Have

• 4+ years of QA or testing experience

• Strong API testing and automation skills

• Understanding of AI/LLM behavior and limitations

• Experience testing non-deterministic systems

• Strong analytical and problem-solving mindset

Nice to Have

• Experience testing AI agents or RAG systems

• Experience writing Python-based test automation

• Familiarity with LLM APIs and prompt engineering

• Experience with observability tools for AI systems

Job Types: Full-time, Permanent

Pay: Up to ₹1,000,000.00 per year

Benefits:

• Cell phone reimbursement
• Flexible schedule
• Health insurance
• Internet reimbursement
• Paid sick time
• Paid time off
• Provident Fund
• Work from home

Application Question(s):

• Total years of experience in QA / testing roles
• Total years of experience in test automation *
• Your experience with API testing and automation
• Hands-on experience testing AI or LLM-powered systems
• Your experience testing non-deterministic systems
• Programming languages used for test automation (mention primary language) *
• What is your current CTC? *
• What is your Expected CTC?
• How soon can you join? *
• If you are currently serving or have served your notice period, please mention your last working day (DD/MM/YYYY) *
• Reason for Change

Work Location: Remote