AI Evaluation Specialist - QA

Kore.ai is a pioneering force in enterprise AI transformation, empowering organizations through our comprehensive agentic AI platform. With innovative offerings across "AI for Service," "AI for Work," and "AI for Process," we're enabling over 400+ Global 2000 companies to fundamentally reimagine their operations, customer experiences and employee productivity.

Our end-to-end platform enables enterprises to build, deploy, manage, monitor, and continuously improve agentic applications at scale. We've automated over 1 billion interactions every year with voice and digital AI in customer service, and transformed employee experiences for tens of thousands of employees through productivity and AI-driven workflow automation.

Recognized as a leader by Gartner, Forrester, IDC, ISG, and Everest, Kore.ai has secured Series D funding of $150M, including strategic investment from NVIDIA to drive Enterprise AI innovation. Founded in 2014 and headquartered in Florida, we maintain a global presence with offices in India, UK, Germany, Korea, and Japan.

You can find full press coverage at https://kore.ai/press/

POSITION: Senior AI Evaluation Specialist

POSITION SUMMARY: We are seeking a Senior AI Evaluation Specialist to design and execute robust evaluation methodologies for Generative and Agentic AI systems. This role bridges AI product quality, evaluation science, and responsible AI governance — ensuring every AI feature, agent, and model release is measured, benchmarked, and validated using standardised frameworks.

The ideal candidate combines a QA mindset, ML evaluation rigour, and hands-on coding expertise to benchmark LLMs, multi-agent workflows, and GenAI APIs, driving consistent, measurable, and safe AI product performance.

LOCATION: Hyderabad (Work from Office)

RESPONSIBILITIES:

1. AI Evaluation & Benchmarking

Build and maintain end-to-end evaluation pipelines for Generative and Agentic AI features (e.g., chat, reasoning agents, RAG workflows, summarization, classification).

Implement standardized evaluation frameworks such as RAGAS, G-Eval, HELM, PromptBench, MT-Bench, or custom evaluation harnesses.

Define and measure core AI quality metrics — accuracy, groundedness, coherence, contextual recall, hallucination rate, and response time.

Create reproducible benchmarks, leaderboards, and regression tracking for models and agents across multiple releases or providers (OpenAI, Anthropic, Mistral, etc.).

2. Agentic AI Evaluation

Evaluate multi-agent systems and autonomous AI workflows, measuring task success rates, reasoning trace quality, and tool-use efficiency.

Assess Agentic AI behaviors such as planning accuracy, goal completion rate, context handoff success, and inter-agent communication reliability.

Validate decision-making transparency and error recovery mechanisms in autonomous agent frameworks (LangGraph, AutoGen, CrewAI, etc.).

Design agent-specific evaluation scenarios — simulated environments, user-in-the-loop testing, and “mission-based” performance scoring.

3. Experimentation & Automation

Develop Python-based evaluation scripts to automate testing using OpenAI, Anthropic, and Hugging Face APIs.

Conduct large-scale comparative studies across prompts, models, and fine-tuned variants, analyzing quantitative and qualitative differences.

Integrate evaluations into CI/CD pipelines to enable continuous AI quality monitoring.

Visualize results using dashboards (Plotly, Streamlit, Dash, or Grafana).

4. Quality Governance & Reporting

Define and enforce AI acceptance thresholds before deployment.

Collaborate with Responsible AI teams to evaluate bias, fairness, safety, and privacy implications.

Produce detailed evaluation reports and audit logs for model releases and governance boards.

Present findings to Product, Data Science, and Executive stakeholders — transforming metrics into actionable insights.

5. Collaboration & Continuous Improvement

Work closely with Prompt Engineers, ML Scientists, and QA Engineers to close the loop between testing and improvement.

Support Product teams in defining evaluation-driven release criteria.

Mentor junior evaluators in AI testing methodologies, benchmarking, and analysis.

Keep abreast of advances in LLM evaluation research, Agentic AI frameworks, and tool-calling reliability testing.

QUALIFICATIONS / SKILLS REQUIRED:

EDUCATION QUALIFICATION:

• Bachelor’s or Master’s degree in Computer Science, AI, Data Science, or related discipline.
• 5 to 10 years total experience with at least 3+ years in AI evaluation, GenAI QA, or LLM quality analysis.
• Strong understanding of AI/ML model lifecycle, prompt engineering, and RAG or agentic architectures.
• Experience contributing to AI safety, reliability, or responsible AI initiatives.