evaluation specialist

Kore.ai

<div>Kore.ai is a pioneering force in enterprise AI transformation, empowering organizations through our comprehensive agentic AI platform. With innovative offerings across &quot;AI for Service,&quot; &quot;AI for Work,&quot; and &quot;AI for Process,&quot; we're enabling over 400+ Global 2000 companies to fundamentally reimagine their operations, customer experiences and employee productivity.
Our end-to-end platform enables enterprises to build, deploy, manage, monitor, and continuously improve agentic applications at scale. We've automated over 1 billion interactions every year with voice and digital AI in customer service, and transformed employee experiences for tens of thousands of employees through productivity and AI-driven workflow automation.
Recognized as a leader by Gartner, Forrester, IDC, ISG, and Everest, Kore.ai has secured Series D funding of $150M, including strategic investment from NVIDIA to drive Enterprise AI innovation. Founded in 2014 and headquartered in Florida, we maintain a global presence with offices in India, UK, Germany, Korea, and Japan.
You can find full press coverage at https://kore.ai/press/ 
POSITION: Senior AI Evaluation Specialist 
POSITION SUMMARY: We are seeking a Senior AI Evaluation Specialist to design and execute robust evaluation methodologies for Generative and Agentic AI systems. This role bridges AI product quality, evaluation science, and responsible AI governance &mdash; ensuring every AI feature, agent, and model release is measured, benchmarked, and validated using standardised frameworks.
The ideal candidate combines a QA mindset, ML evaluation rigour, and hands-on coding expertise to benchmark LLMs, multi-agent workflows, and GenAI APIs, driving consistent, measurable, and safe AI product performance. 
LOCATION: Hyderabad (Work from Office) 
RESPONSIBILITIES: 
1. AI Evaluation &amp; Benchmarking
Build and maintain end-to-end evaluation pipelines for Generative and Agentic AI features (e.g., chat, reasoning agents, RAG workflows, summarization, classification).
Implement standardized evaluation frameworks such as RAGAS, G-Eval, HELM, PromptBench, MT-Bench, or custom evaluation harnesses.
Define and measure core AI quality metrics &mdash; accuracy, groundedness, coherence, contextual recall, hallucination rate, and response time.
Create reproducible benchmarks, leaderboards, and regression tracking for models and agents across multiple releases or providers (OpenAI, Anthropic, Mistral, etc.). 
2. Agentic AI Evaluation
Evaluate multi-agent systems and autonomous AI workflows, measuring task success rates, reasoning trace quality, and tool-use efficiency.
Assess Agentic AI behaviors such as planning accuracy, goal completion rate, context handoff success, and inter-agent communication reliability.
Validate decision-making transparency and error recovery mechanisms in autonomous agent frameworks (LangGraph, AutoGen, CrewAI, etc.).
Design agent-specific evaluation scenarios &mdash; simulated environments, user-in-the-loop testing, and &ldquo;mission-based&rdquo; performance scoring. 
3. Experimentation &amp; Automation
Develop Python-based evaluation scripts to automate testing using OpenAI, Anthropic, and Hugging Face APIs.
Conduct large-scale comparative studies across prompts, models, and fine-tuned variants, analyzing quantitative and qualitative differences.
Integrate evaluations into CI/CD pipelines to enable continuous AI quality monitoring.
Visualize results using dashboards (Plotly, Streamlit, Dash, or Grafana). 
4. Quality Governance &amp; Reporting
Define and enforce AI acceptance thresholds before deployment.
Collaborate with Responsible AI teams to evaluate bias, fairness, safety, and privacy implications.
Produce detailed evaluation reports and audit logs for model releases and governance boards.
Present findings to Product, Data Science, and Executive stakeholders &mdash; transforming metrics into actionable insights. 
5. Collaboration &amp; Continuous Improvement
Work closely with Prompt Engineers, ML Scientists, and QA Engineers to close the loop between testing and improvement.
Support Product teams in defining evaluation-driven release criteria.
Mentor junior evaluators in AI testing methodologies, benchmarking, and analysis.
Keep abreast of advances in LLM evaluation research, Agentic AI frameworks, and tool-calling reliability testing. 
QUALIFICATIONS / SKILLS REQUIRED:<div><div><div> 
Category 
Expected Expertise</div><div> 
Programming 
Python (Pandas, NumPy, LangChain, LangGraph, OpenAI/Anthropic SDKs)</div><div> 
Evaluation Frameworks 
RAGAS, HELM, G-Eval, MT-Bench, PromptBench, custom scoring pipelines</div><div> 
GenAI APIs 
OpenAI GPT-4/5, Claude, Gemini, Mistral, Azure OpenAI</div><div> 
Agentic AI 
Understanding of multi-agent orchestration, tool use, reasoning traces, and planning frameworks (AutoGen, CrewAI, LangGraph)</div><div> 
Metrics Knowledge 
BLEU, ROUGE, cosine similarity, factuality, coherence, bias, toxicity, reasoning success rate</div><div> 
Data &amp; Analytics 
JSON parsing, prompt dataset curation, result visualization</div><div> 
Tooling 
Git, Jupyter/Colab, Jira, Confluence, evaluation dashboards</div><div> 
Soft Skills 
Analytical communication, documentation excellence, cross-team collaboration</div></div></div> 
EDUCATION QUALIFICATION:<ul><li>
Bachelor&rsquo;s or Master&rsquo;s degree in Computer Science, AI, Data Science, or related discipline.</li><li>
5 to 10 years total experience with at least 3+ years in AI evaluation, GenAI QA, or LLM quality analysis.</li><li>
Strong understanding of AI/ML model lifecycle, prompt engineering, and RAG or agentic architectures.</li><li>
Experience contributing to AI safety, reliability, or responsible AI initiatives.</li></ul></div>

AI Evaluation Specialist - QA

Taqat Trading and Business Solutions

Follow-Up and Evaluation SpecialistJob SummaryThe Follow-Up and Evaluation Specialist is responsible for monitoring activities, tracking progress, and evaluating outcomes to ensure programs, projects, or services meet established goals and standards. The role focuses on data collection, analysis, reporting, and continuous improvement through effective follow-up mechanisms.Key Responsibilities<ul><li>Monitor ongoing activities, projects, or cases to ensure alignment with plans, policies, and performance indicators</li><li>Conduct regular follow-ups with stakeholders, teams, or beneficiaries to track progress and address gaps</li><li>Design and implement evaluation tools such as surveys, checklists, and performance metrics</li><li>Collect, verify, and analyze qualitative and quantitative data</li><li>Prepare periodic evaluation, follow-up, and performance reports</li><li>Identify challenges, risks, and areas for improvement, and recommend corrective actions</li><li>Ensure proper documentation and record-keeping of follow-up and evaluation activities</li><li>Coordinate with internal teams and external partners to support evaluation processes</li><li>Track action plans and ensure timely implementation of recommendations</li><li>Support decision-making by providing accurate and timely evaluation insights</li></ul>Required Qualifications<ul><li>Bachelor&rsquo;s degree in Business Administration, Social Sciences, Public Administration, Monitoring &amp; Evaluation, or a related field</li><li>Previous experience in follow-up, monitoring, evaluation, quality assurance, or a related role</li><li>Strong analytical and reporting skills</li><li>Basic accounting knowledge</li><li>Proficiency in data collection and analysis tools (e.g., Excel, databases, or evaluation software)</li><li>Good organizational and time-management skills</li></ul>Required Skills and Competencies<ul><li>Attention to detail and accuracy</li><li>Strong communication and interpersonal skills</li><li>Ability to analyze data and interpret results</li><li>Problem-solving and critical-thinking skills</li><li>Ability to work independently and as part of a team</li><li>Ethical handling of data and confidentiality</li></ul>Job Type: Full-timeLanguage:<ul><li>Arabic Native (Required)</li></ul>Location:<ul><li>Doha (Required)</li></ul>

Follow up and evaluation specialist

YO IT CONSULTING

<div><div><div><h1 class="jobSectionHeader">Job Description: AI Task Evaluation &amp; Statistical Analysis Specialist</h1>
<h2 class="jobSectionHeader">Role Overview</h2>
We're seeking a data-driven analyst to conduct comprehensive failure analysis on AI agent performance across finance-sector tasks. You'll identify patterns, root causes, and systemic issues in our evaluation framework by analyzing task performance across multiple dimensions (task types, file types, criteria, etc.).
<h2 class="jobSectionHeader">Key Responsibilities</h2>
<ul><li>Statistical Failure Analysis: Identify patterns in AI agent failures across task components (prompts, rubrics, templates, file types, tags)
</li><li>Root Cause Analysis: Determine whether failures stem from task design, rubric clarity, file complexity, or agent limitations
</li><li>Dimension Analysis: Analyze performance variations across finance sub-domains, file types, and task categories
</li><li>Reporting &amp; Visualization: Create dashboards and reports highlighting failure clusters, edge cases, and improvement opportunities
</li><li>Quality Framework: Recommend improvements to task design, rubric structure, and evaluation criteria based on statistical findings
</li><li>Stakeholder Communication: Present insights to data labeling experts and technical teams
</li></ul><h2 class="jobSectionHeader">Required Qualifications</h2>
<ul><li>Statistical Expertise: Strong foundation in statistical analysis, hypothesis testing, and pattern recognition
</li><li>Programming: Proficiency in Python (pandas, scipy, matplotlib/seaborn) or R for data analysis
</li><li>Data Analysis: Experience with exploratory data analysis and creating actionable insights from complex datasets
</li><li>AI/ML Familiarity: Understanding of LLM evaluation methods and quality metrics
</li><li>Tools: Comfortable working with Excel, data visualization tools (Tableau/Looker), and SQL
</li></ul><h2 class="jobSectionHeader">Preferred Qualifications</h2>
<ul><li>Experience with AI/ML model evaluation or quality assurance
</li><li>Background in finance or willingness to learn finance domain concepts
</li><li>Experience with multi-dimensional failure analysis
</li><li>Familiarity with benchmark datasets and evaluation frameworks
</li><li>2-4 years of relevant experience
</li></ul>We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.
</div></div><div><div><h2 class="jobSectionHeader">Contract and Payment Terms</h2>
<ul><li>You will be engaged as an independent contractor.</li>
<li>This is a fully remote role that can be completed on your own schedule.</li>
<li>Projects can be extended, shortened, or concluded early depending on needs and performance.</li>
<li>Your work will not involve access to confidential or proprietary information from any employer, client, or institution.</li>
<li>Payments are weekly on Stripe or Wise based on services rendered.</li></ul></div></div></div>

Data Scientist

State of Idaho

<div>Description
Performance Evaluator, Full Time
The Office of Performance Evaluations (OPE) seeks a qualified candidate for the full-time position of performance evaluator. The office is an independent office of the Idaho Legislature, assisting the bipartisan Joint Legislative Oversight Committee in promoting confidence and accountability in state government through rigorous, independent research. The office has been nationally recognized for its influential work.
OPE is a small, eight-person office that serves the Idaho Legislature and the people of Idaho by conducting independent, nonpartisan evaluations that consider a wide range of stakeholder perspectives and are often complex and politically sensitive. The office usually conducts three to five in-depth evaluations per year. These evaluations include elements of performance auditing, program evaluation, policy analysis, and budget analysis.
Working at OPE will allow you to interact with and contribute to a broad range of public services. Our office has done work in the fields of economic development, education (K-12 and higher education), general government, health and human services, natural resources, transportation, and public safety. For more information about the office, see http://legislature.idaho.gov/ope/.
Duties include conducting research, collecting data, administering surveys, conducting interviews, performing qualitative and quantitative analyses, writing reports, and making presentations to legislative committees and other stakeholder groups.
Salary: $28&ndash;$31 per hour depending on experience and education, plus competitive benefits.
Education: A graduate degree is preferred, but relevant experience in public or business administration, social science research, program evaluation, or a related field can be substituted.
Qualifications:
Thorough understanding of research methods, including qualitative and quantitative analyses.
Ability to learn quickly, synthesize information from multiple sources, generate insights, and communicate clearly and coherently by distilling complex ideas into concise and actionable formats.
Proactive, creative, flexible, and service-oriented approach to problem solving.
Demonstrated critical thinking skills.
Advanced ability to communicate complex concepts to policymakers and other stakeholders through writing and data visualization.
Well-developed interpersonal and teamwork skills.
Ability to work independently on multiple projects simultaneously.
Capacity to identify and manage project risks and meet deadlines.
General knowledge of state government.
Ability to conduct objective, nonpartisan work and to help the office maintain nonpartisanship in both appearance and practice. 
 
To apply: Submit a resume, a cover letter, and the names and contact information of three professional references. 
For questions, contact:
Leslie Baker, Administrative Coordinator
Office of Performance Evaluations
Idaho Legislature PO Box 83720 Boise, ID 83720-0055
lbaker@ope.idaho.gov 
Application deadline: Friday, January 23, 2026.
Special Notification: This position is exempt from classified state service and the rules of the Division of Human Resources and the Idaho Personnel Commission.</div>

PERFORMANCE EVALUATOR

Action Against Hunger

<div><div><div><div>Posted date</div> <div>26th December, 2025</div> <div>Last date to apply</div> <div>4th January, 2026</div>
</div> 
<div><div>Country</div> <div>Pakistan</div> <div>Locations</div> <div>Bahria Enclave - Islamabad</div>
</div> 
<div><div>Category</div> <div>Development Sector
</div></div> 
<div><div>Salary</div> <div>134,169</div>
</div> 
<div><div>Type</div> <div>Full Time</div> <div>Position</div> <div>1</div></div></div></div><div></div>
 <div>Position Objective: Design, implement, and maintain MEAL systems that ensure accountability, quality, and learning across the project and partner activities. Provide evidence-based inputs for adaptive management and donor reporting while strengthening partner MEAL capacity.

Relevant Experience:
<ul>
<li>Master&rsquo;s in Social Sciences, Statistics, Monitoring &amp; Evaluation, Development Studies, or related field</li>
<li>Minimum 4 years of overall experience and 2 years of MEAL experience in humanitarian contexts.</li>
</ul>
 Job Description:

Mission 1 MEAL Design and Planning
<ul>
<li>Under supervision of Program Manager, develop and roll-out of the project MEAL plan, including IPTT, data collection tools, QA protocols, and sampling strategies.</li>
<li>Support the development of SMART indicators, aligned with proposal requirements, and integrated into DIPs and partner workplans.</li>
<li>Support in establishment baseline, midline, and endline methodologies where applicable.</li>
</ul>
Mission 2 Monitoring and Data Quality Assurance
<ul>
<li>Oversee routine monitoring activities, including field visits, spot checks, and data verification exercises.</li>
<li>Implement data quality assurance processes (DQAs) and ensure timely corrective actions.</li>
<li>Manage digital and paper-based data collection systems; ensure secure data storage and archiving.</li>
</ul>
Mission 3 Accountability and Feedback Mechanisms
<ul>
<li>Establish and maintain beneficiary feedback and complaints mechanisms in line with CHS and ACF policies.</li>
<li>Ensure timely recording, investigation, and resolution of complaints and feedback.</li>
<li>Promote community engagement and ensure accountability to affected populations.</li>
</ul>
Mission 4 Analysis, Reporting, and Learning
<ul>
<li>Analyze monitoring data and produce weekly, monthly, and quarterly monitoring reports and dashboards for PM and donors.</li>
<li>Document lessons learned, good practices, and case studies; facilitate learning workshops with partners.</li>
<li>Support MEAL inputs into donor reports, evaluations, and external communications.</li>
</ul>
Mission 5 Capacity Strengthening and Coordination
<ul>
<li>Train and mentor partner MEAL staff on tools, data collection, analysis, and reporting.</li>
<li>Coordinate MEAL activities with clusters, government monitoring bodies, and other stakeholders.</li>
<li> Support third-party monitoring and evaluations as required.</li>
</ul>
Mission 6: Ensure compliance with ACFs gender equality, child protection sensitivity and Safeguarding policies 
<ul><li>Read and ensure a full understanding of ACFs Gender, Child Protection and other safeguarding policies</li>
<li>Implement all work related activities with a gender, inclusion and child protection lens</li>
<li>Help build an inclusive office environment where people do not feel disadvantaged or judged according to their gender or limited due to manageable disability</li>
</ul>
This Job Description may evolve according to the needs of the mission and in view the ever changing scope of your designation.

Action against Hunger is committed to the prevention of risks related to abuse, sexual exploitation and the promotion of gender equality and its diversity. Action Against Hunger is an employer that does not tolerate Sexual Exploitation and Abuse. All potential candidates will be subjected to rigorous background checks and controls. The Pakistan Mission encourages the application of the women / men / transgender/ disabled workers. Action Against Hunger's recruitment process is free of charge, no compensation of any form shall be requested from a candidate. It reflects the major policies and its charter through its professionalism and impartiality. If you observe behavior in opposition to these values during this process, please report it to the appropriate teams. All persons hired by Action Against Hunger will also be checked for compliance with the &quot;anti-terrorist&quot; lists, while respecting the rights of employees and the confidentiality of personal data.

In line with our commitment to diversity, equity and inclusion, we particularly welcome applications from qualified female candidates, as we strive to build more gender-balanced teams. All applications will, however, be assessed based on merit and in accordance with our equal opportunity principles</div>

MEAL Officer – Localization & Emergency Response

Search 6390+ Evaluation specialist jobs near you this January. New roles added regularly across leading industries. Apply today.

6390 evaluation specialist jobs NEAR_ME

AI Evaluation Specialist - QA

Follow up and evaluation specialist

Data Scientist

Data Scientist

Data Scientist

PERFORMANCE EVALUATOR

MEAL Officer – Localization & Emergency Response