conversational ai analyst

Mercor

<div>
 
 About The Job
 
 
 
 
 Mercor
 
 connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include
 
 Benchmark
 
 ,
 
 General Catalyst
 
 ,
 
 Peter Thiel
 
 ,
 
 Adam D'Angelo
 
 ,
 
 Larry Summers
 
 , and
 
 Jack Dorsey
 
 .
 
 
 
 Position:
 
 Language Model Evaluator
 
 
 
 Type:
 
 
 Full-time or Part-time Contract Work
 
 
 
 
 Compensation:
 
 
 $23/hour
 
 
 
 
 Location:
 
 
 Geography restricted to Egypt, Saudi Arabia, UAE, USA
 
 
 
 
 Role Responsibilities
 
 
 
 <ul>
 <li>
 Evaluate LLM-generated responses on their ability to effectively answer user queries.
 </li>
 <li>
 Conduct fact-checking using trusted public sources and external tools.
 </li>
 <li>
 Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies.
 </li>
 <li>
 Assess reasoning quality, clarity, tone, and completeness of responses.
 </li>
 <li>
 Ensure model responses align with expected conversational behavior and system guidelines.
 </li>
 <li>
 Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines.
 
 
 
 </li>
 </ul>
 
 Qualifications
 
 
 
 
 Must-Have
 
 
 
 <ul>
 <li>
 Bachelor&rsquo;s degree
 </li>
 <li>
 Native speaker or ILR 5/primary fluency (C2 on the CEFR scale) in Arabic
 </li>
 <li>
 Significant experience using large language models (LLMs)
 </li>
 <li>
 Excellent writing skills
 </li>
 <li>
 Strong attention to detail
 </li>
 <li>
 Adaptable and comfortable moving across topics, domains, and customer requirements
 </li>
 <li>
 Background or experience in domains requiring structured analytical thinking
 </li>
 <li>
 Excellent college-level mathematics skills
 
 
 
 </li>
 </ul>
 
 Preferred
 
 
 
 <ul>
 <li>
 Prior experience with RLHF, model evaluation, or data annotation work
 </li>
 <li>
 Experience writing or editing high-quality written content
 </li>
 <li>
 Experience comparing multiple outputs and making fine-grained qualitative judgments
 </li>
 <li>
 Familiarity with evaluation rubrics, benchmarks, or quality scoring systems
 
 
 
 </li>
 </ul>
 
 Application Process (Takes 20&ndash;30 mins to complete)
 
 
 
 <ul>
 <li>
 Upload resume
 </li>
 <li>
 AI interview based on your resume
 </li>
 <li>
 Submit form
 
 
 
 </li>
 </ul>
 
 Resources &amp; Support
 
 
 
 <ul>
 <li>
 For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome/welcome
 </li>
 <li>
 For any help or support, reach out to: support@mercor.com
 
 
 
 </li>
 </ul>
 
 PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
 
</div>

Conversational AI Analyst - Fully Remote

MERCOR

<div>About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: Language Model Evaluator 
Type: Full-time or Part-time Contract Work 
Compensation: $23/hour 
Location: Geography restricted to Egypt, Saudi Arabia, UAE, USA
Role Responsibilities<ul><li>
Evaluate LLM-generated responses on their ability to effectively answer user queries.</li><li>
Conduct fact-checking using trusted public sources and external tools.</li><li>
Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies.</li><li>
Assess reasoning quality, clarity, tone, and completeness of responses.</li><li>
Ensure model responses align with expected conversational behavior and system guidelines.</li><li>
Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines.</li></ul>
Qualifications
Must-Have<ul><li>
Bachelor&rsquo;s degree</li><li>
Native speaker or ILR 5/primary fluency (C2 on the CEFR scale) in Arabic</li><li>
Significant experience using large language models (LLMs)</li><li>
Excellent writing skills</li><li>
Strong attention to detail</li><li>
Adaptable and comfortable moving across topics, domains, and customer requirements</li><li>
Background or experience in domains requiring structured analytical thinking</li><li>
Excellent college-level mathematics skills</li></ul>
Preferred<ul><li>
Prior experience with RLHF, model evaluation, or data annotation work</li><li>
Experience writing or editing high-quality written content</li><li>
Experience comparing multiple outputs and making fine-grained qualitative judgments</li><li>
Familiarity with evaluation rubrics, benchmarks, or quality scoring systems</li></ul>
Application Process (Takes 20&ndash;30 mins to complete)<ul><li>
Upload resume</li><li>
AI interview based on your resume</li><li>
Submit form</li></ul>
Resources &amp; Support<ul><li>
For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome/welcome</li><li>
For any help or support, reach out to: support@mercor.com</li></ul>
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.</div>

First Service Credit Union

<div>Role:

The Business Systems Analyst serves as the organization's subject matter expert for AI-driven interaction platforms, including the Kore.ai virtual voice and digital agent and other automation technologies. This role is responsible for understanding how automated conversations behave in real operational scenarios and ensuring the systems respond accurately, consistently, and in alignment with business intent. The analyst develops deep functional knowledge of conversational flows, intents, training data, and configuration, and translates operational needs into clear requirements for implementation. The role works closely with operations and technology teams to refine automation, improve customer experience, and prevent inconsistent or unintended behavior. This position supports business decision-making but does not establish business policy or product strategy.

Essential Functions &amp; Responsibilities:
<ul><li>Maintain expert knowledge of conversational AI platforms, including Kore.ai, and related automation technologies.</li></ul>
<ul><li>Monitor and analyze automated interactions to identify misunderstandings, logic gaps, or incorrect responses.</li></ul>
<ul><li>Tune conversational behavior to improve intent recognition accuracy and customer experience.</li></ul>
<ul><li>Ensure consistent automated responses across channels (voice, chat, digital).</li></ul>
<ul><li>Maintain documentation of conversational logic, decision paths, and behavior standards.</li></ul>
<ul><li>Track performance metrics including containment rate, fallback rate, and escalation frequency.</li></ul>
<ul><li>Improve automation accuracy through structured tuning and refinement of logic.</li></ul>
<ul><li>Translate operational scenarios into clear conversational and functional requirements.</li></ul>
<ul><li>Define expected system behavior for new intents, enhancements, or workflow changes.</li></ul>
<ul><li>Provide detailed use cases and acceptance criteria for implementation.</li></ul>
<ul><li>Validate that automation responses match business expectations.</li></ul>
<ul><li>Support rollout and testing of new automation capabilities.</li></ul>
<ul><li>Investigate incorrect or failed automated interactions.</li></ul>
<ul><li>Determine root cause (training data, configuration, workflow logic, or defect).</li></ul>
<ul><li>Coordinate resolution between operations and technical teams.</li></ul>
<ul><li>Guide business teams on automation expectations and system limitations.</li></ul>
<ul><li>Configure approved changes where applicable.</li></ul>
<ul><li>Support conversational testing scenarios and validation cycles.</li></ul>
<ul><li>Evaluate downstream operational impact prior to deployment.</li></ul>
<ul><li>Maintain change history and behavior documentation.</li></ul>
<ul><li>Other job duties as assigned.
</li></ul>Performance Measurements:

1. Accuracy &amp; Automation Quality Goals
<ul><li>Intent Recognition Accuracy &ndash; Achieve and maintain target accuracy rates (e.g., 90&ndash;95%).</li></ul>
<ul><li>Reduction in Unhandled Intents and Fallback Rates.</li></ul>
<ul><li>Improvement in Escalation Accuracy and Proper Routing.</li></ul>
<ul><li>Response Consistency Across Channels.</li></ul>

2. Customer Experience Goals
<ul><li>Increased Containment Rate while maintaining satisfaction levels.</li></ul>
<ul><li>Reduced Member Effort and Confusion.</li></ul>
<ul><li>Improved Conversational Flow and Natural Dialogue Quality.</li></ul>
<ul><li>Positive Member Sentiment or CSAT Feedback on Automated Interactions.</li></ul>

3. Operational Effectiveness Goals
<ul><li>Reduction in automation-related complaints or errors.</li></ul>
<ul><li>Measurable reduction in operational rework due to incorrect automation.</li></ul>
<ul><li>Timely implementation of enhancements and updates.</li></ul>
<ul><li>Effective collaboration with operations, IT, and compliance teams.</li></ul>

4. Continuous Improvement &amp; Optimization
<ul><li>Regular review and tuning of conversational performance.</li></ul>
<ul><li>Documented improvement cycles and performance tracking.</li></ul>
<ul><li>Identification of automation opportunities that improve operational efficiency.</li></ul>

Knowledge and Skills:

Experience: One to three years of similar or related experience in banking, contact center operations, automation platforms, or customer service workflow environments. 

Education: (1) A bachelor's degree, or (2) achievement of formal certifications recognized in the industry as equivalent to a bachelor's degree (e.g., information technology certifications in lieu of a degree)

Interpersonal Skills: Work involves regular collaboration with operations, IT, and business stakeholders. Requires diplomacy, discretion, and the ability to communicate clearly across technical and non-technical audiences.

Other Skills:
<ul><li>Strong analytical thinking and problem-solving ability.</li></ul>
<ul><li>Ability to understand how users interact with systems and why failures occur.</li></ul>
<ul><li>Strong written communication and documentation skills.</li></ul>
<ul><li>Experience with conversational AI platforms preferred (Kore.ai, Dialogflow, Amazon Lex, Nuance, etc.).</li></ul>
<ul><li>Experience writing decision logic, conversational flows, or scripts preferred.</li></ul>
<ul><li>QA, UAT, or system configuration experience preferred.</li></ul>
<ul><li>Understanding of customer interaction journeys.</li></ul>
<ul><li>Basic familiarity with data analysis or reporting tools preferred.</li></ul>
<ul><li>Ability to manage multiple projects with limited supervision.</li></ul>
<ul><li>Strong attention to detail and ability to anticipate edge cases and user misunderstandings.</li></ul>

Physical Requirements:

While performing the duties of the job, the employee is regularly required to:
<ul>
<li>Position yourself to move about the work area.
</li><li>Ability to sit for at least 4 hours at a time.
</li><li>Reach, grab, sort.
</li><li>Communicate in person and via telephone with others.
</li><li>Identify and recognize small font communication.
</li><li>Ability to transport a minimum of 10 pound.</li></ul></div>

Business Systems Analyst - AI and Virtual Agent (Onsite)

YO IT CONSULTING

<div>Location: Remote
Type: Full-time or Part-time Contract Work
 Fluent Language Skills Required: English &amp; Brazilian Portuguese
Why This Role Exists :
This organization works with AI teams to improve the quality, usefulness, and reliability of general-purpose conversational AI systems. These systems are used across a wide range of everyday and professional scenarios, and their effectiveness depends on how clearly, accurately, and helpfully they respond to real user questions.
This project focuses on evaluating and improving general chat behavior in large language models (LLMs). You will assess model-generated responses across diverse topics, provide high-quality human feedback, and help ensure AI systems communicate in ways that are accurate, well-reasoned, and aligned with human expectations.
What You&rsquo;ll Do :
<ul><li><ul><li>Evaluate LLM-generated responses on their ability to effectively answer user queries
</li><li>Conduct fact-checking using trusted public sources and external tools
</li><li>Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies
</li><li>Assess reasoning quality, clarity, tone, and completeness of responses
</li><li>Ensure model responses align with expected conversational behavior and system guidelines
</li><li>Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines
</li></ul></li></ul>Who You Are :
<ul><li><ul><li>You hold a Bachelor&rsquo;s degree
</li><li>You are a native speaker or have ILR 5/primary fluency (C2 on the CEFR scale) in Brazilian Portuguese
</li><li>You have significant experience using large language models (LLMs) and understand how and why people use them
</li><li>You have excellent writing skills and can clearly articulate nuanced feedback
</li><li>You have strong attention to detail and consistently notice subtle issues others may overlook
</li><li>You are adaptable and comfortable moving across topics, domains, and customer requirements
</li><li>You have a background or experience in domains requiring structured analytical thinking (e.g., research, policy, analytics, linguistics, engineering)
</li><li>You have excellent college-level mathematics skills
</li></ul></li></ul>Nice-to-Have Specialties :
<ul><li><ul><li>Prior experience with RLHF, model evaluation, or data annotation work
</li><li>Experience writing or editing high-quality written content
</li><li>Experience comparing multiple outputs and making fine-grained qualitative judgments
</li><li>Familiarity with evaluation rubrics, benchmarks, or quality scoring systems
</li></ul></li></ul>What Success Looks Like :
<ul><li><ul><li>You identify factual inaccuracies, reasoning errors, and communication gaps in model responses
</li><li>You produce clear, consistent, and reproducible evaluation artifacts
</li><li>Your feedback leads to measurable improvements in response quality and user experience</li></ul></li></ul></div>

Brazilian Portuguese Expert - Remote

<div>Work Mode: Remote 
Engagement Type: Independent Contractor 
Schedule: Full-Time or Part-Time Contract
Role:
Partners with leading AI teams to improve the quality, usefulness, and reliability of general-purpose conversational AI systems. These systems are used across a wide range of everyday and professional scenarios, and their effectiveness depends on how clearly, accurately, and helpfully they respond to real user questions.
This project focuses on evaluating and improving general chat behavior in large language models (LLMs). You will assess model-generated responses across diverse topics, provide high-quality human feedback, and help ensure AI systems communicate in ways that are accurate, well-reasoned, and aligned with human expectations.
What You&rsquo;ll Do
<ul><li>Evaluate LLM-generated responses on their ability to effectively answer user queries
</li><li>Conduct fact-checking using trusted public sources and external tools
</li><li>Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies
</li><li>Assess reasoning quality, clarity, tone, and completeness of responses
</li><li>Ensure model responses align with expected conversational behavior and system guidelines
</li><li>Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines
</li></ul>Who You Are
<ul><li>You hold a Bachelor&rsquo;s degree
</li><li>You are a native speaker or have ILR 5/primary fluency (C2 on the CEFR scale) in Arabic
</li><li>You have significant experience using large language models (LLMs) and understand how and why people use them
</li><li>You have excellent writing skills and can clearly articulate nuanced feedback
</li><li>You have strong attention to detail and consistently notice subtle issues others may overlook
</li><li>You are adaptable and comfortable moving across topics, domains, and customer requirements
</li><li>You have a background or experience in domains requiring structured analytical thinking (e.g., research, policy, analytics, linguistics, engineering)
</li><li>You have excellent college-level mathematics skills
</li></ul>Nice-to-Have Specialties
<ul><li>Prior experience with RLHF, model evaluation, or data annotation work
</li><li>Experience writing or editing high-quality written content
</li><li>Experience comparing multiple outputs and making fine-grained qualitative judgments
</li><li>Familiarity with evaluation rubrics, benchmarks, or quality scoring systems
</li></ul>What Success Looks Like
<ul><li>You identify factual inaccuracies, reasoning errors, and communication gaps in model responses
</li><li>You produce clear, consistent, and reproducible evaluation artifacts
</li><li>Your feedback leads to measurable improvements in response quality and user experience
</li></ul><h2 class="jobSectionHeader">Contract and Payment Terms</h2>
<ul><li>You will be engaged as an independent contractor.</li>
<li>This is a fully remote role that can be completed on your own schedule.</li>
<li>Projects can be extended, shortened, or concluded early depending on needs and performance.</li>
<li>Payments are weekly on Stripe or Wise based on services rendered.</li></ul></div>

Arabic & English Ai Trainer - Remote

Search 10000+ Conversational ai analyst jobs near you this March. New roles added regularly across leading industries. Apply today.

10000 conversational ai analyst jobs Near Me

Conversational AI Analyst - Fully Remote

Conversational AI Analyst - Fully Remote

Business Systems Analyst - AI and Virtual Agent (Onsite)

Brazilian Portuguese Expert - Remote

Brazilian Portuguese Expert - Remote

Brazilian Portuguese Expert - Remote

Arabic & English Ai Trainer - Remote