language model evaluator

MERCOR

<div>About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.
Position: Language Model Evaluator 
Type: Full-time or Part-time Contract Work 
Compensation: $23/hour 
Location: Geography restricted to Egypt, Saudi Arabia, UAE, USA
Role Responsibilities<ul><li>
Evaluate LLM-generated responses on their ability to effectively answer user queries.</li><li>
Conduct fact-checking using trusted public sources and external tools.</li><li>
Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies.</li><li>
Assess reasoning quality, clarity, tone, and completeness of responses.</li><li>
Ensure model responses align with expected conversational behavior and system guidelines.</li><li>
Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines.</li></ul>
Qualifications
Must-Have<ul><li>
Bachelor&rsquo;s degree</li><li>
Native speaker or ILR 5/primary fluency (C2 on the CEFR scale) in Arabic</li><li>
Significant experience using large language models (LLMs)</li><li>
Excellent writing skills</li><li>
Strong attention to detail</li><li>
Adaptable and comfortable moving across topics, domains, and customer requirements</li><li>
Background or experience in domains requiring structured analytical thinking</li><li>
Excellent college-level mathematics skills</li></ul>
Preferred<ul><li>
Prior experience with RLHF, model evaluation, or data annotation work</li><li>
Experience writing or editing high-quality written content</li><li>
Experience comparing multiple outputs and making fine-grained qualitative judgments</li><li>
Familiarity with evaluation rubrics, benchmarks, or quality scoring systems</li></ul>
Application Process (Takes 20&ndash;30 mins to complete)<ul><li>
Upload resume</li><li>
AI interview based on your resume</li><li>
Submit form</li></ul>
Resources &amp; Support<ul><li>
For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome/welcome</li><li>
For any help or support, reach out to: support@mercor.com</li></ul>
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.</div>

Content Reviewer

Data Annotator

Senior Content Reviewer

Bilingual LLM Evaluator - Arabic Expert

Mercor

<div>
 
 About The Job
 
 
 
 
 Mercor
 
 connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include
 
 Benchmark
 
 ,
 
 General Catalyst
 
 ,
 
 Peter Thiel
 
 ,
 
 Adam D'Angelo
 
 ,
 
 Larry Summers
 
 , and
 
 Jack Dorsey
 
 .
 
 
 
 Position:
 
 Language Model Evaluator
 
 
 
 Type:
 
 
 Full-time or Part-time Contract Work
 
 
 
 
 Compensation:
 
 
 $23/hour
 
 
 
 
 Location:
 
 
 Geography restricted to Egypt, Saudi Arabia, UAE, USA
 
 
 
 
 Role Responsibilities
 
 
 
 <ul>
 <li>
 Evaluate LLM-generated responses on their ability to effectively answer user queries.
 </li>
 <li>
 Conduct fact-checking using trusted public sources and external tools.
 </li>
 <li>
 Generate high-quality human evaluation data by annotating response strengths, areas for improvement, and factual inaccuracies.
 </li>
 <li>
 Assess reasoning quality, clarity, tone, and completeness of responses.
 </li>
 <li>
 Ensure model responses align with expected conversational behavior and system guidelines.
 </li>
 <li>
 Apply consistent annotations by following clear taxonomies, benchmarks, and detailed evaluation guidelines.
 
 
 
 </li>
 </ul>
 
 Qualifications
 
 
 
 
 Must-Have
 
 
 
 <ul>
 <li>
 Bachelor&rsquo;s degree
 </li>
 <li>
 Native speaker or ILR 5/primary fluency (C2 on the CEFR scale) in Arabic
 </li>
 <li>
 Significant experience using large language models (LLMs)
 </li>
 <li>
 Excellent writing skills
 </li>
 <li>
 Strong attention to detail
 </li>
 <li>
 Adaptable and comfortable moving across topics, domains, and customer requirements
 </li>
 <li>
 Background or experience in domains requiring structured analytical thinking
 </li>
 <li>
 Excellent college-level mathematics skills
 
 
 
 </li>
 </ul>
 
 Preferred
 
 
 
 <ul>
 <li>
 Prior experience with RLHF, model evaluation, or data annotation work
 </li>
 <li>
 Experience writing or editing high-quality written content
 </li>
 <li>
 Experience comparing multiple outputs and making fine-grained qualitative judgments
 </li>
 <li>
 Familiarity with evaluation rubrics, benchmarks, or quality scoring systems
 
 
 
 </li>
 </ul>
 
 Application Process (Takes 20&ndash;30 mins to complete)
 
 
 
 <ul>
 <li>
 Upload resume
 </li>
 <li>
 AI interview based on your resume
 </li>
 <li>
 Submit form
 
 
 
 </li>
 </ul>
 
 Resources &amp; Support
 
 
 
 <ul>
 <li>
 For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome/welcome
 </li>
 <li>
 For any help or support, reach out to: support@mercor.com
 
 
 
 </li>
 </ul>
 
 PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
 
</div>

Conversational AI Analyst - Fully Remote

Invisible Agency

Are you a Hijazi language expert eager to shape the future of AI? Large‑scale language models are evolving from clever chatbots into powerful engines of linguistic discovery. With high‑quality training data, tomorrow's AI can democratize world‑class education, keep pace with cutting‑edge research, and streamline communication for Hijazi speakers everywhere. That training data begins with you—we need your expertise to help power the next generation of AI.
 
We're looking for Hijazi language specialists who live and breathe Hijazi grammar, syntax, morphology, phonology, semantics, and pragmatics. You'll challenge advanced language models on topics like verb conjugation, case system (nominative, accusative, dative, genitive), word order, declensions, gender, and stylistic variation—documenting every failure mode so we can harden model reasoning.
 
On a typical day, you will converse with the model on language scenarios, verify factual accuracy and logical soundness, capture reproducible error traces, and suggest improvements to our prompt engineering and evaluation metrics.
 
A bachelor's degree in Hijazi language, linguistics, or a closely related field is ideal, but we also highly value native-level fluency and a deep, practical understanding of the language's nuances. Extensive experience in teaching, translation, or community-based linguistic projects can substitute for a formal degree. Clear, metacognitive communication—"showing your work"—is essential.
 
Ready to turn your Hijazi language expertise into the knowledge base for tomorrow's AI? Apply today and start teaching the model that will teach the world.
 
We offer a pay range of $6-to- $65 per hour, with the exact rate determined after evaluating your experience, expertise, and geographic location. Final offer amounts may vary from the pay range listed above. As a contractor you'll supply a secure computer and high‑speed internet; company‑sponsored benefits such as health insurance and PTO do not apply.
 
Employment type: Freelance / Contract
 
Workplace type: Remote
 
Seniority level: Entry Level

Hijazi Dialect Specialist - Freelance AI Trainer Project

Search 10000+ Language model evaluator jobs near you this March. New roles added regularly across leading industries. Apply today.

10000 language model evaluator jobs Near Me

Content Reviewer

Data Annotator

Senior Content Reviewer

Bilingual LLM Evaluator - Arabic Expert

Conversational AI Analyst - Fully Remote

Conversational AI Analyst - Fully Remote

Hijazi Dialect Specialist - Freelance AI Trainer Project