Arabic LLM Evaluation & Cultural Alignment Expert

We are looking for linguistically and culturally aware professionals to support the evaluation and enhancement of multilingual prompt-response datasets for large language models (LLMs). This role involves rubric design, evaluation of translations and model outputs, prompt creation, and red teaming focused on identifying and surfacing cultural nuances and biases in LLM behaviour.

Key Responsibilities :

Rubric Definition & Prompt Evaluation

• Update rubric definitions with Arabic specific examples to ensure cultural and linguistic relevance.
• Identify the need for additional rubrics tailored to specific languages or regional contexts.
• Review prompts translated from English into Arabic and revise where translations appear unnatural or inaccurate.
• Writing of thoughtful prompts which can test the cultural awareness of LLM models.
• Rate prompt-response pairs using a standardized evaluation template based on rubrics and provide detailed justifications to base the findings.
• Document problematic outputs and annotate them with clear explanations of rubric violations or cultural insensitivities.

Required Qualifications :

Preferred Qualifications :