AI QA Test Engineer

Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.

About the Role

Experienced AI QA Test Engineer to ensure the quality, reliability, and ethical performance of AI-driven systems and applications. This role requires a deep understanding of AI/ML workflows, data validation, model testing techniques, and automation frameworks. The ideal candidate is highly analytical, detail-oriented, and passionate about building trusted AI products.

Participates in QA activities on project assignments and ensures timely delivery on QA commitments in a fast-paced product development environment.

Review of functional requirements and technical design documentation and creation of test plans, test scenarios and test cases. Execution of test plans and certification of application releases

Key Responsibilities:

AI/ML Testing

• Validate ML models, data pipelines, and inference APIs for accuracy, fairness, and robustness
• Evaluate model behavior across diverse data sets, edge cases, and unexpected inputs
• Monitor model drift, performance degradation, and bias over time
• Conduct prompt testing, hallucination detection, and safety validation for LLM-based systems
• Experience of testing implementation of AI/ML into application
• Manual Testing with automation exposure
• API testing with SOAP UI/ Postman
• Agile methodology experience
• Test & Defect management with JIRA/ Azure DevOps (ADO)
• DB Testing Experience - Good to Have
• Tools - Deepeval/ Langfuse/ Pytest

Quality Engineering & Automation

• Build and maintain automated test suites for AI features
• Develop test scripts for ML workflows using Python, Playwright, and CI/CD tools
• Design and execute manual and automated tests for functional, performance, and security scenarios
• Identify defects, establish test metrics, reporting dashboards, and test traceability

Data Quality & Responsible AI

• Validate training and inference data quality, labeling consistency, and data governance
• Work with data science and product teams to implement responsible AI testing guidelines
• Document and escalate risks related to bias, privacy, explainability, and compliance

Cross-Functional Collaboration

• Partner with Data Scientists, ML Engineers, and Product Managers to define acceptance criteria
• Participate in design reviews, model evaluations, and release readiness checkpoints
• Contribute to QA strategy and Test Plans

Required Qualifications

• Bachelor’s degree in computer science, Engineering, or related field (or equivalent experience)
• 4+ years of experience in software QA engineering
• 2+ years working with AI/ML or data-centric systems
• Strong Python programming skills
• Proficiency in automation frameworks (e.g., Playwright, WebdriverIO, Selenium)
• Experience with ML platforms (TensorFlow, PyTorch, Hugging Face, MLflow, etc.)
• Hands-on experience testing APIs, data pipelines, and distributed systems
• Exposure to RAG pipelines and agent frameworks
• Familiarity with prompt testing and LLM evaluation techniques
• Experience with model evaluation metrics (precision, recall, F1, ROC-AUC, BLEU, perplexity, etc.)
• Knowledge of MLOps, CI/CD, and cloud platforms (AWS, Azure, or GCP)
• Familiarity with Responsible AI frameworks (NIST AI RMF, ISO/IEC standards, fairness toolkits)
• Experience testing vector databases and AI search (Pinecone, Elasticsearch, Redis Vector)

Key Skills

• AI/ML testing & analysis
• Automation & scripting
• Data validation & pipelines
• Risk, ethics, & compliance mindset
• Strong problem-solving & debugging skills
• Excellent documentation & communication skills
• Working with distributed teams

Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.

Candidate Privacy Policy

Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains:

• What information we collect during our application and recruitment process and why we collect it;
• How we handle that information; and
• How to access and update that information.

Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.