QA Engineer – AI Ops

JOB_REQUIREMENTS

Hires in

Not specified

Employment Type

Not specified

Company Location

Not specified

Salary

Not specified

Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.

Job Overview:

As a QA Engineer in a AI Ops Business Pod, you will play a critical role in ensuring the quality, stability, and correctness of GenAI-powered agents and workflows. You will design and execute test plans, automate regression checks, and validate agent behavior under varied inputs and edge cases. This role requires an analytical mindset, understanding of AI workflows, and a keen eye for risk and detail in automated decisions.

Key Responsibilities

Design and implement test strategies and plans for GenAI agents and integrated workflows
Create and execute functional, integration, and regression test cases for agents built on Agent OS and ServiceNow
Validate LLM-driven outputs, including prompt results, decision paths, and workflow accuracy
Collaborate closely with AI Engineers, Pod Lead, and Domain SMEs to understand business logic and critical flows
Build and maintain automated test scripts (where applicable) and simulate edge cases
Participate in design and code reviews to provide early feedback on quality, completeness, and testability
Own defect triage, ensure traceability between requirements and test cases, and manage bug tracking
Validate agent readiness for deployment and provide go/no-go recommendations
Document test results, quality metrics, and lessons learned for continuous improvement

Key skills:

5+ years of experience in QA/test engineering, preferably in enterprise or automation-heavy environments
Experience testing AI/GenAI-powered applications, LLM workflows, or agentic systems is highly desirable
Strong familiarity with API testing, test case design, and working in Agile POD-based teams
Exposure to platforms like ServiceNow, LangChain, Agent OS (or similar workflow automation tools) a plus
Working knowledge of test automation tools (e.g., Pytest, Postman, Selenium, or similar frameworks)
Ability to think critically about test scenarios, edge cases, and quality risks
Excellent documentation and communication skills
Comfortable collaborating across engineering, product, and business teams

What You'll Gain

Opportunity to work on cutting-edge GenAI solutions in a large-scale enterprise automation program
Exposure to proprietary AI agent platforms, enterprise workflows, and real-world AI governance
A collaborative, fast-paced environment where quality is core to delivering business value

Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.

Candidate Privacy Policy

Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, "Orion," "we" or "us") are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) ("Notice") explains:

What information we collect during our application and recruitment process and why we collect it;
How we handle that information; and
How to access and update that information.

Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.

Similar jobs