AI Agent Experience Designer

POSITION SUMMARY

The Prompt Engineer will be responsible for optimizing, evaluating, and advancing the performance of AI-driven agents, such as chatbots and autonomous workflows to ensure they deliver accurate, efficient, and high-quality outcomes. This role combines analytical thinking, user experience insight, and technical expertise to monitor agent behavior, identify improvement opportunities, and implement enhancements that align with business goals.

ESSENTIAL FUNCTIONS

• Define and document required AI agent behavior, including edge cases, compliance constraints, escalation expectations, and response quality targets.
• Convert benefits workflows and policy rules into implementable use cases, key data fields to capture, and testable scenarios that can be built and verified.
• Maintain a clear mapping from requests to delivered behavior so gaps are visible and reviewable.
• Drive stakeholder reviews and acceptance of behavior, flows, and decision logic before changes ship.
• Craft and iterate prompt instructions and response rules that produce accurate, consistent answers in a regulated benefits context.
• Build reusable prompt patterns and maintain a controlled prompt library with clear version history.
• Review real interactions to identify failure modes, then adjust prompts, context, and fallbacks to improve outcomes.
• Partner with engineering and QA to define regression test coverage and release criteria for prompt and flow changes.
• Define the agent’s voice and interaction style so it is professional, on-brand, and easy for members and providers to follow.
• Design multi-step conversation paths that handle ambiguity, corrections, and interruptions without losing the user.
• Define when the agent should answer, when it should ask clarifying questions, and when it should hand off to a live person.
• Validate that wording, terminology, and risk posture match benefits operations expectations.
• Define how the agent should use plan documents to answer questions, including what to do when source material is missing or ambiguous.
• Partner with contact center and workforce management teams to align the agent experience with SOPs and ensure workflows stay current as procedures change.
• Partner with engineering on query shaping and grounding so document-based answers stay tied to supporting text.
• Own the gold standard definition of intended agent behavior after launch, so teams can support and improve it without relying on vendor or institutional knowledge.
• Maintain the core operating artifacts used day to day: use case catalog, flow maps, escalation rules, prompt library, and a running list of known failure modes.
• Create lightweight training and working sessions for internal support partners and business SMEs.
• Establish a change process for updating prompts and flows, including testing expectations, approvals, and version history.
• Requirements and functional specifications for prioritized conversational AI use cases.
• Use case catalog and key information requirements mapped to approved scenarios.
• Conversation flow designs that include fallbacks, error handling, and escalation.
• Versioned prompt library with testing notes and a clear change history.
• UAT scripts and acceptance criteria, plus defect tracking through resolution.
• Maintain a curated set of golden records and test cases that represent critical scenarios and edge cases and use them to support automated regression and quality testing.
• Operator documentation that enables internal support and ongoing improvement.
• Post-launch performance monitoring, including top failure modes and improvement plans.
• Other duties as assigned

EDUCATION

• Bachelor’s degree in Computer Science, Math, Statistics, UI/UX, Process Design or equivalent work experience required.

EXPERIENCE AND SKILLS:

• 6+ years of experience across business analysis, conversational AI, UX writing, and related product roles.
• Direct experience designing and improving chatbots, voicebots, IVR modernization, or contact center automation.
• Ability to translate business workflows into clear user requests, key information to capture (e.g., member identifiers, dates, plan options), and end-to-end conversation paths that engineering can build and QA can test.
• Strong prompt design skills and a test-and-learn mindset using real interaction data.
• Strong writing and editing skills with comfort in a regulated, customer-facing environment.
• Comfort operating with ambiguity in a new and evolving role, and able to create processes and standards where none exist.
• Ability to collaborate with cross-functional stakeholders and external partners.
• Shipped conversational AI experiences in voice and/or contact center environments.
• Experience working with plan documents, knowledge bases, and policy text in regulated environments. Able to translate source material into grounded answers and guardrails.

POSITION COMPETENCIES

• Accountability
• Analytical Problem Solving
• Collaboration
• Execution and Delivery
• Quality and Risk Management
• Systems Thinking
• Technical/Functional Expertise

PHYSICAL DEMANDS

• This is a standard desk role requiring extended sitting and computer work

WORK ENVIRONMENT

Remote

Here at Allied, we believe that great talent can thrive from anywhere. Our remote friendly culture offers flexibility and the comfort of working from home; while also ensuring you are set up for success. To support a smooth and efficient remote work experience, the internet connection must be obtained through a cable broadband or fiber optic internet service provider with speeds of at least 100Mbps download/25Mbps upload. Reliable internet service is essential for staying connected and productive.

The company has reviewed this job description to ensure that essential functions and basic duties have been included. It is not intended to be construed as an exhaustive list of all functions, responsibilities, skills, and abilities. Additional functions and requirements may be assigned by supervisors as deemed appropriate.

Compensation is not limited to base salary. Allied values our Total Rewards, and offers a competitive Benefit Package including, but not limited to, Medical, Dental, Vision, Life and Disability Insurance, Generous Paid Time Off, Tuition Reimbursement, EAP, and a Technology Stipend.

Allied reserves the right to amend, change, alter, and revise, pay ranges and benefits offerings at any time. All applicants acknowledge that by applying to the position you understand that the specific pay range is contingent upon meeting the qualification and requirements of the role, and for the successful completion of the interview selection and process. It is at the Company's discretion to determine what pay is provided to a candidate within the range associated with the role.