Executive Director, AI and Digital Health Assurance

Not specified

Posted 5 days ago

About the role

ABOUT ECRI

ECRI is an independent, nonprofit healthcare safety organization advancing evidence-based, effective healthcare globally. For more than 50 years, we've helped healthcare leaders make smarter, safer decisions — free from vendor influence or conflicting interests. Today, ECRI is home to a family of trusted safety brands, including the Institute for Safe Medication Practices (ISMP) and The Just Culture Company, extending our reach across patient safety, medication safety, and safety culture transformation. Join us, and your work will directly protect patients worldwide.

WHAT WE OFFER

Comprehensive healthcare coverage: Medical, dental, vision, life insurance, disability coverage, critical illness insurance, and more.
Retirement savings: Employer-matching Retirement Savings Plan.
Culture of Belonging: Participate in employee groups to learn from, growth with, and support colleagues in our collective success.
Purpose beyond the paycheck: 16 paid hours annually through ECRI Cares to volunteer with preapproved charities.

POSITION SUMMARY

ECRI is building the healthcare industry's independent assurance capability for artificial intelligence. The Executive Director, AI and Digital Health Assurance designs, launches, and leads this new client-facing offering: a closed-loop practice that independently tests and validates AI-enabled solutions before deployment, surveils real-world AI safety events, investigates suspected AI-related harm, and monitors deployed systems for drift, degradation, and inequitable performance.

Scope spans AI-enabled clinical decision support and large language model applications, Software as a Medical Device (SaMD) and Software in a Medical Device (SiMD), connected and wearable devices, and EHR-integrated software. The Executive Director sets ECRI's AI evaluation strategy and methodology, builds the team and technical infrastructure to execute it at scale, and carries commercial accountability for the design, launch, and growth of the offering.

The role operates from a position no other organization holds: ECRI's federally listed Patient Safety Organization and its patient safety event corpus, its device hazard and alerts apparatus, and more than five decades of independent, conflict-free technology evaluation. Preserving that independence is a core obligation of the role, not an administrative formality.

The underlying technology moves in months, not years. This role requires continuous, hands-on command of the frontier of AI capability, tooling, adoption patterns, and threats, sustained as an ongoing discipline rather than a body of knowledge fixed at hire.

ESSENTIAL FUNCTIONS

Reasonable Accommodation Statement:

To accomplish this job successfully, an individual must be able to perform, with or without reasonable accommodation, each essential function satisfactorily. Reasonable accommodations may be made to help enable qualified individuals with disabilities to perform the essential functions.

Essential Functions:

Design and launch the offering. Define the service architecture, methodology, delivery model, and commercial structure for ECRI's AI assurance offering, and take it from concept to an operating, revenue-generating practice spanning advisory, independent evaluation, event investigation, and ongoing monitoring.

Carry commercial accountability. Partner with sales, marketing, and product strategy to build pipeline, serve as the senior technical authority in client pursuits, and own bookings, revenue, margin, and retention for the practice.

Productize what repeats. Convert recurring elements of the practice into scalable products and tools, including assessment frameworks, benchmark and reference datasets, monitoring instrumentation, and certification-readiness toolkits, in partnership with product strategy and the data and analytics products team.

Set evaluation strategy and methodology. Establish ECRI's approach to evaluating AI-enabled solutions: evidence requirements, test design, reference and adversarial datasets, data provenance and fitness-for-context, subgroup and fairness performance, acceptance criteria, and reporting standards. Require disparities to be quantified and mitigated, not asserted away. Ensure methods are defensible, reproducible, and published where doing so advances the field.

Maintain frontier fluency and track adoption. Personally track foundation model releases, agentic architectures, and clinical AI platforms as they ship, distinguishing genuine capability advances from marketing claims. Maintain a current view of how healthcare organizations are actually deploying AI, including the shift of agentic AI from pilot to production and the prevalence of unsanctioned "shadow AI" use. Translate both into updated evaluation criteria and where ECRI's evaluation and monitoring capacity is applied.

Build and operate the test environment. Direct the design and operation of a secure sandbox for exercising AI systems against curated, edge-case, and adversarial inputs, covering accuracy, robustness, and failure modes, and, for generative and LLM-based tools, non-determinism, confabulation, and prompt sensitivity.

Stand up AI safety event surveillance. Working with ECRI's Patient Safety Organization and hazard reporting programs, build the intake, taxonomy, and analytic capability to collect and classify AI-related safety events, near misses, and performance failures at scale, and to generate actionable signal for clients, vendors, and the field.

Lead investigations of suspected AI-related harm. Direct root cause analyses of events in which an AI-enabled tool is implicated, applying systems-based and just culture methods to distinguish model failure, data failure, integration failure, workflow failure, and human-automation interaction failure, and translate findings into corrective action.

Design post-deployment monitoring and close the loop. Specify how clients detect model drift, data distribution shift, degradation following vendor updates, and erosion of clinical benefit over time, including thresholds, cadence, escalation paths, and triggers for revalidation or decommissioning. Ensure findings from surveillance and investigation systematically update ECRI's evaluation criteria, hazard communications, and published guidance, so real-world field experience continuously strengthens pre-deployment testing.

Advise senior leaders and track the regulatory landscape. Serve as ECRI's senior authority on AI assurance to boards, executive teams, AI governance committees, and vendor executives, translating technical findings, including limitations, uncertainty, and the recommendation not to deploy, into decisions those audiences can act on. Track and interpret FDA expectations for AI-enabled device software functions, Joint Commission's Responsible Use of AI in Healthcare certification standards, CHAI governance playbooks, NIST AI RMF, and ISO/IEC 42001, ISO 14971, and IEC 62304, and position ECRI's offering to help clients meet them.

Direct security and threat evaluation of AI-enabled systems. Oversee assessment of AI-specific threats, including prompt injection and adversarial manipulation, agentic and tool-use attack surfaces (e.g., Model Context Protocol-style integrations), model and data supply chain exposure, unsanctioned "shadow AI" use, and access control for connected and EHR-integrated deployments. Reference frameworks such as the OWASP Top 10 for LLM Applications and MITRE ATLAS, in coordination with ECRI's cybersecurity capability.

Build and lead the team. Recruit, develop, and lead a multidisciplinary team spanning clinical, data science, engineering, human factors, and cybersecurity expertise. Establish standard operating procedures, training, quality control, and peer review so that output is consistent, reproducible, and defensible at scale.

Protect ECRI's independence. Own and enforce the conflict-of-interest and firewall protocols that keep ECRI's evaluations free of vendor influence, including clear separation between advisory work, independent evaluation, and any vendor-facing revenue.

Additional Responsibilities:

Represent ECRI in external standards, consensus, and policy forums on health AI assurance, including regulatory dockets, professional societies, and industry coalitions.

Contribute AI-related content to ECRI's flagship publications and thought leadership, including the annual Top 10 Health Technology Hazards and Top 10 Patient Safety Concerns.

Advise ECRI's internal AI program on the safe, appropriate, and defensible use of AI within ECRI's own products, services, and operations.

Other duties, as assigned.

Accountability Metrics:

Design and launch of the AI assurance offering against agreed milestones, including a published methodology and a defined, priced service catalog.

Bookings, revenue, and margin for the practice; number and caliber of reference clients secured.

Volume, cycle time, and quality of completed evaluations and investigations, with documented adherence to methodology and peer review standards.

Growth and analytic yield of ECRI's AI safety event corpus, and demonstrable feedback of findings into evaluation criteria and published guidance.

Client satisfaction and retention, and evidence of client action taken on ECRI findings.

Team recruitment, retention, and capability development against plan.

Integrity of ECRI's independence, evidenced by zero substantiated conflict-of-interest findings.

Meet or exceed annual organizational, divisional, departmental, and individual goals.

POSITION QUALIFICATIONS

Experience:

10+ years of progressive experience in healthcare technology evaluation, clinical software quality and validation, medical device or health IT safety, or a closely related discipline.

3+ years of hands-on leadership or delivery experience specifically involving AI- or ML-enabled systems, including meaningful direct exposure to generative AI, agentic systems, or large language model applications.

5+ years leading teams, with demonstrated success building a function, practice, or program from an early or undefined state rather than inheriting a mature one.

Demonstrated experience shipping an AI-enabled product or feature into production and operating it post-launch, with direct accountability for real-world performance, user adoption, and iteration. Academic research, pilots, or third-party evaluation alone do not substitute for this.

Deep and current command of AI and machine learning as applied in healthcare, including the distinct evaluation challenges posed by generative, agentic, and large language model systems, sustained through active use of current tools rather than fluency anchored to a single past project.

Demonstrated experience designing and executing validation or evaluation strategies for clinical software in regulated or highly governed environments, including design controls, traceability, change control, and risk management.

Experience building or operating technical infrastructure to exercise and test AI systems and their outputs in a controlled sandbox environment, including construction of benchmark and adversarial test sets.

Proven client-facing experience translating provider, payer, and vendor needs into measurable evaluation plans, and communicating technical and clinical findings, including limitations, uncertainty, and risk implications, in a practical and action-oriented manner.

Demonstrated commercial contribution: shaping offerings, scoping and pricing engagements, supporting pursuits, and growing client relationships.

Established credibility with senior healthcare and industry leaders, evidenced by publication, standards or consensus body participation, or speaking at recognized venues.

Preferred: working knowledge of patient safety event analysis and root cause methodology, or demonstrated capability leading investigations of clinical harm events. Experience within a Patient Safety Organization, risk management, or quality and safety function is a strong advantage.

Preferred: experience in one or more adjacent domains, such as connected and wearable device validation, EHR-integrated clinical decision support and workflow integration, or familiarity with the health AI assurance frameworks now in market (Joint Commission's Responsible Use of AI in Healthcare certification, CHAI governance playbooks).

Education:

Bachelor's degree required in computer science, biomedical or software engineering, data science, information security, health informatics, or a related technical or scientific field.

Advanced degree strongly preferred: a master's or doctoral degree in a related technical field, or a clinical doctorate (MD, DO, PharmD, DNP, PhD) paired with substantial informatics, data science, or health technology evaluation experience.

An equivalent combination of education and directly relevant experience will be considered.

Computer Skills:

Advanced, current, hands-on fluency across multiple leading foundation model families (proprietary and open-weight), agent and orchestration frameworks, and model/prompt evaluation and observability tooling, maintained through active use rather than point-in-time familiarity.

Working knowledge of AI-specific threat and risk frameworks, such as the OWASP Top 10 for LLM Applications and MITRE ATLAS.

Working familiarity with data analysis environments such as Python, R, or SQL, sufficient to direct technical work and independently interrogate results.

Proficient with Microsoft 365 (Word, Excel, PowerPoint) and standard collaboration tools.

Certifications and Licenses:

Required: None.

Preferred (AI governance and management systems): ISO/IEC 42001 lead implementer or auditor, or an equivalent AI management system credential.

Preferred (Medical device and SaMD quality and regulatory): RAPS RAC (US or Global) and/or ASQ CQA or CQE, or an equivalent quality systems credential.

Preferred (Patient safety): Certified Professional in Patient Safety (CPPS).

POSITION COMPENSATION

The salary range for new employees in this position is $179,994.00 - $224,992.00 based on background, experience, and skills. In addition, new employees in this position are eligible for all of our benefit offerings, including, but not limited to, health and welfare benefits, 403(B) retirement savings, and paid time off (PTO).

PHYSICAL DEMANDS

This position operates in a remote environment and requires the individual to remain in a stationary position, whether sitting or standing, before a desk or other fixed workspace, most of their workday. In addition, this position requires the individual to occasionally move about their workspace to access and inspect work-related materials, such as file cabinets with physical files and standard office equipment. This position requires the ability to operate standard office equipment, including, but not limited to, a laptop, keyboard, mouse, webcam, and phone, as well as effectively communicate information and ideas to a wide variety of audiences in written and oral form.

ADA STATEMENT

ECRI is committed to providing equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, genetics, sexual orientation, gender identity, or veteran status. We value diversity and believe that a diverse workforce enhances our ability to succeed. ECRI complies with applicable federal, state, and local laws governing nondiscrimination in employment and prohibits any form of discrimination or harassment based on these protected characteristics.

EEO STATEMENT

#LI-Remote