software engineer

ai engineer @ pre series a startup Jobs in New York

ai engineer @ pre series a startup

ai evaluations engineer

engineer

engineering manager

ai engineer

VectorShift

<div>
 VectorShift (YC S23) is building the AI enterprise workflow automation. We convert complex, manual processes into fully automated AI systems that scale across global teams.
 <br/>
 <br/>
 <strong>
  About The Role
  <br/>
  <br/>
 </strong>
 As an AI Engineer, you will work across our core AI infrastructure &mdash; from multi-agent systems and RAG pipelines to evaluation frameworks and production deployment. You'll ship AI-powered features that clients rely on daily, ensuring they're performant, reliable, and secure. This is a high-ownership role where you'll contribute across the full stack: backend APIs, data pipelines, frontend interfaces, and cloud infrastructure.
 <br/>
 <br/>
 You'll report directly to the founders and collaborate closely with a tight-knit engineering team building product with real urgency and craft.
 <br/>
 <br/>
 <strong>
  Key Responsibilities
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Design, build, and deploy AI-powered features in production, including RAG systems, multi-agent orchestration, and evaluation frameworks
  </li>
  <li>
   Own backend services and APIs end-to-end (Python / FastAPI / Django)
  </li>
  <li>
   Build data pipelines for model training, evaluation, and continuous improvement
  </li>
  <li>
   Collaborate with founders, product, and design to ship high-impact features
  </li>
  <li>
   Ensure scalability, performance, and security across the platform
  </li>
  <li>
   Contribute to engineering best practices, code quality, and technical decision-making
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Required Qualifications
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   2+ years of professional software engineering experience
  </li>
  <li>
   Hands-on experience building and deploying AI/ML applications in production environments
  </li>
  <li>
   Solid understanding of relational databases and data modeling
  </li>
  <li>
   Experience with cloud infrastructure, containerization (Docker, Kubernetes), and CI/CD pipelines
  </li>
  <li>
   Familiarity with RAG pipelines, LLM orchestration, or AI agent architectures
  </li>
  <li>
   Experience with Git and collaborative development workflows
  </li>
  <li>
   Familiarity with LLM evaluation techniques and tools for measuring reliability
  </li>
  <li>
   Strong problem-solving instincts and a bias toward shipping
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  NICE TO HAVE
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Experience working in or building for large enterprises
  </li>
  <li>
   Prior startup experience or comfort operating in high-ambiguity environments
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  What We Offer
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Direct founder access &mdash; report directly to the founding team and shape the product from the inside
  </li>
  <li>
   Competitive compensation &mdash; salary + meaningful equity in a high-growth company
  </li>
  <li>
   High-impact work &mdash; build AI infrastructure used daily by top enterprises
  </li>
  <li>
   Speed and ownership &mdash; small team, fast cycles, real responsibility from day one
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  What To Expect
  <br/>
  <br/>
 </strong>
 This role is not for everyone. The work is hard, the pace is fast, and priorities will shift as we learn from the market and our clients. You need to be comfortable with ambiguity and energized by building something that hasn't been built before.
 <br/>
 <br/>
 We're assembling a small team of exceptionally talented, tenacious people who care deeply about the craft of building great software. If that's you &mdash; and you're at the point in your career where you want to make a defining contribution &mdash; we'd love to hear from you.
</div>


AI Engineer

Capgemini

<div><div><div><p>New York</p><div>
Gen AI Engineer</div><p>
</p></div></div><p></p><div><div><div><div><div><p>
Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you&rsquo;d like, where you&rsquo;ll be supported and inspired by a collaborative community of colleagues around the world, and where you&rsquo;ll be able to reimagine what&rsquo;s possible. Join us and help the world&rsquo;s leading organizations unlock the value of technology and build a more sustainable, more inclusive world.</p><div><div><div><div>
Job Location - Job Location - NYC NY (Day One Onsite - Hybrid)</div></div></div><div></div><div><div><div>
Job Description</div></div><div><div><p>
Job Description</p></div><div><p>
We are looking for a skilled Generative AI Developer to join our Controls Technology team and contribute to the development and integration of innovative generative AI solutions In this role you will work closely with senior developers AI architects and business stakeholders to deliver impactful AIdriven features and products.</p><p>
</p><p>Key Responsibilities</p><p>
Implement and finetune generative AI models under the guidance of senior team members<br>
Contribute to prompt engineering and development of AIpowered workflows<br>
Assist with the deployment monitoring and maintenance of AI models in production environments<br>
Collaborate with data scientists and engineers to ensure seamless integration of AI capabilities<br>
Perform data preprocessing feature engineering and API development for AI applications<br>
Participate in code reviews testing and documentation to ensure quality and reliability<br>
Stay updated with advancements in GenAI and share relevant learnings with the team</p><p>
</p><p>Required Technical Skills<br>
Handson experience with at least one major machine learning framework PyTorch TensorFlow Keras<br>
Familiarity with LLMs prompt engineering and finetuning eg LoRA QLoRA<br>
Exposure to RAG systems and basic knowledge of hybrid search techniques<br>
Understanding of model deployment and containerization Docker<br>
Proficiency in Python for AI development data preprocessing and scripting<br>
Experience with generative AI tools LangChain Hugging Face LlamaIndex is a plus<br>
Understanding of version control systems Git<br>
Awareness of AI compliance data privacy and responsible AI principles</p><p>
</p><p>Required Soft Skills<br>
Strong teamwork and communication abilities<br>
Willingness to learn new AIML technologies and frameworks<br>
Analytical mindset and attention to detail<br>
Openness to feedback and continuous improvement</p><p>
</p><p>Qualifications<br>
Bachelors degree or Masters degree in Computer Science Data Science AI or a related field<br>
3-5 years of professional experience in AIML development with exposure to Generative AI<br>
Experience delivering AIML projects in a collaborative setting<br>
Exposure to cloud based AIML environments AWS GCP or Azure is a plus</p></div></div></div></div><p>
The base compensation range for this role in the posted location is : $ 56186 - 87556
</p><p>Capgemini provides compensation range information in accordance with applicable national, state, provincial, and local pay transparency laws. The base compensation range listed for this position reflects the minimum and maximum target compensation Capgemini, in good faith, believes it may pay for the role at the time of this posting. This range may be subject to change as permitted by law.</p><p>
The actual compensation offered to any candidate may fall outside of the posted range and will be determined based on multiple factors legally permitted in the applicable jurisdiction.</p><p>
These may include, but are not limited to: Geographic location, Education and qualifications, Certifications and licenses, Relevant experience and skills, Seniority and performance, Market and business consideration, Internal pay equity.</p><p>
It is not typical for candidates to be hired at or near the top of the posted compensation range.</p><p>
In addition to base salary, this role may be eligible for additional compensation such as variable incentives, bonuses, or commissions, depending on the position and applicable laws.</p><p>
</p><p>Capgemini offers a comprehensive, non-negotiable benefits package to all regular, full-time employees. In the U.S. and Canada, available benefits are determined by local policy and eligibility and may include:
</p><ul><li>Paid time off based on employee grade (A-F), defined by policy: Vacation: 12-25 days, depending on grade, Company paid holidays, Personal Days, Sick Leave</li></ul><ul><li>
Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)</li><li>
Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)</li><li>
Life and disability insurance</li><li>
Employee assistance programs</li><li>
Other benefits as provided by local policy and eligibility</li></ul><p>
Important Notice: Compensation (including bonuses, commissions, or other forms of incentive pay) is not considered earned, vested, or payable until it becomes due under the terms of applicable plans or agreements and is subject to Capgemini&rsquo;s discretion, consistent with applicable laws. The Company reserves the right to amend or withdraw compensation programs at any time, within the limits of applicable legislation.</p><p>
</p><p>Disclaimers</p><p>
Capgemini is an Equal Opportunity Employer encouraging inclusion in the workplace. Capgemini also participates in the Partnership Accreditation in Indigenous Relations (PAIR) program which supports meaningful engagement with Indigenous communities across Canada by promoting fairness, accessibility, inclusion and respect. We value the rich cultural heritage and contributions of Indigenous Peoples and actively work to create a welcoming and respectful environment. All qualified applicants will receive consideration for employment without regard to race, national origin, gender identity/expression, age, religion, disability, sexual orientation, genetics, veteran status, marital status or any other characteristic protected by law.</p><p>
This is a general description of the Duties, Responsibilities and Qualifications required for this position. Physical, mental, sensory or environmental demands may be referenced in an attempt to communicate the manner in which this position traditionally is performed. Whenever necessary to provide individuals with disabilities an equal employment opportunity, Capgemini will consider reasonable accommodations that might involve varying job requirements and/or changing the way this job is performed, provided that such accommodation does not pose an undue hardship. Capgemini is committed to providing reasonable accommodation during our recruitment process. If you need assistance or accommodation, please reach out to your recruiting contact.</p><p>
Please be aware that Capgemini may capture your image (video or screenshot) during the interview process and that image may be used for verification, including during the hiring and onboarding process.</p><p></p><p><br>
</p><p>Capgemini is a global business and technology transformation partner, helping organizations to accelerate their dual transition to a digital and sustainable world, while creating tangible impact for enterprises and society. It is a responsible and diverse group of 340,000 team members in more than 50 countries. With its strong over 55-year heritage, Capgemini is trusted by its clients to unlock the value of technology to address the entire breadth of their business needs. It delivers end-to-end services and solutions leveraging strengths from strategy and design to engineering, all fueled by its market leading capabilities in AI, generative AI, cloud and data, combined with its deep industry expertise and partner ecosystem.</p><p>
</p></div></div><div><div><div><div><div>Ref. code</div><div>
426728-en_US</div></div><div><div>
Posted on</div><div>
05 Mar 2026</div></div><div><div>
Experience level</div><div>
Experienced Professionals</div></div><div><div>
Contract type</div><div>
Permanent</div></div><div><div>
Location</div><div>
New York</div></div><div><div>
Business unit</div><div>
FS</div></div><div><div>
Brand</div><div>
Capgemini</div></div><div><div>
Professional communities</div><div>
Data &amp; AI</div></div></div></div></div></div></div></div></div><p></p>

Gen AI Engineer

Stefanini Group

<div>
 <strong>
  Job Description
  <br/>
  <br/>
 </strong>
 Stefanini Group is looking for
 <strong>
  AI Engineer - Microsoft 365 Copilot Specialist
 </strong>
 for a globally recognized company!
 <br/>
 <br/>
 For interested applicants, click the apply button or you may reach out to
 <strong>
  Alfher Hidalgo at (248) 728-2627/Alfher.Hidalgo@stefanini.com
 </strong>
 for faster processing. Thank you!
 <br/>
 <br/>
 <strong>
  Job Summary
  <br/>
  <br/>
 </strong>
 We are looking for an AI Engineer who specializes in Microsoft 365 Copilot to help configure and optimize our AI-driven applications. The ideal candidate will have a deep understanding of the M365 platform, its features, and how they integrate seamlessly to enhance user experience. This role requires a professional who can ensure compliance with security protocols while maximizing the potential of AI tools within the organization.
 <br/>
 <br/>
 <strong>
  Key Responsibilities
  <br/>
  <br/>
 </strong>
 <strong>
  Copilot Configuration:
 </strong>
 Design, configure, and optimize Microsoft 365 Copilot to meet the specific needs of the organization, ensuring smooth integration with existing workflows and applications.
 <br/>
 <br/>
 <strong>
  Feature Integration:
 </strong>
 Collaborate with cross-functional teams to understand how various features within the M365 ecosystem work together, and develop strategies to enhance productivity through AI-driven solutions.
 <br/>
 <br/>
 <strong>
  Security Management:
 </strong>
 Assess and implement security measures related to the use of Copilot, ensuring compliance with organizational policies and industry standards. Stay updated on best practices for data privacy and security in AI applications.
 <br/>
 <br/>
 <strong>
  User Training and Support:
 </strong>
 Develop training materials and conduct workshops to educate team members on how to effectively utilize Copilot features, ensuring user adoption and proficiency.
 <br/>
 <br/>
 <strong>
  Monitoring and Optimization:
 </strong>
 Continuously monitor the performance of AI applications, analyze usage data, and identify opportunities for improvement to enhance functionality and user experience.
 <br/>
 <br/>
 <strong>
  Collaboration:
 </strong>
 Work closely with IT and security teams to address any potential risks associated with AI deployment and implement necessary controls.
 <br/>
 <br/>
 <strong>
  Qualifications
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Data Science, or a related field.
  </li>
  <li>
   Proven experience working with Microsoft 365 Copilot, including configuration and feature integration.
  </li>
  <li>
   Strong understanding of the M365 ecosystem and its applications (e.g., Word, Excel, Teams, SharePoint).
  </li>
  <li>
   Knowledge of AI technologies and frameworks, with a focus on practical applications in a business context.
  </li>
  <li>
   Familiarity with security protocols and best practices related to AI and data handling.
  </li>
  <li>
   Excellent problem-solving skills and the ability to work collaboratively in a fast-paced environment.
  </li>
  <li>
   Strong communication skills, with the ability to explain complex technical concepts to non-technical stakeholders.
   <br/>
   <br/>
  </li>
 </ul>
 <strong>
  Preferred Qualifications
  <br/>
  <br/>
 </strong>
 <ul>
  <li>
   Experience in developing and implementing AI solutions in an enterprise environment.
  </li>
  <li>
   Certifications related to Microsoft technologies (e.g., Microsoft Certified: Azure AI Engineer Associate).
  </li>
  <li>
   Familiarity with machine learning frameworks and tools.
   <br/>
   <br/>
  </li>
 </ul>
</div>


AI Engineer - Microsoft 365 Copilot Specialist

Celonis

<div><div><p>We're Celonis, the global leader in Process Intelligence technology and one of the world's fastest-growing SaaS firms. We believe there is a massive opportunity to unlock productivity by placing AI, data and intelligence at the core of business processes - and for that, we need your help. Care to join us?</p></div><p><b>
Role Description</b></p>
<ul>
<li>As an Applied AI Engineer you are pushing the envelope in solving business-critical problems for our customers. You will be working with our most strategic customers, understanding their objectives and key challenges, and building Celonis solutions using the world's leading Process Intelligence (PI) platform in combination with the largest AI and ML technology partners, such as Microsoft, OpenAI and Databricks. With Celonis' Process Intelligence (PI) platform we feed operational context to AI so it understands our customers' business and enables them to industrialize AI unlocking real ROI on AI deployments and at scale. There is no AI without PI. You will prototype these solutions, demonstrate their value to Executives and ensure successful implementation, adoption and value realization in order to increase the footprint of Celonis at those customers.</li>
</ul>
<p><b>Key Responsibilities:</b></p>
<ul>
<li>AI Discovery &amp; Solutioning: Understand customers AI strategy and business critical challenges. As Celonis product &amp; domain expert, find the best problem-solution fit and translate customer requirements into innovative solutions that move the needle</li>
<li>Hackathons &amp; Prototyping: Think out of the box, have a &bdquo;can-do&quot; attitude and don't shy away from complex problems. Leverage cutting edge AI technologies to rapidly build creative prototypes in customer hackathons solving business critical problems</li>
<li>Agentic Process Transformation: Support our customers in achieving real ROI out of AI deployments at scale enabling a fundamental shift in business operations from traditional, rule-based automation to the use of autonomous AI agents empowered by our Celonis Process Intelligence Platform</li>
<li>Proof projects: End-to-end execution of business-critical Proof-of-Value projects, incl. architecture and deliver secure, scalable LLM/agent systems with RAG, tools, and guardrails; integrating with enterprise data, identity, and compliance frameworks.</li>
<li>Ensure Successful Project Outcome: Applied AI Engineers stay involved with projects until agreed value &amp; adoption thresholds are reached</li>
<li>Specialize in Domains or Industries: To scale knowledge across the organization Applied AI Engineers specialize in domains (e.g. supply chain) and industries</li>
</ul>
<p><b>Requirements:</b></p>
<ul>
<li><b>5+ years of experience leading technical pre-sales</b><b>, including defining AI roadmaps, building compelling ROI/TCO business cases and prototyping of machine learning and generative AI solutions.</b></li>
<li>Understanding of generative AI techniques like RAG, few shot learning, prompt engineering, multi-agent orchestration, multimodal understanding, or fine-tuning that are used to build high-impact use cases like intelligent chatbots and automated text processors.</li>
<li>Understanding of business processes across sectors (such as Supply Chain or Finance) with the ability to translate high-level business needs into specific AI use cases.</li>
<li>Good knowledge of Python and common ML libraries (such as LangChain, pandas, pydantic, sklearn, PyTorch) as well as data engineering tools and technologies.</li>
<li>Strong presentation skills to both internal and external stakeholders (including executives), whether whiteboarding sessions or formal readouts and demos.</li>
<li>Bachelor's Degree required, Masters Degree in computer science, engineering, mathematics or related fields, or equivalent work experience preferred.</li>
</ul>
<p><b>Nice to have (big plus):</b></p>
<ul>
<li>Hands-on experience building agentic systems using LLM orchestration, RAG, function calling, and prompt engineering, while ensuring safety through rigorous evaluations and guardrails.</li>
<li>Working knowledge of tools in the LLM ecosystem such as LangChain, LlamaIndex, or other OSS packages.</li>
<li>Experience in deploying and monitoring models at scale across major cloud platforms (AWS Bedrock, Azure AI, GCP Vertex)</li>
</ul>
<p>Visa sponsorship is not offered for this role.<br>
Learn More about Value Engineering at Celonis Here</p><div><p><b>
What Celonis Can Offer You:</b></p>
<ul>
<li><b>Pioneer Innovation:</b> Work with the leading, award-winning process mining technology, shaping the future of business.</li>
<li><b>Accelerate Your Growth:</b> Benefit from clear career paths, internal mobility, a dedicated learning program, and mentorship opportunities.</li>
<li><b>Receive Exceptional Benefits:</b> Including generous PTO, hybrid working options, company equity (RSUs), comprehensive benefits, extensive parental leave, dedicated volunteer days, and much more. Interns and working students explore your benefits here.</li>
<li><b>Prioritize Your Well-being:</b> Access to resources such as gym subsidies, counseling, and well-being programs.</li>
<li><b>Connect and Belong:</b> Find community and support through dedicated inclusion and belonging programs.</li>
<li><b>Make Meaningful Impact:</b> Be part of a company driven by strong values that guide everything we do: Live for Customer Value, The Best Team Wins, We Own It, and Earth Is Our Future.</li>
<li><b>Collaborate Globally:</b> Join a dynamic, international team of talented individuals.</li>
<li><b>Empowered Environment:</b> Contribute your ideas in an open culture with autonomous teams.</li>
</ul>
<p><b>About Us:</b></p>
<p>
Celonis makes processes work for people, companies and the planet. The Celonis Process Intelligence Platform uses industry-leading process mining and AI technology and augments it with business context to give customers a living digital twin of their business operation. It's system-agnostic and without bias, and provides everyone with a common language for understanding and improving businesses. Celonis enables its customers to continuously realize significant value across the top, bottom, and green line. Celonis is headquartered in Munich, Germany, and New York City, USA, with more than 20 offices worldwide.</p>
<p>
Get familiar with the Celonis Process Intelligence Platform by watching this video.</p><br>
<p><b>
Celonis Inclusion Statement:</b></p>
<p>
At Celonis, we believe our people make us who we are and that &quot;The Best Team Wins&quot;. We know that the best teams are made up of people who bring different perspectives to the table. And when everyone feels included, able to speak up and knows their voice is heard - that's when creativity and innovation happen.</p><br>
<p><b>
Your Privacy:</b></p>
<p>
Any information you submit to Celonis as part of your application will be processed in accordance with Celonis' Accessibility and Candidate Notices</p>
<p>
By submitting this application, you confirm that you agree to the storing and processing of your personal data by Celonis as described in our Privacy Notice for the Application and Hiring Process.</p>
<p>
Please be aware of common job offer scams, impersonators and frauds. Learn more here.</p></div></div>

Senior Applied AI Engineer (Value Engineering) - Life Sciences

Senior Applied AI Engineer (Value Engineering) - Public Sector

uiAgent

<div>
 <p>
  <strong>
   Read carefully before you apply.
  </strong>
 </p>
 <p>
  <br/>
 </p>
 <p>
  We are a pre series a startup building AI Agents for CPAs/Accounting teams. We have significant traction, revenue, and huge upside potential.
 </p>
 <p>
  <br/>
 </p>
 <p>
  Looking to hire only the best of the best. You need to be disciplined, hard-working, and willing to sacrifice weekends for our bigger vision: disrupting an entire industry while helping many people along the way. Over the past 18 months building this product, we've become friends who enjoy each other's company, share dark humor, and find disruptive solutions to interesting coding challenges.
 </p>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   We offer you:
  </strong>
 </p>
 <ul>
  <li>
   Substantial equity
  </li>
  <li>
   Flat organizational structure
  </li>
  <li>
   Competitive salary (140k - 220k)
  </li>
  <li>
   Cash Bonus
  </li>
  <li>
   Free dinners
  </li>
  <li>
   Great equipment
  </li>
  <li>
   Healthcare, dental, vision, commuter benefits, and more!
  </li>
  <li>
   Amazing office in Manhattan with a skyline view
  </li>
 </ul>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   We're looking for:
  </strong>
  &nbsp;
 </p>
 <p>
  Engineers who started coding at a young age, are obsessed with clean code and innovative solutions, and are passionate about algorithms, logic, and mathematics. You'll work on integrating modern AI into an agent platform similar to LangChain with Python, and integrate open-source machine learning models. You'll also handle routine tasks like writing APIs for our Node.js backend. Familiarity with microservices, scalable architecture, and similar concepts is essential.
 </p>
 <p>
  <br/>
 </p>
 <p>
  <strong>
   Your day to day at uiAgent:
  </strong>
 </p>
 <ul>
  <li>
   We work startup hours
  </li>
  <li>
   We have a 0% tolerance for side projects of any kind; we need you to be laser-focused on uiAgent.
  </li>
  <li>
   You'll get a lot of responsibility from day one.
  </li>
  <li>
   We don't allow remote work at all.
  </li>
  <li>
   We are very social with each other, going to dinners together, making a lot of jokes, and trying to enjoy each other's company.
  </li>
  <li>
   Dark humor is welcomed.
  </li>
 </ul>
 <p>
  <br/>
 </p>
 <p>
  Looking forward to building this world-class company together.
 </p>
</div>


BMO Financial Group

151 W 42nd Street New York New York,10036
<br>
<div><p><b>About the Team</b></p><p>
BMO&rsquo;s Applied AI team is responsible for building high‑performing, safe, and reliable AI systems that power real banking experiences. The Evaluations group within Applied AI develops the methods, datasets, and tooling that measure quality, safety, and performance across the full AI lifecycle. Working closely with product, engineering, and research partners, the team ensures evaluation signals are deeply embedded into training loops, deployment workflows, and continuous monitoring processes. This group operates at the intersection of data science, machine learning, and responsible AI, enabling scalable, repeatable, and trustworthy evaluation of advanced AI systems.</p><div></div><p><b>
About the Role</b></p><p>
The AI Evaluation Scientist is an individual contributor role focused on delivering the <b>data science stream</b> of AI evaluations. This includes designing, implementing, and productionizing evaluation methods, metrics, and datasets that directly influence modeling decisions, product quality, and the safety posture of AI systems across the bank. You will work hands‑on with complex models&mdash;particularly LLMs and deep learning systems&mdash;developing rigorous empirical analyses that surface model weaknesses, performance trends, and risk signals.</p><p>
In this role, you will translate evaluation standards into robust, maintainable evaluation code and workflows. You will collaborate with engineers to integrate evaluation signals into CI/CD and training pipelines, and work with product and research partners to ensure evaluation insights meaningfully shape model improvements. This position is highly technical, experimental, and delivery‑oriented, with a strong emphasis on applied data science, reproducible experimentation, and responsible AI practices.</p><div></div><p><b>
Key Responsibilities</b></p><ul><li>
Design and implement advanced evaluation methods for LLMs and ML systems, including robustness, reliability, fairness, explainability, calibration, and safety‑and-performance-focused metrics.</li><li>
Build and maintain high‑quality evaluation datasets, golden sets, challenge sets, and red‑teaming corpora tailored to real banking workflows.</li><li>
Develop reusable evaluation harnesses and pipelines that support multi‑agent workflows, tool use, and retrieval‑augmented generation scenarios.</li><li>
Conduct empirical analyses, including statistical tests, error analysis, and ablation studies, to identify model weaknesses and guide model and product improvements.</li><li>
Integrate evaluation metrics and signals into model training loops, deployment gating checks, and continuous monitoring processes.</li><li>
Prototype and validate novel evaluation algorithms inspired by current research in LLM safety, interpretability, and reliability, and convert prototypes into maintainable components.</li><li>
Produce clear, actionable evaluation reports that translate technical findings into insights for engineering, modeling, product, and business stakeholders.</li><li>
Collaborate with engineering, research, and product teams to align evaluation requirements and deliver production‑ready evaluation capabilities.</li><li>
Ensure reproducibility and reliability of evaluation results through dataset versioning, configuration control, testing practices, and documentation.</li></ul><div></div><p><b>
Qualifications</b></p><ul><li>
7+ years of experience in data science, machine learning, or AI development, with at least 3 years focused on evaluation, safety, reliability, or model performance analysis.</li><li>
Master&rsquo;s or PhD in Computer Science, Data Science, Statistics, Engineering, or a related quantitative field, or equivalent practical experience.</li><li>
Strong proficiency in Python and SQL, with experience using PyTorch or TensorFlow, scikit‑learn, and modern data science libraries.</li><li>
Demonstrated experience building evaluation pipelines for LLMs or ML systems, including metric implementation, dataset creation, and CI/CD integration.</li><li>
Solid understanding of statistical testing, calibration, sampling design, and error analysis.</li><li>
Experience with evaluation of RAG systems, tool‑use workflows, long‑context scenarios, adversarial/jailbreak attacks, toxicity/bias detection, or privacy/PII leakage tests.</li><li>
Familiarity with MLOps/LLMOps practices, including experiment tracking, artifact management, and cloud‑based ML infrastructure.</li><li>
Strong communication skills with the ability to translate complex evaluation findings for both technical and non‑technical audiences.</li><li>
Experience with interpretability or fairness techniques (e.g., SHAP, counterfactuals, model probing) is an asset.</li><li>
Contributions to research or open‑source projects in evaluation, safety, reliability, or interpretability are an asset.</li></ul><p></p>
<p><b>Salary:</b></p><p>
$122,400.00 - $228,000.00</p><p><b>
Pay Type:</b></p><p>
Salaried</p><p>
The above represents BMO Financial Group&rsquo;s pay range and type.</p><p>
Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Group&rsquo;s expected target for the first year in this position.</p><p>
BMO Financial Group&rsquo;s total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards. BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans. To view more details of our benefits, please visit: https://jobs.bmo.com/global/en/Total-Rewards</p>
<p><b>About Us</b></p><p>
At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world.</p><p>
As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one &ndash; for yourself and our customers. We&rsquo;ll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we&rsquo;ll help you gain valuable experience, and broaden your skillset.</p><p>
To find out more visit us at https://jobs.bmo.com/us/en
</p><p>BMO is proud to be an equal employment opportunity employer. We evaluate applicants without regard to race, religion, color, national origin, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, gender expression, transgender status, sexual stereotypes, age, status as a protected veteran, status as an individual with a disability, or any other legally protected characteristics. We also consider applicants with criminal histories, consistent with applicable federal, state and local law.</p><p>
BMO is committed to working with and providing reasonable accommodations to individuals with disabilities. If you need a reasonable accommodation because of a disability for any part of the employment process, please send an e-mail to BMOCareers.Support@bmo.com and let us know the nature of your request and your contact information.</p><p>
Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO, directly or indirectly, will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid, written and fully executed agency agreement contract for service to submit resumes.</p></div>

AI Evaluations Engineer

Explore 399+ Ai engineer in New York this March. Find roles across top companies and apply now to boost your career.