applied ai engineer

evaluation scenario writer - ai agent testing specialist Jobs in New York United States

quality assurance (qa) tester - appian Jobs in New York United States

quality assurance (qa) tester - appian

account coordinator Jobs in New York United States

account coordinator

actuarial analyst Jobs in New York United States

actuarial analyst

evaluation scenario writer ai agent testing specialist

Mindrift

<div>
 
 
 Please submit your CV in English and indicate your level of English proficiency.
 
 
 
 
 Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.
 
 Participation isproject-based, not permanent employment.
 
 
 
 
 What This Opportunity Involves
 
 
 
 You'll create challenging coding test cases that push AI coding systems to their limits:
 
 
 <ul>
 <li>
 Review and refine realistic coding tasks based on provided production codebases with realistic scope, requirements and information sources
 </li>
 <li>
 Write comprehensive functional tests that validate actual end-to-end behavior and edge-cases, not just superficial checks
 </li>
 <li>
 Craft "fair but hard" challenges where the AI has all the context it needs, but has to work for it (information scattered across files and external sources, complex reasoning required)
 </li>
 <li>
 Analyze AI failures to understand what the model struggles with vs. what it masters
 </li>
 <li>
 Iterate based on feedback from expert QA reviewers who score your work on 7 quality criteria
 
 
 </li>
 </ul>
 
 What We Look For
 
 
 
 This opportunity is a good fit for experienced developers, software engineers, and/or test automation specialists open to part-time, non-permanent projects. Ideally, contributors will have:
 
 
 <ul>
 <li>
 Degree in Computer Science, Software Engineering or related fields
 </li>
 <li>
 5+ years in software development, primarily Python (pytest, async/await, subprocess, file operations)
 </li>
 <li>
 Background in Full-Stack development, with an equal focus on building React-based interfaces and robust Back-end systems
 </li>
 <li>
 Experience writing tests (functional, integration - not just running them)
 </li>
 <li>
 Docker containers (running evaluations locally in containers)
 </li>
 <li>
 CI/CD understanding (GitHub Actions as a user: triggers, labels, reading results)
 </li>
 <li>
 English proficiency - B2
 
 
 </li>
 </ul>
 
 How It Works
 
 
 
 Apply &rarr; Pass qualification(s) &rarr; Join a project &rarr; Complete tasks &rarr; Get paid
 
 
 
 Effort estimate
 
 
 
 Tasks for this project are estimated to take 20 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.
 
 
 
 Payment
 
 
 
 <ul>
 <li>
 Paid contributions, with rates up to $80/hour*
 </li>
 <li>
 Fixed project rate or individual rates, depending on the project
 </li>
 <li>
 Some projects include incentive payments
 </li>
 <li>
 Note: Rates vary based on expertise, skills assessment, location, project needs, and other factors. Higher rates may be offered to highly specialized experts. Lower rates may apply during onboarding or non-core project phases. Payment details are shared per project
 </li>
 </ul>
</div>

Evaluation Scenario Writer - AI Agent Testing Specialist

PermitFlow

<div>PermitFlow is redefining how America builds. We&rsquo;re an applied AI company serving the nation&rsquo;s builders, tackling one of the largest information challenges in the economy: understanding what can be built, where, and how. Our AI agent workforce helps the fastest-growing construction companies navigate everything from permitting and licensing to inspections and project closeouts &ndash; accelerating housing, clean-energy, and infrastructure development across the country.
Despite being a $1.6T industry, construction still suffers from massive delays, wasted capital, and lost opportunity. PermitFlow has already delivered unprecedented speed, accuracy, and visibility to over $20B in development, helping contractors reduce compliance time, de-risk projects, and scale with confidence.
As the U.S. enters a new capex supercycle across data centers, factories, housing, and renewables, joining PermitFlow means building the AI infrastructure at the core of every construction project driving the next wave of reindustrialization.
We&rsquo;ve raised over $90M, most recently completing our Series B, from top-tier investors including Accel, Kleiner Perkins, Initialized, Y Combinator, Felicis, and Altos Ventures, with backing from leaders at OpenAI, Google, Procore, ServiceTitan, Zillow, PlanGrid, and Uber.
Our HQ is in New York City with a hybrid schedule (3 in-office days per week). We prefer NYC-based candidates or those open to relocation.<h1 class="jobSectionHeader">
Role Overview</h1>
As an Applied AI Engineer, you will design and deploy the AI agents that power PermitFlow&rsquo;s pre-construction platform. You will own the full lifecycle of intelligent agents&mdash;from model development to orchestration, productionization, and evaluation&mdash;while contributing to core platform engineering.<h1 class="jobSectionHeader">
What You&rsquo;ll Do</h1><ul><li>
Develop and deploy AI agents that automate permitting workflows end-to-end</li><li>
Orchestrate multiple models and tools to create reliable, multi-step AI systems</li><li>
Build the internal tools and infrastructure that power and monitor these agents</li><li>
Design and implement backend and frontend services, APIs, and data pipelines, integrating AI functionality into user-facing features when needed</li><li>
Contribute to system architecture and core platform design with the engineering team</li><li>
Design and maintain a suite of evaluations and benchmarks to measure agent accuracy, reliability, and cost-effectiveness</li><li>
Optimize inference pipelines and backend systems for speed, scalability, and cost</li></ul><h1 class="jobSectionHeader">
What We&rsquo;re Looking For</h1><ul><li>
3+ years in software or ML engineering post-BS/MS</li><li>
Hands-on experience shipping production AI/ML systems or AI-powered products</li><li>
Experience building multi-step ML systems, pipelines, or agentic workflows</li><li>
Strong backend and API development skills, with some frontend experience</li><li>
Track record of building reliable products with attention to quality and detail</li><li>
Startup grit and comfort in fast-moving, evolving environments</li></ul><h1 class="jobSectionHeader">
What We Offer</h1><ul><li>
Competitive salary and meaningful equity in a high-growth company</li><li>
Comprehensive medical, dental, and vision coverage</li><li>
Flexible PTO and paid family leave</li><li>
Home office &amp; equipment stipend</li><li>
Hybrid NYC office culture (3 days in-office/week) with direct access to leadership</li><li>
In-Office Lunch &amp; Dinner Provided</li></ul>
PermitFlow provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, genetics, sexual orientation, gender identity, gender expression, or family status, as protected by applicable law. 
We are committed to a diverse and inclusive workforce and welcome people from all backgrounds, experiences, perspectives, and abilities. All employment decisions are based on merit, qualifications, and business needs.
Compensation Range: $175K - $250K</div>

Applied AI Engineer

Millennium Management

<div>AI Full Stack Engineer - Equities Technology
We are building the next generation of Large Language Model applications driven by Portfolio Manager requirements that deliver immediate value and scale as a core product. We are looking for an AI Full Stack Engineer to lead this work within the Equities Technology AI group. This role will be responsible for designing, expanding, and optimizing the architecture of a strategic, evolving AI platform and its user-facing applications. The ideal candidate is an experienced engineer who enjoys building and owning high-performance, production-grade AI systems end-to-end (UI through backend) from the ground up.
Responsibilities<ul><li>
Understand and translate Portfolio Manager and business problems into robust, production-ready AI solutions.</li><li>
Design, build, test, deploy, and own LLM-based products that solve specific Portfolio Manager workflows and generalize into reusable AI capabilities.</li><li>
Design and implement agentic AI systems that perform multi-step reasoning, planning, and tool execution.</li><li>
Own the end-to-end LLM architecture and lifecycle, including prompt design, model selection, evaluation, deployment, and iteration.</li><li>
Work closely with stakeholders across business and technology organizations to optimize product design and adoption in production environments.</li><li>
Design and build user-facing web applications for AI-powered workflows, including conversational and task-driven UIs used by Portfolio Managers.</li><li>
Implement and own front-end architecture (state management, component design, performance, accessibility) and integrate it with LLM/agent services and data APIs.</li><li>
Partner with design/product to iterate quickly on UX, while maintaining production-grade engineering standards across both client and server.</li></ul>
Qualifications<ul><li>
Master&rsquo;s or Bachelor&rsquo;s degree in a technical field with 5+ years of industry experience.</li><li>
Strong full stack engineering experience across Python and modern TypeScript/JavaScript stacks, including microservices, REST APIs, and web application development.</li><li>
Hands-on experience building and operating production-grade LLM applications.</li><li>
Experience with agentic and LLM frameworks.</li><li>
Strong understanding of agentic patterns including tool use, planning, memory, and reflection.</li><li>
Experience with retrieval-augmented generation, vector databases, and semantic search systems.</li><li>
Experience developing end-to-end asynchronous applications and distributed systems.</li><li>
Experience deploying AI systems in cloud environments such as AWS or GCP.</li><li>
Strong experience building production front ends (e.g., React/Next.js or equivalent), including authentication/authorization integration and secure consumption of backend APIs.</li><li>
Strong interpersonal and communication skills with the ability to operate independently and collaboratively in a fast-paced environment.</li></ul><div></div>
Millennium pays a total compensation package which includes a base salary, discretionary performance bonus, and a comprehensive benefits package. The estimated base salary range for this position is $175,000 to $250,000, which is specific to New York and may change in the future. When finalizing an offer, we take into consideration an individual&rsquo;s experience level and the qualifications they bring to the role to formulate a competitive total compensation package.</div>

AI Full Stack Engineer - Equities Technology

DataAnnotation

<div>
 We are looking for a Freelance Writer to join our team to train AI models. You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the quality of each model.
 
 
 In this role you will need to hold an expert level of writing, editing, and content development. Other related fields include, but are not limited to: Communications and/or Journalism.
 
 
 
 Benefits
 
 
 
 <ul>
 <li>
 This is a full-time or part-time REMOTE position
 </li>
 <li>
 You&rsquo;ll be able to choose which projects you want to work on
 </li>
 <li>
 You can work on your own schedule
 </li>
 <li>
 Projects are paid hourly starting at $20+ USD per hour, with bonuses on high-quality and high-volume work
 
 
 </li>
 </ul>
 
 Responsibilities
 
 
 
 <ul>
 <li>
 Give AI chatbots writing and editing tasks and evaluate their outputs
 </li>
 <li>
 Evaluate the quality produced by AI models for correctness and performance
 
 
 </li>
 </ul>
 
 Qualifications
 
 
 
 <ul>
 <li>
 Fluency in English (native or bilingual level)
 </li>
 <li>
 Detail-oriented
 </li>
 <li>
 Excellent command of grammar, style, and brand voice
 </li>
 <li>
 A current, in progress, or completed Bachelor&rsquo;s degree is preferred but not required
 
 
 </li>
 </ul>
 
 Notes
 
 
 
 Payment is made via PayPal. We will never ask for any money from you. PayPal will handle any currency conversions from USD. Only applicants in the United States will be considered for this role. This is an independent contract position.
 
 
 #generalist
</div>

Freelance Writer

<div>
 We are looking for a Content Writer to join our team to train AI models. You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the quality of each model.
 
 
 In this role you will need to hold an expert level of writing, editing, and content development. Other related fields include, but are not limited to: Communications and/or Journalism.
 
 
 
 Benefits
 
 
 
 <ul>
 <li>
 This is a full-time or part-time REMOTE position
 </li>
 <li>
 You&rsquo;ll be able to choose which projects you want to work on
 </li>
 <li>
 You can work on your own schedule
 </li>
 <li>
 Projects are paid hourly starting at $20+ USD per hour, with bonuses on high-quality and high-volume work
 
 
 </li>
 </ul>
 
 Responsibilities
 
 
 
 <ul>
 <li>
 Give AI chatbots writing and editing tasks and evaluate their outputs
 </li>
 <li>
 Evaluate the quality produced by AI models for correctness and performance
 
 
 </li>
 </ul>
 
 Qualifications
 
 
 
 <ul>
 <li>
 Fluency in English (native or bilingual level)
 </li>
 <li>
 Detail-oriented
 </li>
 <li>
 Excellent command of grammar, style, and brand voice
 </li>
 <li>
 A current, in progress, or completed Bachelor's degree is preferred but not required
 
 
 </li>
 </ul>
 
 Notes
 
 
 
 Payment is made via PayPal. We will never ask for any money from you. PayPal will handle any currency conversions from USD. Only applicants in the United States will be considered for this role. This is an independent contract position.
 
 
 #generalist
</div>

Content Writer

Finch Legal

<div><h2 class="jobSectionHeader">About Finch</h2>
We believe every American household deserves access to counsel in life&rsquo;s biggest moments. At Finch, we&rsquo;re building the infrastructure to make justice radically more accessible. Our modern approach to consumer law automates the admin work and puts clients first, starting with personal injury.
Since launching in April, we&rsquo;ve grown 10x and are now powering pre-litigation for top law firms across the country. We combine expert operators with purpose-built AI to handle intake, claim opening, medical records, police reports, lien management, demands, and everything in between.
We&rsquo;re backed by Sequoia, Redpoint, and the founders &amp; CEOs of generational companies like DoorDash, Ironclad, and Digits. We&rsquo;re rebuilding how the law serves everyday Americans from first principles, and we&rsquo;re hiring exceptional operators to help us scale it nationwide.<h2 class="jobSectionHeader">
This Role</h2>
Legal work is buried in unstructured documents, repetitive workflows, and data that no existing system handles well &mdash; and we're building the AI to fix it. As a Machine Learning Engineer at Finch, you'll own the full lifecycle of AI systems, from prototype to production, working on problems where a single breakthrough can meaningfully change how law firms operate.<h2 class="jobSectionHeader">
What You'll Do</h2>
This is a hands-on applied AI role where you'll build and ship production systems &mdash; not just run experiments. The scope will grow as our product and team do.<ul><li>
Build voice agents, browser agents, OCR pipelines, and LLM-powered workflows that work reliably in production.</li><li>
Design rigorous evaluation frameworks and feedback loops to systematically improve model accuracy and reliability.</li><li>
Own the full ML lifecycle &mdash; model selection, fine-tuning, prompt design, deployment, and monitoring.</li><li>
Collaborate directly with product, ops, and legal experts to make sure the AI is solving the right problems.</li><li>
Track emerging research and tools, and make deliberate calls about when to bring them into our stack.</li></ul><h2 class="jobSectionHeader">
Who We Have in Mind</h2>
This role is for engineers who care about outcomes over algorithms and are just as comfortable in production as they are in a notebook. Here's what that requires.<h3 class="jobSectionHeader">
Must-Haves</h3><ul><li>
3+ years building and deploying production ML systems.</li><li>
Strong Python skills and experience working across the ML stack end-to-end.</li><li>
Hands-on experience with LLMs, prompt engineering, and evaluation design.</li><li>
A track record of shipping observable, maintainable AI systems &mdash; not just prototypes.</li></ul><h3 class="jobSectionHeader">
Nice-to-Haves</h3><ul><li>
Experience with NLP, OCR, speech, or agent frameworks (LangChain, OpenAI APIs, etc.).</li><li>
Prior work at an early-stage startup where you helped define ML infrastructure from scratch.</li><li>
Familiarity with legal tech, document-heavy workflows, or regulated industries.</li></ul><h3 class="jobSectionHeader">
This Role Might Not Be For You If</h3><ul><li>
You prefer research or experimentation over owning systems in production.</li><li>
You do your best work with a well-scoped problem and a stable, established ML platform.</li><li>
You're looking for a remote-first role &mdash; this one is 4 days/week in our NYC office.</li></ul><h2 class="jobSectionHeader">
Compensation</h2>
The expected package for this role includes base salary + commission + equity.<ul><li>
Base salary = $170,000 - $250,000</li></ul><h2 class="jobSectionHeader">
Benefits</h2><ul><li>
100% coverage for health, dental, and vision.</li><li>
401(k) retirement plan.</li><li>
In-office snacks, drinks, and daily team lunches and dinners.</li><li>
Flexible PTO (we trust you to take the time you need).</li></ul><h2 class="jobSectionHeader">
Interview Process</h2><ul><li>
Intro: An initial call to see if there's a mutual fit.</li><li>
Interview: An interview that typically involves a case assignment of sorts.</li><li>
Onsite: A visit to our NYC office to interview, meet the team, and have lunch.</li></ul><h2 class="jobSectionHeader">
At Finch Legal, we believe in practicing what we advocate.</h2>
As a company dedicated to upholding justice and protecting people in the workplace, we are equally committed to fostering a safe, inclusive, and equitable environment within our own walls. We welcome and support individuals from all backgrounds and lived experiences &mdash; regardless of race, ethnicity, gender identity, sexual orientation, religion, disability, or veteran status.
We recognize that diversity strengthens our team, enriches our perspectives, and empowers us to better serve our clients and communities. At Finch Legal, inclusion isn&rsquo;t just a value &mdash; it&rsquo;s a practice.
Compensation Range: $170K - $250K</div>

Machine Learning Engineer

Explore 583+ Evaluation scenario writer ai agent testing specialist in New York United States this March. Find roles across top companies and apply now to boost your career.

583 evaluation scenario writer ai agent testing specialist jobs In New York United States

Evaluation Scenario Writer - AI Agent Testing Specialist

Evaluation Scenario Writer - AI Agent Testing Specialist

Applied AI Engineer

AI Full Stack Engineer - Equities Technology

Freelance Writer

Content Writer

Machine Learning Engineer