AI Model Replication Project | Remote

Welcome to Project Dravite! Pareto.AI is a human data-collection platform connecting leading AI researchers with trusted industry experts to collaborate on AI alignment, safety, and training projects. By working together, we can better align AI models with human values and develop more helpful, honest, and harmless AI models.

About the Project:

We are partnering with a frontier lab to test their AI model’s ability to replicate paper findings.

The project involves:

• Finding a suitable causal Economics paper with replication data
• Writing prompts asking the AI model to replicate the findings given the research question, data & code book, and information on the context
• Writing rubrics to evaluate the AI model’s performance on the task

We’re testing whether the model can follow a connected chain of empirical decisions (data cleaning, then variable construction, then specification choice, then robustness judgment), where each step is verifiable. The goal is to identify areas where the AI model makes non-trivial errors at one of those steps.

Pilot stage:

We currently need 10 paper replication tasks to calibrate difficulty and workflow with our research team. Since this initial phase only requires a small set of tasks, we’re bringing on a limited number of Economists for now to closely collaborate with. If you don’t hear from us at this stage, please know that we may still reach out in the coming weeks as the project scales.

Compensation:

$100 per hr for initial scoping and pilot phase.

Job Type: Contract
Contract length: 6 months

Pay: Up to $100.00 per hour

Expected hours: 8 per week

Work Location: Remote