Research Engineer – Model Evaluations

About the role

We are looking for a Research Engineer to design, implement, and scale evaluations of Anthropic’s Claude models. You will turn ambiguous notions of intelligence into clear, defensible metrics that guide research, product, and public understanding.

Key responsibilities

• Design and run new evaluations of Claude’s reasoning, agentic behavior, knowledge, and safety properties, and create visualizations for researchers and decision‑makers.
• Build and harden a distributed evaluation execution platform that reliably runs hundreds of evals against live training checkpoints.
• Own dashboards that monitor model health, improve signal‑to‑noise, reduce latency, and make regressions impossible to miss.
• Debug anomalous evaluation results during training runs, determine root causes, and communicate findings under time pressure.
• Improve tooling, libraries, and workflows used by researchers to implement and iterate on evaluations.

Required profile

• Strong Python programming experience, especially in production or research infrastructure.
• Experience building or operating reliable distributed systems, data pipelines, or similar large‑scale infrastructure.
• Clear written and verbal communication skills for explaining technical results to non‑specialists.
• Willingness to be on‑call or provide production support during live training runs.
• Interest in the societal impacts of AI and a commitment to safe, beneficial systems.

Required skills

• Python
• Distributed systems and data pipeline development
• Production infrastructure and on‑call support
• Large language model prompting, sampling, and scaffolding
• Data visualization and dashboard creation
• Evaluation metric design for language models
• Observability, monitoring, and experiment‑tracking
• Statistics and experimental design
• Large‑scale dataset sourcing, curation, and processing
• ML training infrastructure

What we offer

• Annual salary range $500,000 – $850,000 USD.
• Competitive compensation and benefits, including optional equity donation matching, generous vacation and parental leave.