Member of Technical Staff – Evals Lead

About the role

We’re hiring an Evals Lead to benchmark video and world‑model capabilities with and without our data. The role requires deep understanding of evaluation philosophy, including contamination, leakage, construct validity, and ensuring metrics reflect true capabilities.

Key responsibilities

• Design and run benchmarks for video and world models, both with Build data and without, using a consistent protocol.
• Ensure honest comparisons through held‑out tasks, contamination and leakage checks, and avoid data cooking.
• Build an analytics layer covering model performance, failure patterns, and the impact of dataset scaling on capabilities.
• Collaborate with research and dataset/quality teams so collection and evaluations inform each other.
• Drive consequential evaluations that cause teams to change plans when numbers shift.
• Design processes that improve evaluation quality, repeatability, and scalability as new tasks and regions are added.

Required profile

• You have shipped or driven evals that mattered, influencing training, hiring, product, or data decisions.
• You understand evaluation philosophy well enough to argue about validity, not just plot curves.
• Experience with video, robotics, world models, or multimodal domains is ideal, but the focus is on consequential evaluations.
• You will not mistake a pretty dashboard for an evaluation that can drive decisions.

What we offer

• Competitive pay.
• Medical, dental, and vision packages with generous premium coverage.
• $500 per month credit for waiving medical benefits.
• Housing subsidy of $2k per month for those living within walking distance of the office.
• Relocation support for moves to San Francisco (Financial District) or Shenzhen (Nanshan).
• Wellness benefits covering fitness, mental health, and more.
• Daily lunch and dinner in the office.
• Unlimited compute budget subject to ROI justification.
• Unlimited Codex and Claude credits.