AI Benchmark Engineer – Research & Knowledge

About the role

Turing is seeking a highly analytical AI Benchmark Engineer to design and evaluate multi‑agent benchmark tasks for frontier AI research. This remote contractor role blends deep research expertise with practical engineering to create rigorous evaluation datasets and tools.

Key responsibilities

• Build multi‑agent benchmark tasks that require reading, analyzing, and synthesizing large document collections.
• Curate real‑world research corpora—including academic papers, case studies, and technical reports—and design comprehensive analysis questions.
• Write structured ground‑truth oracles in JSON with verifiable answers to ensure agents truly read source material.
• Design LLM judge prompts that evaluate agent outputs field‑by‑field against the oracle.
• Create decomposition guides that split research across parallel sub‑agents for document‑level and domain‑level processing, followed by synthesis.

Required profile

• 5+ years of research experience (academic or industry) in any scientific domain.
• Strong reading comprehension with ability to extract structured data from unstructured text.
• High attention to detail for precise evaluation oracle creation.

Required skills

• Proficiency in Python scripting for data processing and evaluation.
• Strong experience with JSON schema design and output validation.
• Hands‑on experience writing Dockerfiles, building images, and debugging containers.

What we offer

• Fully remote work environment.
• Opportunity to contribute to cutting‑edge AI projects with leading LLM companies.
• Potential contract extension based on performance and project needs.