Pre-training Research Engineer

About the role

As a Pre‑training Research Engineer, you’ll focus on model implementation, scaling, and improving the quality of byte‑native and multimodal foundation models. You’ll build and iterate quickly on research ideas, contribute production‑grade training code and infrastructure, and help deliver high‑quality base models that can serve real‑world use cases at scale.

Key responsibilities

• Train large byte‑native and multimodal foundation models across massive, heterogeneous corpora.
• Implement and evaluate new model architectures, training objectives, and optimization methods.
• Develop stable pre‑training recipes and run scaling experiments for novel architectures.
• Conduct ablations and analyze training dynamics, model behavior, and base‑model quality.
• Work with data and distributed training engineers to improve training efficiency, reliability, and scalability.

Required profile

• 5+ years of experience in machine learning research or engineering, with a proven track record of developing and pre‑training large language or multimodal foundation models.
• Strong software engineering skills with the ability to write robust and performant training code.
• Solid understanding of deep learning fundamentals and modern pre‑training methods and literature.
• Ability to quickly implement research ideas and evaluate them using clear baselines, ablations, metrics, and analysis.
• Hands‑on experience running training workloads in GPU‑based environments and familiarity with distributed training.
• MS in Computer Science, Machine Learning, Artificial Intelligence, Mathematics, or a related field.

Required skills

• JAX
• Flax
• XLA
• FSDP
• ZeRO
• Megatron

What we offer

• Medical, dental, and vision insurance
• 401k plan
• Daily lunch, snacks, and beverages
• Flexible time off
• Competitive salary and equity
• Equal opportunity employment