Member of Technical Staff – Data Infrastructure

About the role

We are building a Large Physics foundation Model to enable causal AI that can predict and influence physical systems. As a Member of Technical Staff on the Data Infrastructure team, you will design and operate the core data platform that ingests, stores, and serves massive scientific datasets for model training.

Key responsibilities

• Design and run petabyte‑scale lakehouse storage, selecting file formats and data layouts for batch and real‑time queries.
• Own the shared compute and orchestration layer (e.g., Spark, Ray, workflow schedulers) that powers ingestion and research pipelines.
• Optimize end‑to‑end data flow from storage to training, ensuring high‑throughput loading up to the tensor boundary.
• Build systems for cataloging, deduplication, lineage, search and reproducibility across the data lifecycle.
• Implement platform‑level quality, monitoring and alerting tooling for data and research teams.
• Scale infrastructure to improve engineering velocity while maintaining reliability.
• Develop and operate critical ingestion pipelines when mission‑critical data is required.

Required profile

• Proven experience building large‑scale data pipelines and distributed compute systems.
• Deep knowledge of modern data ingestion, storage and loading technologies and their performance impact.
• Strong familiarity with cloud infrastructure, data lake architectures and both batch and streaming pipelines.
• Ability to own deliverables end‑to‑end, from requirements gathering to autonomous execution.

Required skills

• Spark
• Ray
• Beam
• Parquet
• Zarr
• Delta Lake
• cloud infrastructure
• data lake architecture
• batch pipeline design
• streaming pipeline design