Member of Technical Staff - Distributed Training Engineer

About the role

Liquid‑AI is building general‑purpose AI systems that run efficiently across data‑center accelerators and on‑device hardware. As a Member of Technical Staff – Distributed Training Engineer you will join the Training Infrastructure team to design, implement, and optimise the distributed systems that power next‑generation foundation models.

Key responsibilities

• Design and build core systems that make large‑scale training runs fast and reliable.
• Develop scalable distributed training infrastructure for GPU clusters.
• Implement and tune parallelism and sharding strategies for evolving model architectures.
• Optimise distributed efficiency through topology‑aware collectives, communication/computation overlap, and straggler mitigation.
• Build data‑loading pipelines that eliminate I/O bottlenecks for multimodal datasets.
• Create checkpointing mechanisms that balance memory constraints with recovery needs.
• Develop monitoring, profiling, and debugging tools to improve training stability and performance.

Required profile

• Loves the complexity of distributed systems and enjoys debugging long training runs across GPU clusters.
• Thrives in ambiguous environments where model architectures evolve rapidly.
• Aligns with team priorities, pushes back with data‑driven arguments, and delivers high‑ownership solutions.
• Has hands‑on experience building distributed training infrastructure and diagnosing performance bottlenecks.

Required skills

• Experience with PyTorch Distributed (DDP/FSDP), DeepSpeed ZeRO, Megatron‑LM tensor/pipeline parallelism.
• Proficiency in profiling and resolving NCCL/collective issues, out‑of‑memory errors, and stragglers.
• Understanding of hardware accelerators, GPU clusters, and networking topologies.
• Ability to optimise data pipelines for machine‑learning workloads.