Staff Slurm Cluster & HPC Engineer

About the role

Bitdeer is looking for a Staff Slurm Cluster & HPC Engineer to own and operate Slurm as a productized scheduling layer across its global GPU fleet. The role is hands‑on, customer‑facing and sets the engineering standards for the platform team.

Key responsibilities

• Design, deploy and operate production Slurm clusters on bare‑metal and VM GPU nodes, ensuring high availability and seamless version upgrades.
• Implement topology‑aware scheduling for GPU fabrics, model physical fabrics in topology.conf and validate placement quality with NCCL bandwidth tests.
• Define and enforce multi‑tenant scheduling policies, including partitions, QOS, fairshare, preemption and per‑tenant resource limits.
• Lead the integration of the Slinky slurm‑operator and slurm‑bridge to enable co‑scheduling of Kubernetes workloads on the same GPU pool.
• Automate cluster provisioning and lifecycle using Terraform, Ansible, PXE, Redfish and IPMI.
• Build health‑check and reliability systems with prolog/epilog checks, DCGM diagnostics and automatic job re‑queue on failures.
• Instrument observability, accounting and billing pipelines via Prometheus/Grafana exporters.

Required profile

• 8+ years in HPC, systems or cloud infrastructure engineering, with 4+ years operating production Slurm clusters at 100+ GPU‑node scale.
• Deep hands‑on expertise in Slurm configuration, authentication, accounting and live upgrades.
• Strong knowledge of GPU drivers, NVIDIA Fabric Manager, InfiniBand/RoCE, GPUDirect RDMA and NCCL tuning.
• Experience with Kubernetes operators, especially Slinky or comparable Slurm‑on‑Kubernetes stacks.
• Proficiency in Python and Bash for automation; Go is a plus.

Required skills

• Slurm (slurm.conf, gres.conf, topology.conf, slurmdbd, slurmrestd)
• Kubernetes operator pattern, Slinky slurm‑operator, slurm‑bridge
• Terraform, Ansible, PXE, Redfish, IPMI provisioning
• Python, Bash (Go optional)
• NVIDIA drivers, Fabric Manager, DCGM, MIG
• InfiniBand/RoCE, NVLink, NCCL tuning
• Parallel storage systems (Lustre, GPFS, WEKA, NFS)