Senior ML Systems Engineer, Inference

About the role

Runpod is seeking a Senior ML Systems Engineer to lead LLM inference performance improvements. You will own end‑to‑end serving performance, measuring, diagnosing, and optimizing latency and cost for customers.

Key responsibilities

• Define and build rigorous tooling to measure inference performance metrics such as throughput, time to first token, inter‑token latency, and cost per token.
• Profile and diagnose performance issues across the serving stack, from scheduling and memory management to kernels and interconnect.
• Improve serving efficiency for state‑of‑the‑art models on single‑node and multi‑node GPU deployments.
• Translate findings into production‑ready runtimes, configurations, and defaults for customers.
• Collaborate with product and infrastructure teams to shape inference offerings.
• Stay current with the inference ecosystem and decide on adoption, building, or contributing back.

Required profile

• 5+ years of professional system engineering experience.
• Hands‑on experience with vLLM, SGLang or comparable serving engines at production or benchmark scale.
• Strong software engineering skills in Python and ability to work in large, performance‑critical codebases.
• Solid understanding of LLM inference performance drivers: batching, memory, parallelism, latency vs throughput trade‑offs.
• Experience with modern inference optimizations such as quantization, speculative decoding, or distributed serving.
• Rigorous benchmarking and GPU profiling expertise.

Required skills

• Python
• vLLM
• SGLang (or comparable serving engine)
• CUDA or Triton kernel development
• GPU profiling tools
• Quantization techniques
• Speculative decoding
• Distributed serving

What we offer

• Competitive base salary $150,000–$220,000
• Equity stock options
• Generous medical, dental & vision plans
• Flexible PTO
• Remote‑first work environment
• $1,200 home office and equipment stipend