Staff GPU Inference SDET

About the role

Cerebras Systems builds a large AI chip designed for fast training and inference, and is looking for a Staff GPU Inference SDET in Sunnyvale. As the founding quality, reliability and validation lead for a new GPU Inference Development team, you will design, build and scale the end-to-end release qualification and automated test ecosystem for the GPU inference stack and rack-scale accelerated compute fleets. You will work closely with engineering leads and cross-functional systems infrastructure teams.

Key responsibilities

• Build GPU release qualification systems: automated test frameworks, regression gates and release pipelines covering API services, model-serving workers, container runtimes, serving engines, driver stacks and firmware.
• Benchmark and stress-test distributed LLM serving frameworks, including prefill vs decode worker performance, continuous batching, prefix caching, KV-cache efficiency and tensor/expert parallelism.
• Build automated workload replay and benchmarking tools to validate GPU performance models, tracking TTFT, ITL, request throughput, P99 tail latency and capacity efficiency.
• Ensure model accuracy, FP16/FP8 and quantization precision stability, determinism and output correctness across software updates, kernel fusions and hardware revisions.
• Engineer chaos engineering and fault-injection suites simulating node failures, network degradation, GPU memory leaks and driver/firmware mismatches, with automated recovery paths.
• Integrate test pipelines with telemetry tools such as Prometheus and Grafana to create repeatable engineering gates and continuous performance monitoring.

Required profile

• 8+ years of software engineering experience as an SDET, Infrastructure Quality Lead or Systems Test Engineer.
• Hands-on experience bringing up, provisioning and validating multi-node GPU clusters (NVIDIA or AMD ecosystem) on public cloud or enterprise data centers.
• Deep understanding of LLM serving engines and distributed runtimes, including prefill vs decode disaggregation, KV-cache management and dynamic batching.
• Proven background in root-cause analysis, stress testing and node failure simulation in distributed systems.

Required skills

• Expert Python for custom test automation frameworks, diagnostics and CI/CD integration.
• Container orchestration (Kubernetes, Slurm, Ray) and cluster interconnects (InfiniBand, RoCE, NCCL).
• Nice to have: AMD (ROCm/HIP) or NVIDIA software stack experience, workload replay tools, MLPerf Inference benchmarks, and low-level kernel profiling tools (PyTorch Profiler, NVTX, ROCm profilers) or C++.