Senior SDET - AI Cluster

About the role

Cerebras Systems builds the world's largest AI chip, which delivers training and inference speeds over 10 times faster than GPU-based cloud services. In its AI infrastructure organization, the company focuses on simplifying large hardware deployments with observability and monitoring and on software that provides built-in resiliency. As a Senior Software Development Engineer in Test (SDET) in the AI Cluster, you will help test and validate thousands of nodes in large deployments so the cluster stays 99.999% reliable.

Key responsibilities

• Innovate and execute tests on cutting-edge AI infrastructure, defining optimized test strategies and methodologies.
• Aim for 100% automated tests covering high availability, failure scenarios, performance, stress and security.
• Test cluster software such as Kubernetes, Prometheus and Grafana, and hardware components such as ML wafer-scale accelerators, CPU runtime nodes, the SwarmX interconnect and the memoryX interconnect.
• Champion cluster security, reliability for 99.9999% uptime and ease of use with observability.

Required profile

• Bachelor's or master's degree in computer science, electrical engineering, AI, data science or a related field.
• 5+ years of experience testing enterprise software, distributed systems, or datacenter hardware and software.
• Deep understanding of large-scale distributed ML training and inference, and ability to break large systems into unit-testable components.
• Experience debugging large distributed systems, hardware and software, using tools such as pdb, gdb, strace and network monitors.
• Understanding of datacenter layout, server, memory, BIOS, PCIe, networking and storage characteristics.

Required skills

• Strong coding in Python, Go or C/C++.
• Cloud technologies such as AWS, Kubernetes and Docker.
• Monitoring tools such as Grafana and Prometheus (huge plus).
• ML model training and inference, ML hardware accelerators (GPU, ASIC) and debugging skills (huge plus).