Staff AI Observability & Telemetry Engineer

About the role

We are seeking a Staff AI Observability & Telemetry Engineer to architect the nervous system of our AI‑native NeoCloud platform. You will build a high‑fidelity perception layer that captures, stores, and interprets millions of hardware and software signals per second to enable deep insight into GPU workloads and network fabric.

Key responsibilities

• Architect and scale high‑cardinality telemetry infrastructure using highly available time‑series databases such as VictoriaMetrics, Thanos, or Mimir.
• Integrate hardware exporters (NVIDIA DCGM, network switch telemetry, IPMI/Redfish) into the Kubernetes observability stack.
• Build eBPF‑based diagnostic tools for tracing network congestion, kernel I/O latency, and distributed training bottlenecks.
• Develop automated dashboards and alerting pipelines to proactively cordon degraded hardware.
• Design metric pipelines for multi‑tenant consumption billing based on real‑time GPU and network utilization.
• Collaborate with GPU Systems and Scheduling teams to define observability standards for AI‑native workloads.
• Lead technical design reviews and mentor team members on high‑performance telemetry collection.

Required profile

• Bachelor’s or Master’s degree in Computer Science, Electrical Engineering, or related field.
• 6+ years of software or SRE experience with deep hands‑on expertise in the Prometheus/OpenTelemetry ecosystem.
• Proven track record operating, debugging, and scaling large‑scale telemetry stacks in HPC or cloud environments.
• Strong technical leadership and communication skills.

Required skills

• Go programming language.
• Kubernetes metric exporters and operators.
• eBPF and BCC for kernel‑level tracing.
• Linux performance tuning.
• AI hardware metrics (GPU power states, SM utilization, memory bandwidth).
• High‑performance network telemetry.
• Time‑series databases (VictoriaMetrics, Thanos, Mimir).