Senior Inference Software Engineer
hpe · Spring
Job description
About the role
HPE's Private Cloud AI organization is seeking a Senior Software Engineer to build and evolve the model runtime in HPE AI Essentials, the inference platform used by enterprises to run large language models on their own hardware, including air-gapped and sovereign environments. The focus is on execution efficiency: low tail latency and high GPU utilization on customer hardware of varying generations. The role is hybrid, with about two days per week in an HPE office.
Key responsibilities
- Design, implement and own major components of the LLM serving deployment, including engine integration, continuous batching, KV cache management and reuse, and quantized execution.
- Partner with inference teams to improve time-to-first-token, inter-token latency, throughput per GPU and P95/P99 tail latency.
- Build distributed execution capabilities such as disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU memory, host memory and RDMA-attached storage.
- Evaluate emerging runtimes, quantization schemes, speculative decoding and mixture-of-experts serving and recommend adoption.
- Contribute to the orchestration layer, including model admission, GPU scheduling and partitioning, cache-aware request routing and autoscaling.
- Triage and resolve customer issues end-to-end, and review code and mentor team members.
Required profile
- Minimum of 8 years of software engineering experience, including 1-2+ years on LLM inference runtimes or production model serving.
- Degree in Computer Science or a related field.
- Preferred: upstream contributions to vLLM, SGLang, TensorRT-LLM, llm-d, LMCache or KServe, and experience with air-gapped or regulated enterprise software delivery.
Required skills
- Familiarity with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI or NVIDIA NIM, including engine internals.
- Understanding of continuous batching, paged attention, KV cache reuse, quantization and speculative decoding.
- Working knowledge of tensor and pipeline parallelism, NCCL and the GPU memory hierarchy.
- Advanced Kubernetes knowledge, including operators, custom resources, controllers and scheduling.
- Strong Go and Python, and the ability to debug and profile C++/CUDA with tools such as Nsight.
What we offer
- Base salary of USD 144,000-273,000 in Colorado, and USD 137,000-315,000 in North Carolina and Texas; variable incentives may also be offered.
- Benefits covering health and wellbeing, and programs for professional development.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in the United States.
Salary: Senior Software Engineer Based on 42 job offers in the United StatesSalaries by job title
- Delivery Specialist 330
- Line Cook 264
- Restaurant Host 251
- Dishwasher 241
- Shift Supervisor 226
- Parts Specialist 208
- Retail Service Specialist 176
- Server - Red Robin 140
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 6 hours ago
Expires 1 month from now
1 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
hpe
Spring