Principal Software Engineer, Inference
hpe · Spring
Job description
About the role
HPE's Private Cloud AI organization is seeking a Principal Software Engineer to lead the model runtime within HPE AI Essentials, the inference platform used by enterprises to run large language models on their own hardware, including air-gapped and sovereign environments. The focus is on sustained execution efficiency: low tail latency and high GPU utilization on customer-owned hardware of varying generations. The role is hybrid, with about two days per week in an HPE office.
Key responsibilities
- Define and own the technical direction of LLM serving, including engine integration, continuous batching, KV cache management and reuse, and quantized execution.
- Partner with inference performance teams, accountable for time-to-first-token, inter-token latency, throughput per GPU and P95/P99 tail latency.
- Define distributed inferencing strategy, including disaggregated prefill/decode, tensor and pipeline parallelism, and KV cache offload across GPU, host memory and RDMA-attached storage.
- Evaluate emerging runtimes, quantization schemes, speculative decoding and mixture-of-experts serving, and decide whether each is adopted, developed in-house or declined.
- Define the orchestration layer: model admission, GPU scheduling and partitioning, cache-aware request routing and autoscaling.
- Mentor engineers, lead design and architecture reviews, and present technical direction to business unit and executive audiences.
Required profile
- Minimum 12 years of software engineering experience, including more than 1 year on LLM inference runtimes or production model serving.
- Degree in Computer Science or a related field.
- Production experience with LLM inference engines such as vLLM, SGLang, TensorRT-LLM, TGI or NVIDIA NIM, including modifying engine internals.
- Deep understanding of inference internals: continuous batching, paged attention, KV cache reuse, chunked prefill, quantization and speculative decoding.
- Expert-level knowledge of Kubernetes operators, custom resources, controllers and scheduling.
- Strong Go and Python, and the ability to debug and profile C++/CUDA with tools such as Nsight.
Preferred
- Upstream contributions to vLLM, SGLang, TensorRT-LLM, LLM-D, LMCache or KServe.
- Experience with disaggregated prefill/decode, KV cache offload at scale, RDMA, GPUDirect Storage, InfiniBand or RoCE, MIG and multi-tenant GPU isolation.
- On-premises, air-gapped or regulated enterprise software delivery.
What we offer
- Base salary of USD 165,500 to 313,500 in Colorado, and USD 157,000 to 361,000 in North Carolina and Texas; variable incentives may also be offered.
- Benefits covering health and wellbeing, and professional development programs.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in the United States.
Salary: Senior Software Engineer Based on 42 job offers in the United StatesSalaries by job title
- Delivery Specialist 330
- Line Cook 264
- Restaurant Host 251
- Dishwasher 241
- Shift Supervisor 226
- Parts Specialist 208
- Retail Service Specialist 176
- Server - Red Robin 140
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 7 hours ago
Expires 1 month from now
2 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
hpe
Spring