Inference Software Engineer

About the role

Etched builds AI chips hard-coded for individual model architectures, aiming for much higher throughput and lower latency than current GPUs. As an Inference Software Engineer you will help design and implement the high-performance software stack that runs on its Sohu chip, working with firmware and driver teams and with AI model researchers and product teams.

Key responsibilities

• Contribute to the architecture and design of the Sohu host software stack.
• Implement high-performance, modular code across the stack, a mix of Rust, C++ and Python.
• Interface with firmware and drivers teams to deliver a high-performance hardware/software stack.
• Work with AI model researchers and product-facing teams on the Etched serving front-end.
• Build scheduling logic for continuous batching and real-time inference.
• Implement inference-time acceleration techniques such as speculative decoding, tree search and KV cache sharing.
• Implement distributed networking primitives for efficient multi-server inference.

Required profile

• Experience with C++ and Python.
• Familiarity with transformer model architectures and inference serving stacks (vLLM, SGLang, etc.) or experience in distributed inference/training environments.
• Experience working cross-functionally in large software and hardware organizations.

Required skills

• Preferred: experience with Rust.
• Preferred: familiarity with GPU kernels, the CUDA compilation stack and related tools, or other hardware accelerators.
• Preferred: understanding of distributed systems, networking and parallel programming.

What we offer

• Full medical, dental and vision coverage, with 100% of premiums covered.
• Housing subsidy of $2,000 per month for those living within walking distance of the office.
• Daily lunch and dinner provided in the office.
• Relocation support for people moving to Cupertino.