Senior AI Engineer – Inference & Agent Systems

About the role

Arcana Analytics is building real‑time AI agents that synthesize information from heterogeneous sources and deliver structured, reasoned answers. As a Senior AI Engineer you will own the performance, reliability, and correctness of the inference and multi‑step agent pipelines that power this product.

Key responsibilities

• Drive time‑to‑first‑token (TTFT) below 400 ms for multi‑step agent pipelines, including streaming first‑token delivery while sub‑agents continue processing.
• Design and implement KV‑cache strategies, prompt compression, and dynamic context‑window management.
• Implement multi‑provider routing that selects models (OpenAI, Anthropic, Gemini, open‑weight) based on latency, cost, and task type.
• Build Plan‑Execute‑Synthesize DAG pipelines that run sub‑agents in parallel, using Temporal for reliable orchestration (retries, timeouts, idempotency).
• Enforce structured JSON output with schema validation, retry loops, and graceful degradation for malformed LLM responses.
• Own the end‑to‑end evaluation framework: ground‑truth datasets, automated scoring, regression detection, LLM‑as‑judge pipelines, and latency regression testing (p50/p95/p99).
• Develop model‑serving and cold‑start optimizations, async worker architecture for parallel execution, and full observability of token flow, tool calls, and synthesis steps.

Required profile

• Proven experience shipping inference pipelines at production scale with TTFT as a primary metric.
• Hands‑on experience building multi‑step agent systems and troubleshooting their failures in production.
• Track record creating evaluation harnesses and ground‑truth datasets that drive meaningful product improvements.
• Deep understanding of LLM non‑determinism and techniques to make systems resilient to it.
• Experience with streaming LLM responses and handling partial outputs.

Required skills

• Go programming language
• Temporal workflow orchestration
• OpenAI, Anthropic, Gemini APIs (and open‑weight model integration)
• LLM prompt engineering, KV‑cache, prompt compression
• JSON schema validation
• Streaming response handling
• Model serving and cold‑start optimization
• Async worker architecture
• Observability and tracing of token‑level events