Lead Engineer, AI Platform

About the role

Circle builds an AI-powered, all-in-one platform for digital businesses, and its AI Quality engineering team owns the foundation for measuring, diagnosing and improving the quality of AI-powered features. The team builds the infrastructure to measure, diagnose and improve production AI systems, rather than doing ML research or model training. Circle is looking for a Lead Engineer to build evaluation frameworks, observability tooling and diagnostic infrastructure for its AI Agents, and to lead and manage the AI Quality engineering team. The role is hands-on and player-coach.

Key responsibilities

• Build and own the evaluation infrastructure: design CI/CD pipelines, scorers and datasets that show whether AI agents (planners, tool-callers and sub-agents) work, from a single tool call to a full multi-turn conversation.
• Diagnose where quality breaks down, tracing failures across the agent pipeline (plan creation vs. execution, tool selection, tool trajectories in workflows, site builder and analytics).
• Turn findings into prototypes that solve issues or set priorities for AI core engineering.
• Grow the datasets used for evaluation, including annotation workflows and AI-generated and simulated conversations.
• Run structured experiments across prompts, models and the agent harness, and evaluate new and open-source models against the production baseline.
• Build the framework for deciding when to shift models, and pursue cost and latency improvements through model swaps, caching and routing by plan complexity.
• Lead and grow the AI Quality engineering team, setting technical direction and managing day-to-day priorities while staying hands-on in the code.
• Partner closely with AI Core engineering so engineers know their changes actually improve quality.

Required profile

• 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product.
• Experience with complex, tool-using systems (multiple tools, planning or orchestration, sub-agents), not just simple single-turn assistants.
• Experience building evaluation or observability infrastructure for ML/AI systems, such as eval pipelines, scorers, dashboards or CI/CD for evals.
• Experience designing datasets, annotation workflows or labeling pipelines for ML/AI evaluation.
• Comfortable learning fast, running many experiments and working in a fast-paced, ambiguous environment.
• Proficiency in English at CEFR Level C2 / ILR Level 5.

Required skills

• Ruby on Rails (production system) and Python (strong plus, especially for data and evaluation).
• Evaluation frameworks such as Braintrust, LangSmith or similar.

What we offer

• Compensation benchmarked to the U.S., equity with ongoing refresh grants, and 35 days of paid time off per year.
• Remote-only work with two company retreats a year.
• A benefits package supporting health, wellbeing and professional growth.