Senior Software Engineer, AI Evals

About the role

As a Senior Software Engineer on Sentry’s AI/ML team, you will build the evaluation infrastructure that measures the accuracy, reliability and real‑world performance of the company’s AI systems. You will design datasets, benchmarks and test harnesses that turn ambiguous model behavior into measurable signals, enabling the team to ship AI features with confidence.

Key responsibilities

• Design and implement robust evaluation frameworks for accuracy, regressions and edge‑case detection in AI models.
• Create and curate high‑quality datasets, golden test cases and production‑grounded benchmarks.
• Build automated test harnesses and metrics pipelines that continuously evaluate models, prompts and agentic workflows.
• Partner with applied AI engineers and product leaders to define measurable criteria for “good” model behavior.
• Own the full evaluation lifecycle from early experimentation through production monitoring.

Required profile

• Minimum 5 + years of professional software engineering experience with a Bachelor’s degree in Computer Science, Machine Learning or a related field.
• Experience building testing, evaluation or data infrastructure for complex AI/ML systems.
• Comfort writing production‑quality code in Python and TypeScript.
• Familiarity with modern ML evaluation techniques such as offline metrics, online evaluation and regression testing.

Required skills

• Python
• TypeScript
• Machine learning / AI
• Data pipelines and labeling workflows
• Evaluation frameworks and metrics

What we offer

• Competitive base salary ranging from $155,000 to $400,000 per year.
• Equity grants, incentive compensation and comprehensive health insurance.
• Paid time off and other employee benefit programs.