Software Engineer – Agent Dev Velocity

About the role

Join Notion’s Agent Dev Velocity team to build the tooling and evaluation backbone that enables fast, safe AI shipping. You’ll work at the intersection of developer tooling, distributed systems, and measurement, creating scalable evaluation infrastructure for the AI organization.

Key responsibilities

• Design, build, and improve scalable evaluation runners that operate locally, in CI, and on scheduled runs.
• Create reusable eval workspaces, templates, fixtures, and debugging tools to simplify high‑signal evaluations.
• Develop and maintain benchmark and dataset tooling, including curation pipelines, versioning, artifact management, and regression tracking.
• Enhance reliability and observability of eval execution with retries, idempotency, cost/latency visibility, and failure triage.
• Collaborate closely with AI product, platform, and infrastructure teams to embed evaluations into daily shipping workflows.

Required profile

• Strong software engineering fundamentals with experience shipping production systems.
• Proficiency in TypeScript/Node.js and/or Python.
• Experience building reliable distributed systems (queues, retries, idempotency, backfills).
• Comfort working with data pipelines (batch processing, data quality, versioning, reproducibility).
• Practical experience designing measurement or evaluation systems; AI expertise not required but curiosity is essential.

Required skills

• TypeScript
• Node.js
• Python
• Distributed systems concepts (queues, retries, idempotency)
• Data pipeline development
• Benchmarking and evaluation tooling