Principal AI Engineer - Agentic Systems and Evaluation Platform
amgen
Job description
About the role
Amgen is looking for a Principal AI Engineer to own the architecture, engineering and ongoing operation of evaluations as a shared component of its enterprise AI platform. The goal is to build and manage the services, integrations and infrastructure that let teams evaluate agentic AI systems consistently, securely and at scale, with self-service interfaces and reusable framework integrations. The role leads a small engineering team of about 2-3 engineers and combines platform engineering with a deep understanding of how agentic systems are evaluated.
Key responsibilities
- Own the evaluation platform from architecture and delivery through adoption, operations and continuous improvement, defining its roadmap, service boundaries, integration contracts and engineering standards.
- Build self-service APIs, SDKs, developer integrations and user interfaces for configuring evaluations, submitting and tracking runs and accessing results.
- Create an extensible integration layer for evaluation frameworks, scorers, model endpoints and agent runtimes.
- Engineer reliable evaluation execution services with asynchronous orchestration, scheduling, concurrency management, retries, cancellation and workload isolation.
- Provide versioning and lifecycle management for datasets, configurations, evaluator definitions and result artifacts, with metadata and lineage.
- Integrate evaluation execution and results with CI/CD, experiment tracking, observability and promotion workflows, with configurable quality thresholds and evidence collection.
- Deliver consistent result schemas, comparison views, scorecards and reporting APIs.
- Embed authentication, authorization, project isolation, auditability and data retention controls, working with security and governance partners.
- Operate the component as a production service, monitor reliability and performance, manage capacity and cost, and lead incident resolution and upgrades.
- Drive adoption through documentation, onboarding patterns, reference integrations and technical support.
Required profile
- Bachelor's or master's degree in computer science, AI or a related field.
- 10+ years of experience in software engineering, AI engineering, ML systems or platform engineering.
- Experience building and operating shared platform services used by multiple applications or engineering teams.
- Hands-on experience integrating AI evaluation frameworks, experiment tracking or automated AI quality workflows into production engineering environments.
- Ability to define service contracts, make architecture tradeoffs and lead engineering delivery and operations across teams.
Required skills
- Strong Python and software engineering skills, including production APIs, reusable libraries and distributed backend services.
- Understanding of LLM and agentic architectures, including orchestration, RAG, tool use and state management.
- Experience with asynchronous processing, cloud services, CI/CD, observability, versioning and secure access to enterprise systems.
- Preferred: experience with MLflow, RAGAS, DeepEval or Promptfoo, plugin architectures, workload isolation, and enterprise security and governance controls.
Questions fréquentes
Why are you reporting this job?
Explore further
Salaries, guides and searches in the United States.
Salaries by job title
- Delivery Specialist 330
- Line Cook 264
- Restaurant Host 251
- Dishwasher 242
- Shift Supervisor 226
- Parts Specialist 208
- Retail Service Specialist 176
- Server - Red Robin 140
Apply in 30 seconds
Enter your email to apply. An account will be created automatically.
By continuing, you accept our terms of use.
Already have an account? Login
A question about this job?
Ask it here: you will get the full job summary by e-mail, right away.
Published 1 week ago
Expires 1 month from now
5 views · 0 interested
Boost your chances
Upload your CV — we will match you with relevant openings.
Analyzing your CV...
amgen