Jobiglo

No results.

Principal AI Engineer - Agentic Systems and Evaluation Platform

amgen

Senior 🇬🇧 English
Python CI/CD Observability RAG MLflow RAGAS DeepEval Promptfoo

Job description

About the role

Amgen is looking for a Principal AI Engineer to own the architecture, engineering and ongoing operation of evaluations as a shared component of its enterprise AI platform. The goal is to build and manage the services, integrations and infrastructure that let teams evaluate agentic AI systems consistently, securely and at scale, with self-service interfaces and reusable framework integrations. The role leads a small engineering team of about 2-3 engineers and combines platform engineering with a deep understanding of how agentic systems are evaluated.

Key responsibilities

  • Own the evaluation platform from architecture and delivery through adoption, operations and continuous improvement, defining its roadmap, service boundaries, integration contracts and engineering standards.
  • Build self-service APIs, SDKs, developer integrations and user interfaces for configuring evaluations, submitting and tracking runs and accessing results.
  • Create an extensible integration layer for evaluation frameworks, scorers, model endpoints and agent runtimes.
  • Engineer reliable evaluation execution services with asynchronous orchestration, scheduling, concurrency management, retries, cancellation and workload isolation.
  • Provide versioning and lifecycle management for datasets, configurations, evaluator definitions and result artifacts, with metadata and lineage.
  • Integrate evaluation execution and results with CI/CD, experiment tracking, observability and promotion workflows, with configurable quality thresholds and evidence collection.
  • Deliver consistent result schemas, comparison views, scorecards and reporting APIs.
  • Embed authentication, authorization, project isolation, auditability and data retention controls, working with security and governance partners.
  • Operate the component as a production service, monitor reliability and performance, manage capacity and cost, and lead incident resolution and upgrades.
  • Drive adoption through documentation, onboarding patterns, reference integrations and technical support.

Required profile

  • Bachelor's or master's degree in computer science, AI or a related field.
  • 10+ years of experience in software engineering, AI engineering, ML systems or platform engineering.
  • Experience building and operating shared platform services used by multiple applications or engineering teams.
  • Hands-on experience integrating AI evaluation frameworks, experiment tracking or automated AI quality workflows into production engineering environments.
  • Ability to define service contracts, make architecture tradeoffs and lead engineering delivery and operations across teams.

Required skills

  • Strong Python and software engineering skills, including production APIs, reusable libraries and distributed backend services.
  • Understanding of LLM and agentic architectures, including orchestration, RAG, tool use and state management.
  • Experience with asynchronous processing, cloud services, CI/CD, observability, versioning and secure access to enterprise systems.
  • Preferred: experience with MLflow, RAGAS, DeepEval or Promptfoo, plugin architectures, workload isolation, and enterprise security and governance controls.

Questions fréquentes

Le salaire n'est pas communiqué publiquement par le recruteur. Vous pouvez postuler et négocier directement avec amgen.
Cliquez sur "Postuler maintenant" en haut de la page. Vous pouvez importer votre CV en 1 clic — Jobiglo extrait automatiquement vos informations et postule pour vous.
Source : ats:workday

Why are you reporting this job?

Thank you for your report. We will review this job.

Explore further

Salaries, guides and searches in the United States.

Apply in 30 seconds

Enter your email to apply. An account will be created automatically.

Apply now →

By continuing, you accept our terms of use.

Already have an account? Login

A question about this job?

Ask it here: you will get the full job summary by e-mail, right away.

Published 1 week ago

Expires 1 month from now

5 views · 0 interested

Boost your chances

Upload your CV — we will match you with relevant openings.

Analyzing your CV...

amgen