AI Engineering Manager – Tester

About the role

Mastercard’s Business & Market Insights group is seeking an AI Engineering Manager focused on testing. The role leads AI quality engineering for generative AI, large language models (LLMs) and agentic systems, ensuring they are accurate, safe, reliable and ready for enterprise deployment.

Key responsibilities

• Design and own end‑to‑end LLM evaluation frameworks, including automated prompt regression, output scoring, semantic benchmarking and hallucination detection.
• Build comprehensive test suites for agentic AI systems, covering tool selection, inter‑agent coordination, task decomposition and failure handling.
• Develop retrieval‑augmented generation (RAG) pipeline evaluation using RAGAS, TruLens and DeepEval.
• Lead structured red‑team and adversarial testing, maintaining an evolving adversarial test library.
• Execute fairness, bias and Responsible AI audits, validating explainability, citations and confidence scores.
• Run inference performance benchmarks and enforce LLM quality gates within CI/CD pipelines on Databricks (AWS).
• Implement production monitoring and drift detection using Grafana, Datadog and CloudWatch.

Required profile

• Proven experience leading AI testing or quality engineering initiatives for LLMs or generative AI.
• Strong understanding of AI safety, red‑team tactics and Responsible AI principles.
• Ability to define evaluation metrics, testing roadmaps and documentation standards.

Required skills

• RAGAS
• TruLens
• DeepEval
• Databricks
• AWS
• Grafana
• Datadog
• CloudWatch
• LLMOps