Senior Serverless Spark Migration Engineer

About the role

We are looking for a Senior Serverless Spark Migration Engineer to lead the modernization of a large‑scale enterprise data platform. You will drive the migration of on‑premise Apache Spark workloads to serverless, cloud‑native architectures on AWS and GCP.

Key responsibilities

• Lead migrations of enterprise Spark workloads from on‑premise Hadoop/Spark environments to AWS and GCP.
• Assess applications, clusters, configurations, dependencies, data flows and resource utilization.
• Design and implement serverless architectures using AWS EMR Serverless, S3, Glue, Lake Formation, GCP Dataproc Serverless, GCS and BigQuery.
• Refactor legacy PySpark/Scala/Spark SQL code for cloud portability, scalability and reliability.
• Benchmark performance, optimise workloads for cost and reliability, and build reusable migration tooling and automation.

Required profile

• 8+ years of experience in data engineering, distributed systems, cloud or platform engineering.
• 5+ years of hands‑on Apache Spark experience in enterprise environments.
• Proven experience migrating large‑scale Spark workloads across infrastructure platforms.
• Hands‑on experience with both AWS and GCP cloud services.

Required skills

• Apache Spark, PySpark, Scala, SQL.
• AWS services: EMR Serverless, S3, Glue, Lake Formation.
• GCP services: Dataproc Serverless, GCS, BigQuery.
• Hadoop ecosystem: HDFS, YARN, Hive.
• Infrastructure as Code and CI/CD tools: Terraform, Git.
• Container technologies: Docker, Kubernetes (nice to have).