Data Engineer with Databricks

About the role

Codest Ltd. is seeking an experienced Data Engineer to work on a banking application for a leading Japanese financial group. The role focuses on designing, building, and managing scalable data solutions using Databricks and related technologies.

Key responsibilities

• Implement Medallion Architecture (Bronze, Silver, Gold) for lakehouses with performance optimization.
• Build data quality frameworks with automated testing and monitoring.
• Develop and maintain complex ETL/ELT pipelines processing large‑scale datasets daily.
• Design CI/CD workflows for data pipelines using Databricks Asset Bundles or equivalent tools.
• Create real‑time and batch processing solutions with Structured Streaming and Delta Live Tables.
• Optimize Spark jobs for cost efficiency, leveraging auto‑scaling and resource management.
• Develop custom integrations with Databricks APIs and external systems.
• Design scalable data architectures using Unity Catalog, Delta Lake, and Apache Spark.
• Establish data mesh architectures with governance and lineage tracking.

Required profile

• 6+ years of experience in data engineering, delivering production‑grade pipelines and ETL/ELT workflows.
• Proficiency in Python, PySpark, and SQL.
• Extensive hands‑on experience with AWS services such as Glue, Lambda, Redshift, and S3.
• Strong background in data governance tools like Unity Catalog and AWS SageMaker Unified Studio.
• Proven experience in data migration and modernising legacy systems.
• Fluent English communication skills for an international team.

Required skills

• Python
• PySpark
• SQL
• AWS Glue
• AWS Lambda
• AWS Redshift
• AWS S3
• Unity Catalog
• AWS SageMaker Unified Studio
• Databricks
• Delta Lake
• Structured Streaming
• Delta Live Tables
• Apache Spark
• Docker (nice to have)
• Kubernetes (nice to have)