Databricks Engineer

About the role

The Databricks Engineer will design, build, and operate a Data & AI platform based on the Medallion Architecture, enabling seamless data flow across enterprise systems such as PeopleSoft, D2L, and Salesforce.

Key responsibilities

• Design, implement, and optimize end‑to‑end data pipelines on Databricks, creating bronze, silver, and gold data layers.
• Develop robust ETL/ELT pipelines using Apache Spark and Delta Lake, and operationalize workflows with Databricks Jobs.
• Integrate data from PeopleSoft, D2L, Salesforce via APIs, JDBC, or other connectors, handling structured and semi‑structured data.
• Implement data quality checks, monitoring (Grafana, Databricks metrics), and governance using Unity Catalog.
• Support AI/ML initiatives by delivering feature‑rich datasets and managing MLOps with MLflow.

Required profile

• Hands‑on experience with Databricks, Delta Lake, and large‑scale Spark engineering.
• Proven ability to build Medallion Architecture pipelines and manage data versioning.
• Strong SQL and programming skills in Python or Scala.
• Experience integrating enterprise platforms such as PeopleSoft, Salesforce, or D2L.

Required skills

• Databricks
• Delta Lake
• Apache Spark
• SQL
• Python
• Scala
• PeopleSoft
• Salesforce
• D2L
• Unity Catalog
• MLflow
• Azure Data Lake Storage or Amazon S3
• Grafana