Data Engineer

About the role

The Data Engineer will join the Data Analytics Unit within the Policy & Community Affairs Division, building and maintaining the data infrastructure that powers public‑policy analysis for New York City’s for‑hire transportation ecosystem.

Key responsibilities

• Design, develop, and maintain scalable ETL pipelines and data models (raw, silver, gold tables) using Databricks, Delta Lake, and Azure.
• Inspect and process billions of trip records, GPS traces, and session data to ensure data quality and reliability.
• Document data schemas, naming conventions, and lineage to guarantee reproducibility and trustworthiness.
• Collaborate with analysts and scientists to resolve data quality issues, often by engaging data providers.
• Implement version‑controlled, modular code and thorough documentation.

Required profile

• Master’s degree and at least 3 years of relevant experience.
• Strong analytical curiosity and ability to spot and investigate data inconsistencies.
• Comfort with ambiguous requests and capability to deliver progress without a perfect specification.
• US work authorization; no visa sponsorship.

Required skills

• Python (clean, readable code)
• SQL
• Databricks
• Delta Lake
• Microsoft Azure
• Git or other version‑control systems

What we offer

• Hybrid work model with up to 2 days per week remote.
• Opportunity to influence city‑wide transportation policy through data.