Senior Data Engineer

About the role

We are seeking a Senior Data Engineer to own and evolve our data platform, turning high‑volume provider data into trusted, production‑ready records for our products. You will lead the transition from serial container‑based workflows to a concurrent, event‑driven architecture.

Key responsibilities

• Design and implement multi‑source ingestion pipelines for APIs, webhooks, files and web‑scraping, handling rate limits and failures.
• Orchestrate batch and event‑driven workflows using Airflow, Dagster or Prefect, including scheduled runs, backfills and Pub/Sub‑based processing.
• Store raw payloads in Google Cloud Storage, transform them with dbt, and maintain medallion layers (raw, standardized, curated).
• Ensure data correctness with idempotent upserts, safe retries, and full traceability to source data.
• Define data contracts, build automated validation tests, and monitor freshness and failures.

Required profile

• Proven experience building production‑grade, multi‑source data platforms from scratch to operation.
• Hands‑on expertise with workflow orchestration tools (Airflow preferred) and dbt on GCP.
• Strong SQL and PostgreSQL skills, including data modeling and incremental loading.
• Solid Python programming, especially asynchronous HTTP and concurrent processing.

Required skills

• Python (asyncio, aiohttp, HTTPX)
• SQL, PostgreSQL
• dbt
• Google Cloud Platform (GCS, Pub/Sub)
• Airflow, Dagster, Prefect (or Temporal)
• Web‑scraping tools (Scrapy, Playwright, Selenium, Beautiful Soup)
• Data orchestration concepts (event‑driven, batch, medallion architecture)