Principal Data Engineer

About the role

CodaMetrix is transforming revenue cycle management with an AI‑powered autonomous coding platform. The Principal Data Engineer joins the Data Platform team, reporting to the Director of Machine Learning Engineering and Data, and leads the design, evolution, and operation of a Databricks‑based lakehouse serving more than 30 healthcare customers.

Key responsibilities

• Define and execute the technical strategy and roadmap for the data platform, ensuring scalability, reliability, regulatory alignment, and operational excellence.
• Design, build, and maintain scalable streaming and batch pipelines on Databricks using PySpark, Unity Catalog, Delta Lake, and Structured Streaming.
• Implement Terraform‑based infrastructure‑as‑code, Jenkins CI/CD workflows, and cost‑optimization measures across environments.
• Enforce data governance and security, applying least‑privilege access, HIPAA and SOC 2 controls, PHI masking, and audit logging.
• Mentor engineers, enable ML, BI, and analytics teams with model‑ready datasets, feature stores, and dashboards, and drive cross‑functional adoption.

Required profile

• 8+ years of data engineering experience with progressive responsibility.
• 5+ years hands‑on with Databricks (Unity Catalog, Delta Lake, Structured Streaming, Spark SQL).
• Strong background in cloud infrastructure (AWS) and infrastructure‑as‑code (Terraform).
• Demonstrated ability to operate in a HIPAA‑regulated environment and design disaster‑recovery solutions.

Required skills

• PySpark, Python, Scala
• Databricks platform (Unity Catalog, Delta Lake, Structured Streaming, Spark SQL)
• Terraform for IaC
• Apache Kafka / AWS MSK
• Medallion / lakehouse architecture patterns
• CI/CD pipelines (Jenkins, GitHub Actions)
• Advanced SQL (CTEs, window functions, performance tuning)
• AWS services (S3, IAM, Secrets Manager, VPC/Private Link, cross‑region replication)
• HIPAA & SOC 2 compliance practices
• Disaster‑recovery design and cost‑optimization tools (CloudZero, AWS Cost Explorer)
• AI assistance tools (Claude, Gemini, Codex)

What we offer

• Health, dental and vision insurance (80% covered by employer)
• 401(k) retirement plan with early eligibility
• Generous paid time off and flexible work arrangements
• Annual performance reviews and professional development support
• Recognition awards and peer‑nominated honors
• Modern office in Boston’s Back Bay neighborhood
• Employer‑paid life, short‑term and long‑term disability insurance