Senior Databricks Data Engineer
Core
Design, develop, and operationalize secure, scalable data pipelines and data products for Government of Canada environments using Databricks and Lakehouse patterns.
Role type
Senior Databricks Data Engineer
Builds
Enterprise data pipelines, ingestion/transformation processes, batch and streaming solutions, Delta Lake tables, and data products.
Domain
Government of Canada, Cloud Data Engineering, Lakehouse Architecture
Required skills
Databricks, Lakeflow, Delta Lake, Unity Catalog, Apache Spark/PySpark, Python, SQL, ETL/ELT, Medallion architecture, Azure data services, Git/CI/CD
Preferred skills
Terraform/Databricks Asset Bundles, Azure DevOps/GitHub, Microsoft Purview, Migration from legacy systems (SAS, Synapse, etc.), Government of Canada delivery experience
Technologies
Databricks, Lakeflow, Delta Lake, Unity Catalog, Apache Spark, PySpark, Python, SQL, Azure, Terraform, Azure DevOps, Microsoft Purview
Responsibilities
Design and build enterprise data pipelines; Develop batch and streaming solutions with Spark; Implement raw, cleaned, and curated layers using medallion patterns; Optimize Delta Lake tables and data products; Connect Databricks with cloud, enterprise, and on-premises sources; Modernize legacy ETL/ELT pipelines; Implement data quality, validation, and monitoring controls; Develop reusable engineering patterns and participate in CI/CD; Support testing, migration, production deployment, and client knowledge transfer.
Seniority
Senior, hands-on IC