Databricks Data Engineer
Core
Design and deliver production data pipelines on Databricks to modernize the bank's Enterprise Data Warehouse from SQL Server/SSIS to a lakehouse architecture.
Role type
Senior Databricks Data Engineer
Builds
Production data pipelines (ingest, Bronze/Silver/Gold modeling) on Databricks for downstream reporting and analytics.
Domain
Financial services / Data Engineering
Deliverable
production ML models
Required skills
PySpark, Spark SQL, Delta Lake, Databricks Jobs/Workflows, SQL (T-SQL to lakehouse), Git, Cloud (Azure), Object Storage (ADLS/S3/GCS), AI coding assistants
Preferred skills
Lakeflow Connect, Unity Catalog, Azure Entra, dbt, Data Vault, Financial services regulation
Technologies
Databricks, Lakeflow, Spark, Delta Lake, Unity Catalog, Azure, GitHub, Tableau, SQL Server, SSIS, Fivetran
Responsibilities
Design and build Databricks pipelines with Lakeflow and custom PySpark; Implement Bronze/Silver/Gold medallion models; Schedule, monitor, and tune jobs for performance and cost; Integrate Databricks with GitHub and Azure DevOps; Validate data quality against legacy EDW outputs; Upskill the EDW team through pairing and design reviews.
Seniority
Senior, hands-on IC
