Senior Data Engineer - Databricks
Core
Design, develop, and maintain scalable ETL/ELT pipelines and data engineering solutions on the Databricks Lakehouse Platform.
Role type
Senior Data Engineer (Databricks)
Builds
Scalable data pipelines, Delta Lake tables, and data governance frameworks for Fortune 500 clients.
Domain
Data Engineering / Cloud Data Platforms
Deliverable
production ML models
Required skills
Python, PySpark, SQL, Databricks, AWS, REST API integrations, Delta Lake, Unity Catalog, Git, CI/CD
Preferred skills
Data Governance, Data Warehousing, Spark performance tuning
Responsibilities
Design and maintain scalable ETL/ELT pipelines; Configure and optimize Databricks infrastructure (clusters, compute, access controls); Implement data governance principles (quality, lineage, metadata, cataloging); Manage Delta Lake tables using Medallion Architecture; Integrate data from databases, S3, files, and REST APIs; Create, schedule, and optimize Databricks Jobs and Workflows; Implement data validation, error handling, and monitoring; Optimize Spark workloads and SQL queries; Troubleshoot pipeline and infrastructure issues.
Seniority
Senior, hands-on IC