Databricks Engineer
Core
Lead the migration of Hadoop workloads to the Databricks Lakehouse platform and build scalable data solutions.
Role type
Mid-level Databricks Engineer (Data Migration & Lakehouse)
Builds
Scalable, repeatable data solutions on Databricks Lakehouse
Domain
Cloud Data Engineering / Big Data
Deliverable
production ML models | product features | infrastructure
Required skills
Databricks, Spark (Scala/PySpark), ETL pipeline development, Hadoop, Azure cloud, Delta Lake, Unity Catalog, Lakehouse Federation, Liquid Clustering
Preferred skills
Databricks Migration Accelerator, synthetic data generation, pipeline health monitoring, performance tuning, query optimization, cost management
Technologies
Azure, Databricks, Hadoop, PySpark, Scala, Unity Catalog
Responsibilities
Identify and classify existing Hadoop jobs and data sources for migration planning. Provision and configure Databricks workspaces with Lakehouse architecture. Execute pilot migrations of Hadoop workloads. Validate and monitor post-migration performance, cost efficiency, and data integrity. Enable and test advanced Databricks capabilities. Collaborate with engineering and governance teams to document learnings. Track and report key success metrics. Contribute to a phased roadmap for full-scale migration.
Seniority
Mid-level, hands-on IC