Lead Data Engineer - Data Migration & Lakehouse
Core
Lead large-scale data migrations from legacy systems (mainframes, SQL, ERP) into modern cloud/lakehouse architectures, designing robust pipelines and establishing data quality frameworks.
Role type
Senior Lead Data Engineer (Data Migration & Lakehouse)
Builds
Modern cloud/lakehouse data architectures and automated validation pipelines
Domain
Data Engineering / Cloud Data Platforms
Deliverable
production ML models
Required skills
Large-scale legacy-to-cloud migration leadership, Databricks ecosystem mastery (Spark, Delta Lake, Unity Catalog, Lakeflow, Asset Bundles), Data modeling & schema mapping across heterogeneous systems, Discovery & gap analysis, Data quality framework design, Stakeholder risk communication, AI/LLM-assisted workflow integration
Preferred skills
Legacy system archaeology (COBOL, EDI), Agentic AI exposure, Metadata management beyond Unity Catalog, LLM-based schema matching tools
Technologies
Databricks, Spark, Delta Lake, Unity Catalog, Lakeflow, Asset Bundles, Great Expectations, dbt
Responsibilities
Lead end-to-end migration projects from legacy to cloud/lakehouse, Design and implement CI/CD-style data pipelines, Own data modeling and reconciliation across heterogeneous systems, Conduct discovery assessments and translate ambiguous specs into requirements, Establish automated data quality validation standards, Surface technical risks to non-technical stakeholders
Seniority
Senior, hands-on IC with leadership responsibilities