Lead Data Engineer
Core
Senior Data Engineer leading the design and build of scalable data solutions on Databricks for Financial Services clients.
Role type
Senior IC Lead Data Engineer
Builds
Scalable data pipelines, data migrations, and data products for risk, trading, and compliance domains.
Domain
Financial Services (Risk, Trading, Compliance, Regulatory)
Deliverable
production ML models | product features | infrastructure
Required skills
Databricks, Azure, Apache Spark, Delta Lake, MLflow, data architecture, data governance, data lineage, performance tuning, cost optimization, security hardening, team leadership, stakeholder management
Preferred skills
None stated
Technologies
Databricks, Azure, Apache Spark, Delta Lake, MLflow
Responsibilities
Lead design and implementation of scalable data architecture solutions; Architect and optimize data pipelines for structured and unstructured financial data; Stand up and operationalize Databricks platforms; Lead large-scale data migrations from legacy platforms; Drive adoption of Delta Lake, Apache Spark, and MLflow; Establish best practices for data governance, lineage, and metadata management; Collaborate with cross-functional teams to align data architecture with financial regulations; Lead performance tuning, cost optimization, and security hardening; Mentor and provide technical leadership to data engineers and analysts; Translate client business needs into data products.
Seniority
Senior, hands-on IC with leadership responsibilities