Lead Data Engineer
Core
Architect, design, and maintain complex data pipelines and scalable data architectures to support business objectives and decision-making.
Role type
Senior IC Lead Data Engineer
Builds
Robust, scalable data workflows and data models
Domain
Financial markets infrastructure and data
Deliverable
production ML models | product features | infrastructure
Required skills
Azure Data Factory, SQL Server, Python (PySpark), Databricks, Spark-based processing, scalable data architecture design
Preferred skills
AWS Athena, Terraform, data governance, distributed systems
Technologies
Azure Data Factory, SQL Server, Python, PySpark, Databricks, Spark, AWS Athena, Terraform
Responsibilities
Architect and maintain complex data pipelines; Build and optimize data models and transformations; Develop Python scripts for automation and integration; Implement and optimize Databricks workflows; Fix pipeline issues and implement long-term improvements; Mentor junior engineers and establish best practices
Seniority
Senior, hands-on IC with leadership responsibilities