Data Engineer
Core
Lead the architecture, design, and development of high-throughput batch and real-time streaming data pipelines at an enterprise scale, handling multi-terabyte to petabyte-scale datasets.
Role type
Senior IC Data Engineering Lead (Spark/Databricks)
Builds
Scalable data pipelines, Lakehouse architectures, and high-performance data processing platforms
Domain
Financial Services / Big Data Engineering
Deliverable
production ML models
Required skills
Scala, Python, Apache Spark, Databricks, SQL, Apache Kafka, Apache Iceberg, Delta Lake, Distributed Systems, Cloud Architecture (AWS/GCP), Team Leadership
Preferred skills
Medallion Architecture, Unity Catalog, Delta Live Tables, Performance Tuning, Mentoring
Technologies
Apache Spark, Databricks, Kafka, Scala, Python, SQL, Apache Iceberg, Delta Lake, AWS, Google Cloud Platform
Responsibilities
Architect and develop high-throughput batch and real-time streaming data pipelines; Optimize Spark jobs and SQL queries for performance; Lead and mentor a global team of data engineers; Enforce enterprise data-modelling standards and Lakehouse practices; Diagnose and resolve complex performance bottlenecks in distributed systems.
Seniority
Senior, hands-on IC with leadership responsibilities