Software Engineering Manager, Data Platform
Core
Lead a team building and scaling a high-throughput, governed data platform (batch processing) on a modern lakehouse architecture to power critical business decisions and AI products.
Role type
Senior Engineering Manager, Data Platform
Builds
Large-scale distributed data processing systems, self-service batch processing capabilities, and platform abstractions for internal teams.
Domain
Data Engineering / Cloud Infrastructure / AI Platform
Deliverable
production ML models | infrastructure
Required skills
Distributed data processing (Spark, Flink), Batch pipelines at scale, AWS, Data lake/lakehouse architectures, Workflow orchestration (Airflow, Temporal), Team leadership, Hiring and mentoring
Preferred skills
Apache Iceberg or Delta Lake, Data observability tools, Metadata and catalog systems, Data governance frameworks
Technologies
Apache Spark (PySpark), Airflow, Temporal, Kubernetes, S3, Apache Iceberg, Ray, SQL engines
Responsibilities
Lead and mentor engineers on batch processing and data platform capabilities; Design and evolve scalable batch processing platforms; Drive adoption of modern lakehouse architecture; Ensure platform adherence to data governance and security standards; Build end-to-end observability and drive incident management; Partner with cross-functional teams to drive platform adoption.
Seniority
Senior, hands-on IC with management responsibilities