Data Engineer, Data Quality & Provenance
Core
Design and operate scalable batch and streaming pipelines to transform petabyte-scale fleet and simulation data into trusted, reproducible datasets for autonomous driving model training and safety analysis.
Role type
Senior Data Engineer (Data Quality & Provenance)
Builds
Petabyte-scale data products, versioned datasets, metadata catalogs, and distributed compute infrastructure for autonomous vehicle development.
Domain
Autonomous driving, AI platforms, sensor data, geospatial data
Deliverable
production ML models
Required skills
Python, SQL, distributed data systems, workflow orchestration (Spark, Flyte, Airflow), cloud data platforms, data quality & lineage, data governance
Preferred skills
Autonomous vehicles, ADAS, robotics, mapping, sensor-rich systems, geospatial data, multimodal datasets
Technologies
Spark, Flyte, Airflow, object storage, cloud infrastructure
Responsibilities
Design and operate scalable batch and streaming pipelines; Build data models, catalogues, indexes and query capabilities; Create versioned, reproducible datasets; Develop workflows for data ingestion, transformation, curation, and quality validation; Partner with engineering teams to define schemas and data contracts; Improve reliability and cost efficiency of storage and compute workloads.