Senior Software Engineer - Data Infrastructure
Core
Build and operate a petabyte-scale data platform to ingest, process, version, and serve high-fidelity spatial data (video, depth, IMU, reconstructions) for training embodied AI models.
Role type
Senior IC Data Infrastructure Engineer
Builds
PB-scale data lake/lakehouse, distributed processing pipelines, data versioning systems, and dataset query layers for AI training clusters.
Domain
AI/ML infrastructure, spatial data, robotics, cloud-native distributed systems
Deliverable
production ML models
Required skills
Petabyte-scale data platform design, object storage (S3), open table formats (Iceberg/Delta Lake), distributed processing (Spark/Ray/Flink), workflow orchestration (Airflow/Prefect), Python, Go/Rust/C++, AWS infrastructure, Terraform, on-call operations
Preferred skills
Multimodal data (video/LiDAR), data versioning (LakeFS/DVC), GPU-accelerated processing, query engines (Trino/DuckDB), multi-region architectures, robotics log formats (ROS/MCAP)
Technologies
S3, Iceberg, Delta Lake, Spark, Ray, Flink, Airflow, AWS (EC2, EKS, S3, IAM), Terraform, Trino, DuckDB, WebDataset, Mosaic Streaming
Responsibilities
Design storage layout and partitioning for petabyte-scale data lakes; implement data versioning and lineage systems; build high-throughput fault-tolerant ingest pipelines; operate distributed processing across CPU/GPU fleets; maintain idempotent data pipelines with retry/backfill logic; build dataset curation and query layers; optimize GPU feeding and cross-region data movement; manage platform reliability, observability, and cost on AWS; contribute to IaC and CI/CD.
Seniority
Senior, hands-on IC
