Software Engineer, Data Infrastructure
Core
Design, build, and operate scalable distributed compute, data orchestration, and multimodal storage infrastructure for LLM research.
Role type
Senior IC data infrastructure engineer
Builds
High-throughput data ingestion, processing, transformation, training-data catalogs, deduplication, quality checks, and search systems for LLM research
Domain
Artificial Intelligence / Data Engineering
Deliverable
infrastructure
Required skills
Python, Rust, Apache Spark, Ray, cloud infrastructure, data lake architectures, batch pipelines, streaming pipelines
Preferred skills
Kafka, dbt, Terraform, Airflow, web crawling, data mining, search systems, Parquet, Delta Lake
Technologies
Apache Spark, Ray, Kafka, dbt, Terraform, Airflow, Parquet, Delta Lake
Responsibilities
Design and build distributed compute and data orchestration systems; Develop high-throughput data ingestion and processing pipelines; Implement monitoring and alerting for platform reliability; Collaborate with research teams to improve data quality and accelerate training cycles
Seniority
Senior, hands-on IC