Software Engineer, Data Infrastructure
Core
Architecting and scaling core infrastructure for distributed training pipelines, multimodal data catalogs, and intelligent processing systems operating over petabytes of data to accelerate LLM research.
Role type
Senior IC data infrastructure engineer
Builds
Distributed compute, data orchestration, storage, and high-throughput data ingestion/processing systems for LLM research
Domain
AI/ML infrastructure, distributed systems, large-scale data engineering
Deliverable
production ML models
Required skills
Python or Rust, Apache Spark, Ray, cloud infrastructure, data lake architectures, batch and streaming pipelines
Preferred skills
Kafka, dbt, Terraform, Airflow, web crawling, data deduplication, data mining, search, Parquet, Delta Lake
Technologies
Spark, Kafka, Beam, Ray, Delta Lake, dbt, Terraform, Airflow, Parquet
Responsibilities
Design and operate scalable, fault-tolerant infrastructure for distributed compute and storage; Develop high-throughput systems for data ingestion, processing, and transformation; Build systems for traceability, reproducibility, and quality control; Implement monitoring and alerting for platform reliability; Collaborate with research teams to accelerate training cycles
Seniority
Senior, hands-on IC