Software Engineer, Data Infrastructure
Core
Architecting and scaling core infrastructure for distributed training pipelines, multimodal data catalogs, and intelligent processing systems operating over petabytes of data to accelerate AI research.
Role type
Senior IC data infrastructure engineer
Builds
Scalable, fault-tolerant data infrastructure for LLM research
Domain
AI/ML infrastructure, distributed systems, data engineering
Deliverable
infrastructure
Required skills
Python or Rust, Apache Spark or Ray, cloud infrastructure, data lake architectures, batch and streaming pipelines
Preferred skills
Kafka, dbt, Terraform, Airflow, web crawling, data deduplication, data mining, search, Parquet, Delta Lake
Technologies
Spark, Ray, Kafka, dbt, Terraform, Airflow, Parquet, Delta Lake
Responsibilities
Design and operate scalable, fault-tolerant infrastructure for distributed compute, data orchestration, and storage; Develop high-throughput systems for data ingestion, processing, and transformation; Build systems for traceability, reproducibility, and quality control; Implement monitoring and alerting for platform reliability; Collaborate with research teams to accelerate training cycles
Seniority
Senior, hands-on IC
