Software Engineer, Data Infra
Core
Design, build, and operate scalable, fault-tolerant infrastructure for distributed LLM research workloads, including data ingestion, processing, transformation, and quality control systems. (via careerplan.io/jobs/22660-software-engineer-data-infra-at-thinking-machines-lab)
Role type
Senior IC data infrastructure engineer (LLM research)
Builds
High-throughput data pipelines, training-data catalogs, deduplication systems, and search infrastructure for LLM research teams.
Domain
Artificial Intelligence / Large Language Models / Data Engineering
Deliverable
production ML models
Required skills
Python, Rust, Apache Spark, Ray, cloud infrastructure, data lake architectures, batch and streaming pipelines, Parquet, Delta Lake
Preferred skills
Kafka, dbt, Terraform, Airflow, web crawler development, scaling deduplication and data mining systems
Technologies
Apache Spark, Ray, Kafka, dbt, Terraform, Airflow, Parquet, Delta Lake
Responsibilities
Design and operate scalable infrastructure for distributed LLM research workloads; develop high-throughput systems for data ingestion and processing; implement monitoring and alerting for platform reliability; collaborate with research teams to accelerate training cycles.
Seniority
Senior, hands-on IC
