CareerPlan

Software Engineer, Data Infrastructure

San FranciscoFull-time$300k–$400k2026-11-16 → 2026-11-19
On-siteVisa sponsorship offered

Core

Architecting and scaling core infrastructure for distributed training pipelines, multimodal data catalogs, and intelligent processing systems operating over petabytes of data to accelerate LLM research.

Role type

Senior IC data infrastructure engineer

Builds

Distributed compute, data orchestration, storage, and high-throughput data ingestion/processing systems for LLM research

Domain

AI/ML infrastructure, distributed systems, large-scale data engineering

Deliverable

production ML models

Required skills

Python or Rust, Apache Spark, Ray, cloud infrastructure, data lake architectures, batch and streaming pipelines

Preferred skills

Kafka, dbt, Terraform, Airflow, web crawling, data deduplication, data mining, search, Parquet, Delta Lake

Technologies

Spark, Kafka, Beam, Ray, Delta Lake, dbt, Terraform, Airflow, Parquet

Responsibilities

Design and operate scalable, fault-tolerant infrastructure for distributed compute and storage; Develop high-throughput systems for data ingestion, processing, and transformation; Build systems for traceability, reproducibility, and quality control; Implement monitoring and alerting for platform reliability; Collaborate with research teams to accelerate training cycles

Seniority

Senior, hands-on IC