Member of Technical Staff - Engineering Lead, Data Ingestion
Core
Lead the Data Ingestion team building the machinery to turn the open web and licensed corpora into well-structured, versioned, auditable datasets for pre-training frontier AI models.
Role type
Engineering Lead, Data Ingestion (hands-on IC)
Builds
Web crawl infrastructure, data ingestion pipelines, and data lakes for training corpora
Domain
AI/ML pre-training data infrastructure
Deliverable
production ML models
Required skills
team building and mentoring, web-scale data acquisition, distributed processing, data orchestration, data lake architecture, LLM training intuition, cross-functional partnership
Preferred skills
experience scaling teams quickly, deep expertise in specific ingestion sub-domains (crawling/extraction/storage)
Technologies
Ray, Beam, Spark, Airflow, Prefect, Parquet, JSONL, WARC, object stores
Responsibilities
Build and mentor a high-performing team of data ingestion engineers; guide technical and architectural decisions across crawling, extraction, and storage; stay hands-on to make targeted contributions; manage day-to-day execution of data acquisition campaigns; partner with research teams to tie ingestion decisions to model impact; onboard new data sources within legal constraints
Seniority
Senior, hands-on IC with leadership responsibilities