Principal Engineer - Data Retrieval
Core
Architect and scale the data acquisition infrastructure that powers NVIDIA's foundation models, including crawling the open web, discovering sources, and delivering high-quality data to training pipelines.
Role type
Principal Engineer (Data Retrieval & Distributed Systems)
Builds
Unified platform for source discovery, URL indexing, crawl frontiers, predictive ranking, scheduling, distributed fetching, and ingestion.
Domain
AI/ML Infrastructure, Web Crawling, Distributed Systems
Deliverable
production ML models
Required skills
Large-scale distributed systems design, web crawling and ingestion infrastructure, system architecture, pipeline engineering (batch/streaming), production debugging, technical leadership, data governance and policy enforcement.
Preferred skills
Architecting web-wide crawl platforms, supplying foundation-model pretraining data, implementing publisher consent and provenance infrastructure, browser-based or agentic data collection.
Technologies
Distributed systems, storage and queueing systems, URL indexing, crawl frontiers, streaming pipelines, observability tools.
Responsibilities
Design scalable frontier storage and queueing systems with high-throughput writes and deduplication; develop a shared crawl execution layer for reliable, polite fetching at scale; build extensible pipelines for open-web, licensed, and partner content; partner with parsing teams to preserve valuable code and structured content; implement provenance and policy enforcement with legal and security partners; own production readiness including capacity planning, fault tolerance, and cost efficiency.
Seniority
Principal, hands-on technical leadership & mentorship
