Staff Software Engineer (Agentic Search) - Crawler
Core
Design, implement, and operate web-scale crawling systems to discover, fetch, and continuously refresh content from the open web and large-scale data sources for an agent-native search platform.
Role type
Staff Software Engineer (Distributed Systems & Infrastructure)
Builds
Distributed crawling, scheduling, and ingestion infrastructure capable of operating at internet scale
Domain
Cloud Infrastructure / AI Data Acquisition
Deliverable
infrastructure
Required skills
Go, C++, distributed systems design, web protocols (HTTP, DNS, TLS), high-throughput data pipelines, scalability, fault tolerance, resource management
Preferred skills
Kafka, Pulsar, NATS, RabbitMQ, Spark, Flink, Beam, MapReduce, streaming data pipelines, event-driven systems, ad tech, search engines
Technologies
Go, C++, Kafka, Pulsar, NATS, RabbitMQ, Spark, Flink, Beam, MapReduce, HTTP, DNS, TLS
Responsibilities
Design and implement web-scale crawling systems for content acquisition; Build ingestion workflows for internal and external data sources; Develop crawl scheduling, prioritization, and freshness strategies; Build systems for URL discovery, deduplication, and content extraction; Ensure reliable operation of crawling infrastructure under high-throughput conditions; Define observability and quality metrics for crawl coverage and throughput.
Seniority
Staff, hands-on IC
