Staff / Senior Software Engineer (Agentic Search) - Crawler
Core
Building distributed crawling, scheduling, and ingestion infrastructure for an agent-native search platform that provides real-world data to AI systems.
Role type
Senior Software Engineer (Backend/Distributed Systems)
Builds
Web-scale crawling systems, ingestion workflows, URL discovery, deduplication, and content extraction pipelines.
Domain
AI Infrastructure / Search Engines / Distributed Systems
Deliverable
production ML models | infrastructure
Required skills
Go, C++, distributed systems design, web protocols (HTTP, DNS, TLS), high-throughput pipelines, scalability, fault tolerance, resource management
Preferred skills
Kafka, Pulsar, NATS, RabbitMQ, Spark, Flink, Beam, MapReduce, streaming data pipelines, event-driven systems
Technologies
Go, C++, Kafka, Pulsar, NATS, RabbitMQ, Spark, Flink, Beam, MapReduce, HTTP, DNS, TLS
Responsibilities
Design and operate web-scale crawling systems; build ingestion workflows for internal and external data sources; develop crawl scheduling and prioritization policies; build systems for URL discovery and content extraction; ensure reliable operation under high-throughput conditions; define observability and quality metrics; monitor resource usage and infrastructure cost; collaborate with indexing and ML teams.
Seniority
Senior, hands-on IC
