Principal Software Engineer
Core
Lead architecture and design of large-scale distributed systems for web crawling, content acquisition, processing, and retrieval at internet scale.
Role type
Principal Software Engineer (Distributed Systems & Crawler Infrastructure)
Builds
Highly reliable, performant services processing billions of daily web crawling requests; intelligent crawl prioritization and scheduling systems.
Domain
Internet-scale search infrastructure, distributed systems, AI/ML for content optimization
Deliverable
production ML models | infrastructure
Required skills
Distributed systems architecture, Crawler scheduling algorithms, Real-time content discovery, AI/ML integration, Scalability engineering, Observability, Technical leadership, Mentorship, System design, Cross-team collaboration
Preferred skills
N/A
Technologies
C, C++, C#, Java, JavaScript, Python
Responsibilities
Drive technical strategy for crawl scheduling and content freshness; Build and operate efficient services for content acquisition; Apply AI/ML to improve crawl efficiency and platform intelligence; Mentor senior engineers and drive engineering excellence.
Seniority
Principal, hands-on IC with strategic influence
