Full-Stack Engineer | Web Crawling (m/f/d)
Core
Build and operate large-scale web crawling and extraction systems that turn unstructured, dynamic websites into clean, structured data.
Role type
Senior Full-Stack Engineer (Web Crawling)
Builds
Scalable crawlers, data pipelines, and containerized cloud applications
Domain
Web scraping, data engineering, cloud infrastructure
Deliverable
production ML models
Required skills
Python (async), web scraping at scale, anti-bot defense strategies, data correctness mindset, Docker, cloud platforms (GCP/Azure/AWS), Git, CLI
Preferred skills
LLM-assisted extraction, Go, Java/Spring Boot, React, Terraform, Kubernetes
Technologies
Python, Docker, GCP, Azure Blob Storage, MongoDB, Go, React
Responsibilities
Build and scale crawlers handling dynamic sites; stay ahead of anti-bot measures; extract structured data from HTML/PDF; design ingestion and deduplication pipelines; ship containerized apps with CI/CD; shape technical architecture
Seniority
Senior, hands-on IC