Senior Data Engineer (Web Scraping)
Core
Design, build, and operate production-grade web-scraping systems and scalable ingestion pipelines to support trusted research and data products.
Role type
Senior hands-on IC data engineer (web scraping)
Builds
Production web-scraping frameworks, scalable ingestion pipelines, and data lakehouse architectures
Domain
Data engineering, web data acquisition, cloud infrastructure
Required skills
Python engineering, web scraping frameworks (Requests, httpx, BeautifulSoup, Scrapy, Playwright, Selenium), HTTP/HTML/APIs, browser automation, data pipelines, cloud deployment, debugging, rate limiting, concurrency, proxies
Preferred skills
AWS, Apache Iceberg, PySpark, Docker, Terraform, Grafana, commercial scraping/proxy services, automated testing, data governance (via careerplan.io/jobs/39c55fe1-f85a-4554-9392-619ffb18158d-senior-data-engineer-web-scraping-at-jobgether)
Technologies
Python, Requests, httpx, BeautifulSoup, Scrapy, Playwright, Selenium, AWS, Apache Iceberg, PySpark, Docker, Terraform, Grafana
Responsibilities
Own development, deployment, and operation of web-scraping pipelines; Design scalable scraping frameworks with reusable patterns; Build and maintain reliable production scrapers; Investigate data sources and determine acquisition methods; Evaluate build-versus-buy options for scraping infrastructure; Ensure compliance with policies, robots.txt, and privacy; Diagnose and resolve scraping challenges (dynamic content, auth, rate limits); Integrate workloads into data-platform and lakehouse architectures; Improve scheduling, monitoring, and operational support.
Seniority
Senior, hands-on IC
