Web Scraping Specialist
Core
Design, implement, and maintain scalable web scraping processes to collect data from online sources for competitor analysis and business intelligence.
Role type
Senior IC web scraping architect
Builds
Automated data extraction pipelines for e-commerce marketplaces and other online sources
Domain
Data engineering / Web scraping
Deliverable
production ML models
Required skills
Python, Scrapy, Selenium, BeautifulSoup, HTML, CSS, XPath, proxy management, CAPTCHA bypass, user-agent rotation, data cleansing, data validation, large-scale data storage, Git
Preferred skills
Cloud-based distributed scraping systems, APIs, non-public data extraction, machine learning, NLP, big data frameworks, JSON/XML/CSV formats
Technologies
Python, Scrapy, Selenium, BeautifulSoup, Git
Responsibilities
Lead and provide expertise in scraping at scale; Identify relevant websites and data sources; Develop and implement web scraping strategies; Create and maintain optimized scraping scripts; Cleanse and validate collected data; Monitor and maintain scraping processes; Optimize procedures for efficiency and scalability; Ensure compliance with legal and ethical considerations; Document processes and methodologies; Collaborate with data analysts, developers, and stakeholders; Implement security measures for data confidentiality.
Seniority
Senior, hands-on IC with leadership experience