Senior Data Engineer
Core
Build and maintain large-scale data pipelines and web scrapers to capture, classify, and transform millions of data points for real-time access by clients and data science teams.
Role type
Senior Data Engineer (Data Infrastructure & Scraping)
Builds
Data pipelines, crawlers, scrapers, and data storage solutions for a cultural intelligence platform
Domain
Internet / Data Infrastructure / Web Scraping
Deliverable
production ML models
Required skills
Python, Node.js, Web Scraping, API Integration, HTML Parsing, SQL, Data Pipelines (ETL), Data Orchestration, Cloud Platforms
Preferred skills
Browser Automation, Proxy Rotation, Anti-blocking Strategies, Reverse Engineering, dbt, Spark, Open Source Projects
Technologies
Python, Node.js, requests, httpx, asyncio, axios, puppeteer, crawlee, SQL, Parquet, JSON, Prefect, Airflow, Temporal, AWS, GCP, Azure, Playwright, Selenium, Scrapy, Crawlee, dbt, Spark
Responsibilities
Develop and maintain crawlers and scrapers of varying complexity; Map new external data sources and evaluate technical feasibility; Automate large-scale data collection flows; Adjust and adapt scrapers to website changes; Ensure crawler quality and continuous updates; Process and organize collected data for pipeline integration; Collaborate with the team to solve business problems and deliver reliable data
Seniority
Senior, hands-on IC