Data Engineer
Core
Design, build, and maintain large-scale data pipelines and platforms to transform raw data into actionable insights for decision-makers.
Role type
Data Engineer
Builds
Data warehouses, data lakes, operational data stores, and scalable batch/real-time data pipelines
Domain
Enterprise IT services, Cloud Data Engineering
Required skills
SQL, Python, ETL/ELT pipeline design, Cloud platforms (AWS, Azure, Google Cloud), Database design (PostgreSQL, MongoDB, etc.), Big Data frameworks (Spark, Kafka, Hadoop), Web scraping, REST APIs, Data modeling
Preferred skills
Databricks, Data Fabric (Microsoft), Data governance, System design
Technologies
AWS (DMS, Lambda, ECS, Glue, S3, Athena), Azure, Google Cloud, SQL Server, SSIS, Databricks, Spark, Kafka, RabbitMQ, Hadoop, MongoDB, PostgreSQL, MySQL, Node.js, Selenium, BeautifulSoup
Responsibilities
Design and deploy data tables, views, and marts in data warehouses and lakes; Build and maintain efficient batch and real-time data pipelines; Integrate and collate data silos; Develop backend APIs and databases; Collaborate with cross-functional teams to build data-driven products; Perform data extraction, cleaning, and transformation including web scraping.
Seniority
Mid-level to Senior, hands-on IC
