Data Engineer
Core
Design, build, and maintain large-scale data pipelines and platforms to transform raw data into actionable insights for decision-makers.
Role type
Data Engineer
Builds
Data warehouses, operational data stores, data lakes, and scalable data pipelines
Domain
Cloud data engineering and big data infrastructure
Required skills
SQL, Python, ETL/ELT pipeline design, database design, cloud platforms (AWS, Azure, Google Cloud), big data frameworks (Spark, Kafka, Hadoop), web scraping, REST APIs
Preferred skills
Databricks, Data Fabric (Microsoft), data governance, system design, data modeling
Technologies
SQL, Python, pandas, R, SSIS, AWS DMS, AWS Lambda, ECS, Eventbridge, AWS Glue, Spring, PostgreSQL, MongoDB, MySQL, Athena, S3, VoltDB, Cassandra, Hadoop, Spark, Kafka, RabbitMQ, BeautifulSoup, CasperJS, PhantomJS, Selenium, Nodejs
Responsibilities
Design and deploy data tables, views, and marts; perform data extraction, cleaning, transformation, and flow including web scraping; build and maintain batch and real-time data pipelines; integrate and collate data silos; develop backend APIs and databases; collaborate with cross-functional teams in an Agile environment
Seniority
Mid-level to Senior, hands-on IC