Data Engineer - A26141
Core
Design, develop, and deploy data tables, views, and marts in data warehouses, operational data stores, data lakes, and data virtualization environments.
Role type
Data Engineer
Builds
Large-scale batch and real-time data pipelines, backend APIs, and databases supporting applications.
Domain
Cloud data engineering, ETL/ELT, big data frameworks
Deliverable
production ML models | product features | dashboards & analysis | infrastructure
Required skills
SQL, Python, R, pandas, data cleaning and transformation, ETL pipeline construction, database design, cloud technologies (AWS, Azure, Google Cloud), big data frameworks (Hadoop, Spark, Kafka, RabbitMQ), web scraping (BeautifulSoup, Selenium, Nodejs), REST API, data modeling, data governance
Preferred skills
Microsoft Dynamics, IBM Curam CRM, Spring, AWS Lambda, ECS Container task, Eventbridge, AWS Glue, AWS S3, Athena, MongoDB, PostgreSQL, GIS, MySQL, SQLite, VoltDB, Cassandra, W3C DOM
Technologies
SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), AWS, Azure, Google Cloud, Hadoop, Spark, Kafka, RabbitMQ, BeautifulSoup, CasperJS, PhantomJS, Selenium, Nodejs
Responsibilities
Design, build, launch, and maintain efficient and reliable large-scale batch and real-time data pipelines; Integrate and collate data silos in a scalable and compliant manner; Collaborate with cross-functional teams to build scalable data-driven products; Develop backend APIs and work on databases to support applications; Perform data extraction, cleaning, transformation, and flow including web scraping
Seniority
Mid-level, hands-on IC
