Data Engineer - Manager/Specialist
Core
Lead a data engineering team to design, implement, and maintain scalable data pipelines, storage solutions, and data services for internal and external consumers.
Role type
Manager, Data Engineering
Builds
Scalable data pipelines, ETL/ELT processes, data storage solutions, and data services
Domain
Data Engineering / Cloud Platforms
Deliverable
production ML models
Required skills
ETL/ELT design and implementation, Big data ecosystems (Spark, Hadoop, Kafka, Flink), Cloud platforms (Azure, AWS, GCP), Python, Java, SQL, Data modeling and schema design, Workflow orchestration (Airflow, Data Factory), Version control (Git), CI/CD practices
Preferred skills
Energy and Commodities or Financial Services sector background
Technologies
Spark, Hadoop, Kafka, Flink, Azure, AWS, GCP, Databricks, Snowflake, BigQuery, Synapse, Airflow, Data Factory, Python, Java, SQL, Git
Responsibilities
Design, implement, and test data pipelines and ETL/ELT processes; Build and maintain scalable, reliable, and high-performance pipelines for structured and unstructured data; Work with big data technologies and distributed systems to process and transform large datasets; Collaborate with data modelers, data scientists, and solution architects to ensure efficient data flows; Partner with data scientists and analysts to deliver clean, well-structured, and accessible data; Coordinate with integration teams and data source owners for smooth ingestion of data; Automate provisioning, deployments, and environment management for data platforms; Create and support APIs and data services for internal and external consumers
Seniority
Manager, team leadership