Data engineer AWS Python Apache Iceberg
Core
Design, develop, and maintain scalable ETL/ELT pipelines for massive data collection, transformation, and loading on AWS.
Role type
Senior Data Engineer (AWS, Python, Apache Iceberg)
Builds
Real-time and batch data pipelines, optimized storage tables, and data quality mechanisms
Domain
Cloud Data Engineering, Big Data, Real-time Streaming
Required skills
Python, AWS (Glue, Lambda, EMR, Kinesis), Apache Kafka, Apache Iceberg, JSON, SQL, Docker, Kubernetes, Terraform
Preferred skills
Data modeling, DevOps best practices, mentoring junior consultants
Technologies
AWS, Python, Apache Kafka, Apache Iceberg, JSON, Parquet, Avro, Docker, Kubernetes, Terraform
Responsibilities
Design and maintain ETL/ELT pipelines for batch and streaming data; Optimize pipeline performance and scalability on AWS; Implement data quality mechanisms (validation, monitoring, alerting); Collaborate with DevOps for cloud deployment; Mentor junior consultants on Python, Kafka, and Iceberg
Seniority
Senior, hands-on IC