Data Engineer
Core
Design, develop, and maintain data solutions for data generation, collection, and processing, specifically migrating data from legacy on-prem systems to AWS.
Role type
Senior IC Data Engineer (AWS Data Migration)
Builds
ETL pipelines and data migration solutions for high-volume datasets
Domain
Cloud data engineering (AWS)
Deliverable
production ML models | product features
Required skills
AWS Athena, PySpark, SQL, ETL, Unix, Iceberg, Astronomer, data warehousing concepts, data pipeline architecture
Preferred skills
AWS Step Functions, AWS EventBridge, AWS Glue, AWS Lambda, CloudWatch, data quality validation, incremental loads
Technologies
AWS Athena, AWS Glue, PySpark, SQL, Unix, Iceberg, Astronomer, Step Functions, EventBridge, Lambda, CloudWatch
Responsibilities
Implement end-to-end ETL pipelines using AWS native services; handle high-volume datasets with error handling and retry mechanisms; test pipelines for performance and deploy with DevOps support; monitor and troubleshoot pipelines using CloudWatch; optimize for cost and process scalability; support data quality validation and incremental loads.
Seniority
Senior, hands-on IC