Data Engineer AWS
Core
Build and sustain resilient, scalable data pipelines in AWS to ensure regulatory data products reach their destination without latency or failures.
Role type
Senior Data Engineer (AWS)
Builds
Production data pipelines and validation suites for regulatory compliance
Domain
Financial Services / Regulatory Compliance (Brazil Central Bank Resolution 18)
Deliverable
production ML models | product features
Required skills
Python, AWS ecosystem (EMR, Spark), Great Expectations, data quality rule implementation, pipeline automation, audit log generation
Preferred skills
Domain knowledge of RC 18 obligations, experience with Airflow
Technologies
AWS, EMR, Spark, Airflow, Great Expectations
Responsibilities
Configure expectation suites for critical RC 18 domains; Integrate quality validations into Airflow/Spark/EMR pipelines; Ensure coverage of final Data Lake tables; Automate pre-submission CADOC validations and generate audit logs
Seniority
Senior, hands-on IC