AWS Cloud Developer - Data Streaming
Core
Designing and optimizing large-scale real-time data streaming and batch ETL data processing platforms on AWS for the Treasury division.
Role type
Senior IC AWS Cloud Developer (Data Streaming)
Builds
Low-latency, event-driven pipelines, fault-tolerant scalable solutions, and enterprise data lakes.
Domain
Financial Services / Cloud Data Engineering
Deliverable
production ML models | product features | dashboards & analysis
Required skills
Amazon Kinesis, Amazon MSK (Kafka), AWS Lambda, AWS Glue, Amazon EMR, Amazon Redshift, Amazon S3, Python, PySpark, SQL, AWS CDK, CloudFormation, Airflow/MWAA, CloudWatch, CloudTrail, GIT, Bitbucket, performance tuning, Change Data Capture (CDC), IAM, encryption, data access controls
Preferred skills
DevOps/CI principles, root cause analysis, automation, documentation, after-hours release deployment
Technologies
Airflow, AWS, AWS Glue, Lambda, Redshift, BitBucket, CI/CD, CloudWatch, DevOps, ETL, Git, GitHub, IAM, Kafka, Python, PySpark, SQL, Spark, Snowflake
Responsibilities
Establish standards and best practices for real-time streaming and batch ETL; Develop and implement low-latency, event-driven pipelines; Create fault-tolerant and scalable solutions for high-volume event data; Integrate Change Data Capture (CDC) pipelines; Design and optimize batch ETL/ELT processes; Construct and manage enterprise data lakes; Set up monitoring, alerting, and observability; Execute data quality checks, validation rules, and reconciliation processes; Monitor pipelines and address production failures; Implement CI/CD pipelines for data workflows; Analyze and resolve issues in real-time; Propose innovative solutions and automate routine processes; Provide support for essential production systems; Conduct scheduled maintenance and assist with after-hours release deployment activities
Seniority
Senior, hands-on IC
