Data Engineer
Core
Design, develop, and maintain enterprise-scale data pipelines by analyzing legacy Omnia-based Spark logic and translating it into scalable cloud solutions.
Role type
Senior hands-on IC Data Engineer
Builds
Scalable data pipelines, ingestion, transformation, and integration solutions on AWS and Snowflake
Domain
Cloud data engineering, big data processing, financial services data platforms
Deliverable
production ML models
Required skills
PySpark, AWS (S3, Glue, EMR, Lambda, Athena, Step Functions, SQS, IAM, Secrets Manager, CloudWatch), Snowflake, dbt, Apache Airflow, SQL, Python, Git, CI/CD, DevSecOps
Preferred skills
CBA experience, Omnia platform expertise
Technologies
AWS, Snowflake, dbt, Airflow, PySpark, Python, SQL, Jenkins
Responsibilities
Analyze existing end-to-end data pipelines and source code to understand business logic; Derive and implement new transformation logic; Design and develop reliable, scalable data pipelines using cloud-native technologies; Develop batch and event-driven data processing solutions; Build ingestion, transformation, and enrichment pipelines across data lakes and warehouses; Optimize complex SQL for data transformation and reconciliation; Troubleshoot pipeline failures and performance bottlenecks; Collaborate with stakeholders to translate business requirements into technical data rules.
Seniority
Senior, hands-on IC
