Senior Azure Data Engineer
Core
Design and build end-to-end Big Data pipelines and real-time data pipelines for enterprise clients in the banking and payments sector.
Role type
Senior Azure Data Engineer
Builds
ETL pipelines, real-time data pipelines, and data transformation logic using Hadoop ecosystem and Spark.
Domain
Banking (BFSI), Cards and Payments
Deliverable
production ML models
Required skills
Apache Spark, PySpark, Spark SQL, Hive, Python, Sqoop, MySQL, Oracle, MongoDB, MS SQL, IBM DB, Jenkins, Bitbucket, Control-M, Sterling
Preferred skills
Agile methodology, SCD type 2 logic, shell scripting, root cause analysis (RCA), production deployment
Technologies
Apache Spark, Hive, PySpark, Spark SQL, Sqoop, MySQL, Oracle, MongoDB, MS SQL, IBM DB, Jenkins, Bitbucket, Control-M, Sterling
Responsibilities
Design and build ETL and real-time data pipelines; develop transformation logic using PySpark and Spark SQL; manage CI/CD pipelines for code deployment; implement SCD type 2 logic for data history; analyze jobs and provide root cause analysis for issues; collaborate on production deployment and design walkthroughs; create Control-M jobs for ingestion and transformation layers.
Seniority
Senior, hands-on IC