DATA ENGINEER (Scala/Spark)
Core
Migrate Hadoop infrastructure to cloud (Kubernetes, COS, Spark, Airflow) and implement data transformation pipelines ensuring consistency and accuracy.
Role type
Senior Data Engineer (Scala/Spark)
Builds
Cloud-based data infrastructure and automated data pipelines
Domain
Banking / Big Data Engineering
Deliverable
production ML models | infrastructure
Required skills
Scala, Apache Spark, SQL, CI/CD, Apache Airflow, Kubernetes, Hadoop, Shell scripting, NoSQL databases
Preferred skills
Elasticsearch, Kafka, Dremio, Vault, Gen AI prompt engineering
Technologies
Kubernetes Engine, Cloud Object Storage (COS), Parquet, ORC, Gitlab, Jenkins, S3, Oozie
Responsibilities
Migrate Hadoop infrastructure to cloud; Implement data transformation and quality checks; Set up CI/CD pipelines; Automate data ingestion and orchestration; Write technical documentation; Collaborate with cross-functional teams
Seniority
Senior, hands-on IC