DATA ENGINEER (Scala/Spark)
Core
Migrate Hadoop infrastructure to cloud (Kubernetes, COS, Spark, Airflow) and build automated data pipelines for transformation and quality.
Role type
Senior Data Engineer (Scala/Spark)
Builds
Cloud-native data infrastructure and automated ETL pipelines
Domain
Banking / Big Data Engineering
Deliverable
production ML models | infrastructure
Required skills
Scala, Apache Spark, SQL, CI/CD, Apache Airflow, Kubernetes, Hadoop, S3/COS, Parquet/ORC
Preferred skills
Dremio, Elasticsearch, Kafka, Vault, Gen AI prompt engineering
Technologies
Kubernetes, COS, Spark, Airflow, Gitlab, Jenkins, HDFS, S3, Parquet, ORC, Dremio, Elasticsearch, Kibana, Kafka, Vault
Responsibilities
Migrate Hadoop to cloud infrastructure; Implement data transformation and quality checks; Set up CI/CD pipelines; Automate data ingestion and orchestration; Write technical documentation; Collaborate with cross-functional teams.
Seniority
Senior, hands-on IC