Big Data Engineer (Spark) @ Connectis_
Core
Designing and implementing pipelines for ingesting, transforming, and processing large volumes of data for B2B and B2C banking projects.
Role type
Senior Big Data Engineer (Spark/Scala)
Builds
Scalable, reliable, fault-tolerant data processing solutions and cloud migration initiatives.
Domain
Banking (B2B/B2C) + Big Data Engineering
Deliverable
production ML models | product features | infrastructure
Required skills
Apache Spark, SQL, Hive, Scala, Java, TDD, Cloud migration, Microservices architecture, REST API
Preferred skills
CI/CD (Jenkins, GitLab CI), Python (Airflow), Azure (ADF, ADLS, Databricks), SAFe/Release Train
Technologies
Apache Spark, Hive, SQL, Scala, Java, Jenkins, GitLab CI, Airflow, Azure ADF, Azure ADLS, Azure Databricks
Responsibilities
Designing and implementing data ingestion, transformation, and processing pipelines; Industrializing data processing solutions for scalability and reliability; Developing and optimizing Spark/Scala jobs (batch processing, data mapping, performance tuning); Participating in Big Data architecture design; Analyzing data sources and translating business requirements into technical specifications; Code review and maintaining TDD/BDD standards; Production support (L3), troubleshooting, and risk identification; Creating documentation and test plans in Agile/Scrum environments.
