Data Engineer (Big Data)
Core
Design and maintain big data ingestion pipelines, configure KPI/KRI processing, and monitor Spark applications for a European digital consulting firm.
Role type
Senior Data Engineer (Big Data)
Builds
Data ingestion processes, KPI/KRI reporting systems, and real-time processing pipelines
Domain
Big Data, Cloud Data Platforms, Real-time Analytics
Deliverable
production ML models | product features | infrastructure
Required skills
Advanced SQL (Hive, Impala), Spark SQL, Spark Streaming, Linux/Unix administration, Bash/PowerShell/Python scripting, Kafka, Flink, RabbitMQ, Jenkins, Yarn, GitHub, ServiceNow, ALM
Preferred skills
Experience resolving failed processes, knowledge of data replication and high availability
Technologies
HDFS, Spark, Hive, Impala, Kafka, Flink, RabbitMQ, Jenkins, Yarn, GitHub, ServiceNow, Bash, PowerShell, Python
Responsibilities
Develop data ingestion processes from databases and intermediate systems; Configure and execute KPI/KRI ingestion files; Execute and monitor Spark applications; Perform unit tests and prepare deployments; Integrate new processes into GitHub; Analyze and restructure failed processes; Manage incidents and requests
Seniority
Senior, hands-on IC
