Especialista Cloudera Sênior (Big Data e Google Cloud)
Core
Senior Cloudera specialist leading a strategic data platform migration from AWS to Google Cloud for a large financial institution.
Role type
Senior Big Data Engineer (Cloud Migration)
Builds
Data platform migration (lift and shift) preserving behavior of 13k Spark/PySpark jobs and 2.5PB of data
Domain
Big Data, Cloud Migration, Financial Services
Deliverable
production ML models
Required skills
Cloudera (CDP/CDH), Spark, PySpark, Hive, Impala, Kafka, HBase, NiFi, Ranger, Atlas, Kerberos
Preferred skills
Google Cloud (Dataproc, BigQuery), Apache Flink, CDC (Debezium), Airflow, Cloudera Data Engineering (CDE)
Technologies
Cloudera, Google Cloud, Spark, PySpark, Hive, Impala, Kafka, HBase, NiFi, Flink, Airflow
Responsibilities
Analyze current Cloudera environment (CDP), services, configurations, and dependencies; Support discovery of inventory and dependency mesh; Map Cloudera components to Google Cloud equivalents; Support migration of Spark, Hive, Impala, Kudu, HBase, Kafka, and NiFi; Investigate differences between legacy and target environments; Support migration of security and governance policies; Collaborate with client specialists on proprietary frameworks; Document legacy knowledge for engineering teams
Seniority
Senior, hands-on IC