Lead Software Engineer, Full Stack
Core
Lead the migration of a massive, high-velocity data ecosystem from a Cloudera-based stack to a modern AWS-native serverless/managed architecture.
Role type
Lead Software Engineer (Data Infrastructure)
Builds
AWS-native data pipelines and serverless data architectures
Domain
Financial Services / Big Data Infrastructure
Deliverable
production ML models | infrastructure
Required skills
HBase, Apache Kafka, AWS MSK, AWS Lambda, AWS Glue, NiFi, data governance, security protocols, cloud-native architecture
Preferred skills
Hadoop ecosystem, AWS DocumentDB, DynamoDB, interactive AI tooling, AWS Certified Data Engineer, AWS Certified Solutions Architect, AWS Certified CloudOps
Technologies
Cloudera, HBase, Kafka, NiFi, AWS EC2, AWS MSK, AWS Lambda, AWS Glue, DocumentDB, DynamoDB
Responsibilities
Maintain and optimize the current Cloudera/HBase/Kafka production environment; Design and execute the phased migration of billions of records to AWS Stack; Work with application developers to refactor data consumers; Implement and maintain strict data governance and security protocols; Transition legacy NiFi and Kafka workflows into streamlined, cloud-native pipelines
Seniority
Lead, hands-on IC with architectural responsibilities