(Senior) Data Engineer (Kafka)
Core
Designing, building, and supporting scalable event-driven data ingestion and transformation pipelines for an enterprise Operational Data Store (ODS) in a cloud-native AWS environment.
Role type
Senior Data Engineer (Event-Driven/Kafka)
Builds
Scalable data pipelines, reusable metadata-driven frameworks, and end-to-end ODS data flows integrating APIs and event streams.
Domain
Financial Services / Insurance / Data Engineering
Deliverable
production ML models | product features | dashboards & analysis
Required skills
Data Engineering, Kafka/MSK, AWS Glue, Python, PySpark, ETL/ELT, Data Modeling, JSON/Avro schema integrations, CI/CD, Data Quality/Validation
Preferred skills
Apache Kafka/MSK, CDC technologies (Debezium), Metadata-driven data platforms, Financial Services/Insurance domain experience
Technologies
Kafka, MSK, AWS Glue, S3, ECS, Aurora, PostgreSQL, Python, PySpark, Debezium
Responsibilities
Building and maintaining scalable data pipelines using Kafka/MSK, AWS Glue, Python, and PySpark; Designing and implementing ingestion, transformation, validation, and publishing processes; Delivering end-to-end ODS data flows integrating APIs, event streams, and batch sources; Developing reusable frameworks and metadata-driven solutions; Ensuring data quality, integrity, and adherence to enterprise data models; Supporting CI/CD pipelines, monitoring, and operational excellence; Collaborating with Product Owners, Architects, and Domain SMEs to deliver business outcomes.
Seniority
Senior, hands-on IC