Senior Data Engineer - Hadoop/Spark/OCP/Big Data
Core
Design, develop, and maintain scalable data pipelines and data service infrastructure using Spark, SQL, and distributed platforms to support batch and real-time processing for multiple applications.
Role type
Senior Data Engineer (Big Data/Cloud)
Builds
Scalable ETL/ELT frameworks, data lake solutions (Iceberg, S3/ADLS), and containerized data workloads (Docker, Kubernetes/OCP).
Domain
Financial Services / Big Data Engineering
Deliverable
production ML models | product features | infrastructure
Required skills
Spark, SQL, Distributed Systems, ETL/ELT, Docker, Kubernetes, OCP, CI/CD, Data Modeling, Performance Optimization
Preferred skills
Data Analytics, Data Mining, Machine Learning, Data Architecture
Technologies
Apache Spark, Apache Iceberg, Docker, Kubernetes, OpenShift (OCP), Jenkins, GitHub, Bitbucket, Azure DevOps, S3, ADLS
Responsibilities
Design and build data service infrastructure; optimize Spark jobs for performance and resource efficiency; implement CI/CD pipelines for data workflows; partner with stakeholders to define data requirements; provide technical leadership and mentor junior engineers.
Seniority
Senior, hands-on IC with leadership responsibilities
