Senior Data Engineer - Spark / Kafka / NiFi
Core
Design, build, and maintain large-scale batch and real-time data processing platforms using distributed systems and streaming architectures.
Role type
Senior IC data engineer (Spark/Kafka/NiFi)
Builds
Scalable data pipelines, validation suites, and reusable engineering frameworks for high-volume datasets.
Domain
Payments / Distributed Systems / Real-time Data Processing
Deliverable
production ML models | product features
Required skills
Apache Spark (Batch & Structured Streaming), Apache Kafka, Apache NiFi, Scala/Python, SQL, Distributed Storage (Ozone/Ceph/HDFS), Linux, Git, CI/CD
Preferred skills
Containerization (Docker/Kubernetes), Observability solutions, Data governance, Performance engineering, Engineering automation
Technologies
Apache Spark, Apache Kafka, Apache NiFi, Apache Ozone, Ceph, HDFS, Docker, Kubernetes
Responsibilities
Develop and validate scalable batch and streaming applications; Build validation suites for fault-tolerant distributed systems; Create reusable frameworks and shared platform components; Optimize system performance, throughput, and resource utilization; Monitor production environments and lead troubleshooting/root-cause analysis; Mentor junior engineers and contribute to engineering standards.
Seniority
Senior, hands-on IC
