Senior Data Engineer
Core
Design, develop, test, and maintain production-grade distributed data-processing workloads using PySpark and Apache Spark, focusing on performance optimization and troubleshooting.
Role type
Senior IC data engineer (Spark internals & optimization)
Builds
Production-grade batch and incremental processing pipelines
Domain
Data engineering / Distributed systems
Deliverable
production ML models
Required skills
Python, PySpark, Apache Spark internals, SQL, performance optimization, distributed processing, data structures, exception handling
Preferred skills
Spark Structured Streaming, Kafka, Delta Lake, Apache Iceberg, Databricks, Amazon EMR
Technologies
PySpark, Apache Spark, Spark SQL, Kafka, Delta Lake, Apache Iceberg, Databricks, Amazon EMR
Responsibilities
Design and develop production-grade PySpark applications; diagnose and resolve complex performance bottlenecks; build and maintain reliable batch and incremental processing pipelines; troubleshoot job failures and performance regressions; evaluate technical trade-offs and collaborate with engineers and architects.
Seniority
Senior, hands-on IC
