Senior Staff Software Engineer - Apache Spark (Java or Scala)
Core
Design and deliver enterprise-grade features for Cloudera's distribution of Apache Spark and Livy, supporting production systems processing petabytes of data across thousands of nodes.
Role type
Senior Staff Software Engineer (Distributed Systems / Data Processing)
Builds
Cloudera Data Engineering Experience stack, specifically Apache Spark and Iceberg components
Domain
Big Data, Distributed Systems, Data Engineering
Deliverable
production ML models | product features
Required skills
Distributed systems design, Large-scale data processing, Systems design, Java/Scala/Python, SQL planning, Fault tolerance, Table formats (Parquet/Iceberg), Root cause analysis, Open-source contribution
Preferred skills
Cloud services (AWS/Azure/GCP/OpenShift), Performance optimization, Scheduling
Technologies
Apache Spark, Livy, Apache Parquet, Apache Iceberg, Java, Scala, Python
Responsibilities
Design new features for data engineering experience, Contribute to Apache Spark and Livy development, Develop features in Scala/Java/Python, Debug system-level deployment issues and perform root cause analysis, Work on large-scale distributed systems (100s-1000s of nodes), Improve internal infrastructure
Seniority
Senior Staff, hands-on IC with leadership responsibilities
