Technical Lead, Spark
Core
Design and build core components for Cloudera's data platform, specifically the Cloudera distribution of Apache Spark and Livy, enabling enterprise-grade data storage, management, and processing at massive scale.
Role type
Senior Staff Software Engineer (Spark/Java)
Builds
Enterprise-grade distributed data processing systems for customers running Spark on thousands of nodes
Domain
Big Data / Distributed Systems / Data Engineering
Deliverable
production ML models | product features
Required skills
Distributed systems expertise, Systems design, Java/Scala/Python proficiency, SQL planners, Data layout and table formats (Parquet, Iceberg), Fault tolerance, Root cause analysis, Large-scale cluster management
Preferred skills
Apache Spark/Livy development, Cloud platforms, SQL optimizers
Technologies
Apache Spark, Livy, Java, Scala, Python, Apache Parquet, Apache Iceberg
Responsibilities
Design new features for the data engineering experience, Contribute to Apache Spark and Livy, Develop features in Scala/Java/Python, Debug system-level deployment issues, Work on improving internal infrastructure
Seniority
Senior Staff, hands-on IC with leadership responsibilities
