Senior Software Engineer, Apple Data Platform
Core
Build Apache DataFusion Comet, an open-source accelerator for Apache Spark, to improve distributed query engine performance and efficiency.
Role type
Senior IC distributed systems engineer (query engine optimization)
Builds
Apache DataFusion Comet (open-source accelerator for Apache Spark)
Domain
Big Data, Distributed Systems, Query Engines
Deliverable
production ML models | product features
Required skills
Apache Spark internals, Apache DataFusion, Apache Arrow, Apache Iceberg, Apache Parquet, distributed systems design, native language development, performance profiling, memory optimization, I/O optimization, cloud service scaling
Preferred skills
columnar/vectorized query execution, Kubernetes, Spark connectors to external data sources
Technologies
Apache Spark, Apache DataFusion, Apache Arrow, Apache Iceberg, Apache Parquet, Kubernetes
Responsibilities
Design and implement operators and expressions in native language on Apache DataFusion and Apache Arrow; extend Comet's Spark integration across versions; accelerate reads and writes of Apache Iceberg and Apache Parquet data; profile and benchmark real workloads; tune memory, shuffle, and I/O; contribute upstream to open-source communities.
Seniority
Senior, hands-on IC
