Principal Data Processing Engineer-OSS
Core
Design, implement, and optimize high-performance data processing engine components and open-source platforms (Apache Gluten, Velox, Spark, Flink) for large-scale analytics and GenAI preprocessing.
Role type
Principal Data Processing Engineer (Individual Contributor)
Builds
Data processing engine components and integrations with open-source platforms
Domain
Distributed systems, data processing, compilers, cloud-native computing
Deliverable
production ML models
Required skills
C, C++, Java, Apache Spark, Apache Flink, Linux, performance optimization, distributed systems architecture
Preferred skills
Apache Gluten, Velox, Presto/Trino, hardware-accelerated computing
Technologies
Apache Gluten, Velox, Apache Spark, Apache Flink, C, C++, Java, Linux
Responsibilities
Influence engine architecture interfaces with OSS platforms; Lead design of functional and performance enhancements to OSS platforms; Design, implement, test, and maintain engine components; Analyze technology roadmaps to identify leadership opportunities; Partner with engineering, product, and community teams; Foster best practices in design, code reviews, and CI/CD.
Seniority
Principal, hands-on IC with mentorship