Data Processing Engineer - I/O
Core
Design and implement high-performance data read/write interfaces and adapters for a large-scale distributed data processing engine.
Role type
Senior IC data processing engineer (I/O subsystems)
Builds
Data processing engine components, adapters, connectors, and data format handlers
Domain
Distributed systems, large-scale data processing, storage formats
Required skills
C, C++, Rust, Linux, Apache Parquet, Apache ORC, Apache Iceberg, Apache Spark, Apache Flink, Presto/Trino, performance optimization, system architecture
Preferred skills
Rust, open table formats, catalog services
Technologies
Apache Spark, Apache Flink, Apache Parquet, Apache ORC, Apache Iceberg, Apache Arrow, Rust, C++, C, Linux
Responsibilities
Architect interfaces with data sources/sinks and catalogs; Design functional and performance enhancements for adapters/connectors; Design, implement, test, optimize, and maintain engine components; Analyze technology roadmaps for emerging data formats; Partner with product management on the data I/O roadmap; Foster best practices in design, code reviews, and CI/CD.
Seniority
Senior, hands-on IC
