Staff Software Engineer - Streaming
Core
Drive the technical direction and architecture of Spark Structured Streaming, building state-of-the-art stream processing capabilities for the Databricks Data Intelligence Platform.
Role type
Staff Software Engineer (Distributed Systems & Stream Processing)
Builds
Apache Spark Structured Streaming engine, state management systems, and new operators for the Databricks Data Intelligence Platform
Domain
Big Data, Distributed Systems, Database Internals
Deliverable
production ML models | product features
Required skills
Distributed systems architecture, Apache Spark internals, Database internals, State management design, Performance optimization, Language design, System design for high throughput/low latency, Technical vision setting, Architecture decision making
Preferred skills
Experience with open source contributions, Cross-functional product leadership, Mentorship of senior engineers
Technologies
Apache Spark, Databricks Data Intelligence Platform
Responsibilities
Set and drive the technical vision for Spark Structured Streaming across OSS and the platform; Design and build core engine capabilities including state management and new operators; Improve engine architecture for latency, throughput, and cost; Guide long-term architecture and technical-debt decisions; Mentor and technically lead engineers on the team.
