Staff Software Engineer, Lakeflow Pipelines DR
Core
Design and implement distributed systems to replicate and recover Lakeflow pipelines across regions, ensuring consistency and safe recovery without data loss.
Role type
Staff Software Engineer (Distributed Systems & Reliability)
Builds
Resilient streaming and batch ETL pipelines, streaming tables, and materialized views for the Lakehouse platform
Domain
Cloud data infrastructure, distributed systems, disaster recovery
Deliverable
production ML models | product features
Required skills
Distributed systems design, consistency models, transaction management, idempotency, replication, checkpointing, failover/failback logic, data lineage, production-grade software engineering
Preferred skills
Deep clone implementation, metadata reconciliation, observability engineering, failure-injection testing, high-fidelity recovery simulation, formal reasoning about failure modes
Technologies
Java, Scala, C++, Go, Python
Responsibilities
Design cross-region replication and recovery strategies for pipelines and materialized views; Implement distributed consistency mechanisms across pipeline dependencies and transaction logs; Develop failover and failback workflows with correctness guardrails; Build deep clone and metadata reconciliation tools; Create observability and failure-injection testing frameworks; Conduct high-fidelity recovery simulations and game-day testing
Seniority
Staff, hands-on IC with multi-year technical vision
