Staff Software Engineer, Data Governance & Foundations
Core
Architecting and delivering the open lakehouse foundation, governance, and multi-engine compute strategy for Instacart's data ecosystem.
Role type
Staff Software Engineer, Data Infrastructure (Data Governance & Foundations)
Builds
Unified table formats, storage governance, and a multi-engine compute portfolio (interactive, batch, streaming) enabling portability and preventing lock-in.
Domain
Grocery retail / Data Infrastructure / Distributed Systems
Deliverable
production ML models | infrastructure
Required skills
Distributed systems architecture, Open table formats (Apache Iceberg, Delta Lake, Hudi), Distributed query/compute engines (Trino, Spark, ClickHouse), Event-driven/streaming infrastructure (Kafka, Flink), Platform migration strategy, Cost/benefit and TCO modeling, Architecture documentation, Technical leadership, Cross-org alignment
Preferred skills
Platform-level governance controls, Compliance frameworks (SOX, CPRA, GDPR), FinOps, SQL, Python/Scala, Orchestration (Airflow), Data transformation (dbt)
Technologies
Apache Iceberg, Apache Flink, Trino, ClickHouse, Apache Kafka, Apache Spark, Snowflake, Databricks, Confluent, Airflow, dbt, Delta Lake, Scala, Python, Postgres, AWS
Responsibilities
Translate data strategy into an actionable multi-year architecture roadmap; Own the open lakehouse foundation and multi-engine compute portfolio; Drive real-time and streaming infrastructure for critical use cases; Pioneer AI-native data infrastructure engineering; Elevate engineering excellence through architecture reviews and mentorship
Seniority
Staff, hands-on IC with strategic leadership