Software Engineer - Data Ingestion
Core
Build high-throughput ingestion and processing pipelines, efficient storage on object stores, and systems for data governance and usage reporting.
Role type
Senior IC software engineer (distributed systems & data ingestion)
Builds
Cloud-native, production systems for data ingestion, processing, and storage
Domain
Observability, distributed systems, data engineering
Required skills
Distributed systems design, Apache Kafka, Rust, Scala, Postgres, Redis, Object storage, Database design, Production operations
Preferred skills
Kafka Streams, Column-oriented data formats, Apache Arrow, DataFusion
Technologies
Rust, Scala, Kafka, Postgres, Redis, Kubernetes, Object storage
Responsibilities
Develop and operate distributed systems in production, Build Kafka-based ingestion and processing pipelines, Design systems for data governance, retention, deletion, and usage reporting, Work with Postgres and Redis, Implement efficient persistence using column-oriented data formats and object storage
Seniority
Senior, hands-on IC
