Sr. Software Engineer - Ingestion Core team
Core
Build distributed platform systems to ingest high-volume, petabyte-scale structured and unstructured data from diverse sources into Delta Lake, optimizing for throughput, latency, and cost.
Role type
Senior Software Engineer (Distributed Systems & Data Ingestion)
Builds
Distributed infrastructure for streaming ingestion, incremental processing, replication, and Agentic SDKs with evaluations.
Domain
Cloud Data Platforms, Distributed Systems, Data Engineering
Deliverable
production ML models | product features | infrastructure
Required skills
Java, Scala, Go, C++, or Python; Large-scale distributed system architecture; Streaming data processing; Spark; Database integration; CDC implementation; Observability design; AI/ML evaluation techniques
Preferred skills
Experience with RAG and AI agents; SDK development
Technologies
Delta Lake, Spark, SQS, ADLS, GCS, Oracle, SQL Server, MySQL, Postgres, JSON, Parquet, CSV
Responsibilities
Design and optimize streaming and distributed workloads for scale and reliability; Reduce end-to-end latency and costs; Build monitoring and observability capabilities; Collaborate on use cases for RAG and AI agents; Build Agentic SDKs with strong evaluations
Seniority
Senior, hands-on IC
