Senior Software Engineer, Big Data
Core
Design, build, deploy, and maintain data ingestion pipelines, batch and streaming big data processing systems, and MLOps frameworks for ML-powered products.
Role type
Senior IC big data and MLOps engineer
Builds
Data lakes, event-stream consumers, MLOps frameworks, and data exploration tools (Sift)
Domain
Big Data, Machine Learning Operations, Distributed Systems
Deliverable
production ML models
Required skills
Apache Spark, distributed systems, containerized services, data structures, algorithms, data-intensive systems, workflow orchestration
Preferred skills
AWS services (S3, IAM, EC2, MSK), Parquet, object storage, AWS Glue, Apache Iceberg, Delta Lake, Kafka, Apache Flink, Spark Structured Streaming, Apache Airflow, Kubernetes, Docker
Technologies
Apache Spark, Kafka, Apache Flink, AWS Glue, Apache Iceberg, Delta Lake, Kubernetes, Docker, Apache Airflow
Responsibilities
Design and maintain data ingestion pipelines for web events and data lakes; Develop batch and streaming big data processing systems; Build and maintain MLOps frameworks; Build and maintain data documentation frameworks; Build and maintain Sift, an agentic data exploration tool; Build and maintain data platforms supporting streaming, batch ingestion, and data discovery
Seniority
Senior, hands-on IC
