Software Engineer, Big Data
Core
Design, build, deploy, and maintain data ingestion pipelines, datalake systems, and MLOps frameworks for web events and downstream applications.
Role type
Senior IC big data and MLOps engineer
Builds
Data platforms supporting streaming and batch ingestion, ETL pipelines, data permissions, data discovery, and user-facing products
Domain
Big Data, Machine Learning Operations, Distributed Systems
Deliverable
production ML models
Required skills
Apache Spark, containerized services, distributed systems, data structures, algorithms, testing
Preferred skills
AWS services (S3, IAM, EC2, MSK), Parquet, AWS Glue, Apache Iceberg, Delta Lake, Kafka, Apache Flink, Spark Structured Streaming, Apache Airflow, Kubernetes, Docker
Technologies
Apache Spark, Kafka, Apache Flink, Apache Airflow, AWS, Kubernetes, Docker, Parquet, Delta Lake, Iceberg
Responsibilities
Design and build data ingestion pipelines and datalake systems; Develop batch and streaming big data processing systems; Build and operate MLOps frameworks; Create frameworks for data documentation and quality; Build and maintain Sift, an agentic data exploration tool
Seniority
Mid-level, hands-on IC
