Senior Data Engineer
Core
Design and operate scalable event-driven and batch-processing systems to onboard and enrich multimedia content (images, videos, music) for training datasets and customer-facing applications.
Role type
Senior IC data engineer (multimedia data pipelines)
Builds
Scalable metadata-processing systems, ingestion pipelines, and shared frameworks for data science workloads
Domain
AI/ML data infrastructure, multimedia content licensing
Deliverable
production ML models
Required skills
Python, distributed systems, event-driven architectures, batch processing, cloud infrastructure (AWS/GCP), data contracts, lineage, data quality, system design, cost estimation, mentoring
Preferred skills
Containerized ML workloads, event-driven orchestration, Snowflake/ClickHouse/Iceberg, image/video/audio processing
Technologies
AWS, GCP, SageMaker, AWS Batch, ECS, Cloud Batch, GKE, EKS, Step Functions, Lambda, EventBridge, Kinesis, Workflows, Pub/Sub, Snowflake, ClickHouse, Apache Iceberg
Responsibilities
Implement architectural direction for metadata-processing systems; extend orchestration for enrichment and backfill workloads; evolve shared frameworks for data science; define and maintain metadata contracts and quality controls; estimate workload cost and capacity; partner with platform teams on cloud infrastructure; deliver time-sensitive customer systems
Seniority
Senior, hands-on IC
