Machine Learning Engineer
Core
Build and scale distributed data pipelines for ingesting, processing, and delivering large-scale video and multimodal data for model training.
Role type
Senior IC machine learning engineer (data infrastructure & curation)
Builds
Production data pipelines, dataset curation systems, and VLM-based captioning workflows for social AI models
Domain
Social AI, video processing, multimodal learning
Deliverable
production ML models
Required skills
distributed data processing, workflow orchestration, container orchestration, cloud data storage optimization, VLM-based captioning, semantic filtering, video processing, Python
Preferred skills
dataset versioning, quality scoring models, deduplication pipelines, image-text pair curation
Technologies
PySpark, Ray, Airflow, Docker, Kubernetes, AWS, GCS, Azure, FFmpeg, PyAV, DALI, OpenCV, Decord, torchvision, PyTorchVideo, torchaudio, CLIP
Responsibilities
Design and scale distributed data pipelines for preprocessing and dataset generation; Own workflow orchestration, job scheduling, monitoring, and failure recovery; Implement containerized pipeline infrastructure; Optimize cloud-based data storage and movement; Define and implement best practices for dataset storage layout and versioning; Design and implement curation pipelines for video and image content selection; Build and improve VLM-based captioning and metadata generation workflows; Develop and apply quality and aesthetic scoring models for data selection; Build tooling to support deduplication workflows at scale; Analyze dataset composition and iterate on curation logic
Seniority
Senior, hands-on IC