Software Engineer
Core
Building advanced data pipelines, foundational datasets, and infrastructure to support AI researchers in processing petabytes of anime image and video data.
Role type
Senior generalist software engineer (data infrastructure)
Builds
Scalable data pipelines, data stores, and infrastructure for AI model ingestion
Domain
Generative AI, Anime, Video Games, Big Data
Deliverable
production ML models | infrastructure
Required skills
Distributed systems, Big data toolsets (Ray, Spark, Airflow), Cloud platforms (AWS, GCP, Azure), Data stores (Snowflake, Redshift, Hive, SQL/NoSQL), Scripting (TypeScript, Python, Bash), Complex SQL queries, Large-scale data transfer
Preferred skills
Experience building large-scale big data pipelines, Interest in interfacing with AI researchers, Knowledge of anime aesthetic
Technologies
Ray, Spark, Airflow, AWS, GCP, Azure, Snowflake, Redshift, Hive, Bigquery, S3, TypeScript, Python, Bash
Responsibilities
Define processes and infrastructure to transform and make data readily available across the company, Design, build, and monitor pipelines that scale with growing data size, Partner with AI researchers to understand needs and implement technical solutions
Seniority
Senior, hands-on IC
