AI Data Engineer
Core
Building and scaling data pipelines for AI services that evaluate generated test quality and model student knowledge levels.
Role type
Senior Data Engineer (AI/ML infrastructure)
Builds
Data pipelines, offline evaluation systems, and storage architecture for AI/ML inference
Domain
Education technology, AI/ML infrastructure
Deliverable
production ML models
Required skills
Python, SQL, Data pipeline orchestration (Airflow/Dagster/Prefect), Kubernetes, Cloud data platforms (Google Cloud/BigQuery), Data modeling (lakehouse/medallion), Database scaling (partitioning/indexing/replication), Data quality validation
Preferred skills
Spark/PySpark, ML model integration (batch/LLM inference), Delta Lake/Apache Iceberg, dbt, Vector/Graph databases (Qdrant/Neo4j), Terraform, CI/CD, Observability (Prometheus/Grafana), Statistics
Technologies
Python, SQL, Airflow, Dagster, Prefect, Kubernetes, Helm, Google Cloud, BigQuery, GCS, Spark, PySpark, vLLM, Delta Lake, Apache Iceberg, dbt, Qdrant, Neo4j, Terraform, GitLab CI, Prometheus, Grafana, Arize Phoenix
Responsibilities
Design and build data pipelines from raw data to datasets and metrics for AI services; Develop offline evaluation pipelines for generated test quality; Integrate ML models and LLM inference into data processing pipelines; Design data storage architecture for scaling to thousands of schools; Align data standards with cross-functional teams
Seniority
Senior, hands-on IC
