Backend AI & Data Pipeline Engineer
Core
Design and maintain scalable, event-driven data processing pipelines that generate semantic embeddings and feed a knowledge graph for personalized career pathway recommendations.
Role type
Backend AI & Data Pipeline Engineer
Builds
Scalable data pipelines, semantic embeddings, knowledge graphs, and matching APIs
Domain
EdTech / JobTech / Recommendation Systems
Deliverable
production ML models | infrastructure
Required skills
Python, AWS serverless (Lambda, Fargate, EventBridge, Step Functions), data pipeline design, vector databases, semantic similarity search, cost-conscious infrastructure, HTML scraping
Preferred skills
Knowledge graphs, association rule mining (FP-Growth), LLM re-ranking, edtech/jobtech domain experience
Technologies
AWS Lambda, Amazon Bedrock, MongoDB Atlas Vector Search, FP-Growth, HTML scraping
Responsibilities
Design and maintain three distinct processing pipelines (ingestion, event-driven, knowledge graph builder), generate and manage semantic embeddings, build knowledge graphs linking jobs/courses/skills, improve discovery and matching API, right-size Fargate Spot instances, maintain job and institution scrapers
Seniority
Junior, hands-on IC
