AI & Data Engineer
Core
Architecting the infrastructure for an LLM-queryable Knowledge Base by transforming unstructured testing data into intelligent, queryable insights for a crowdsourcing platform.
Role type
Senior IC AI & Data Engineer (GenAI focus)
Builds
A scalable data stack and RAG systems on AWS to power a knowledge base for product testing and insights.
Domain
SaaS / Crowdsourcing / Generative AI
Deliverable
production ML models | infrastructure
Required skills
RAG system design, vector databases, embedding models, AWS CDK, data pipeline orchestration, metadata modeling, chunking strategies, retrieval optimization, cloud security (IAM/KMS/VPC)
Preferred skills
Serverless architectures, cost-optimized scaling, CI/CD pipelines, monitoring and alerting
Technologies
AWS (S3, Glue, Athena, Lambda, Step Functions, Bedrock, SageMaker), OpenSearch, Pinecone, pgvector, LangChain, OpenAI, Cohere, Amazon Titan
Responsibilities
Design and implement data ingestion and normalization pipelines from heterogeneous sources; Build a data lake on AWS and orchestrate data flows; Implement RAG systems using vector databases and LLM models; Model metadata and define chunking strategies for NLU-queryable documents; Ensure data security, governance, monitoring, and cost optimization; Collaborate with the Product team to integrate the knowledge base into the existing platform.
Seniority
First hire, full ownership over architecture, implementation, and scalability