Software Development Engineer in Test - AI
Core
Design advanced evaluation frameworks for LLMs, AI agents, and RAG-based applications to ensure quality, reliability, and safety of AI outputs.
Role type
Senior IC Software Development Engineer in Test (AI)
Builds
Automated validation pipelines, AI evaluation frameworks, and testing solutions for intelligent systems
Domain
Artificial Intelligence / Machine Learning / Quality Engineering
Deliverable
production ML models
Required skills
Python, AI/LLM testing, Prompt engineering, RAG architecture validation, AI agent frameworks, API testing, CI/CD integration, Cloud platforms (AWS/Azure/GCP/OCI), Vector databases, Docker, Kubernetes
Preferred skills
Robot Framework, Playwright, AI development tools (Cursor/Copilot/Claude/Gemini), MLOps, AI red teaming, Agent reliability testing, Cost optimization
Technologies
LangChain, LangGraph, CrewAI, AutoGen, Semantic Kernel, OpenAI SDKs, Pinecone, Weaviate, ChromaDB, Qdrant, FAISS, GitHub Actions, Jenkins, GitLab CI/CD, Azure DevOps, SQL
Responsibilities
Design and implement testing strategies for AI-powered applications and autonomous agents; Develop AI evaluation frameworks to assess hallucinations, bias, and safety; Build automated validation pipelines using semantic scoring and embeddings; Validate multi-agent orchestration workflows and tool calling; Optimize AI application latency, token consumption, and costs; Integrate automated testing into CI/CD pipelines; Monitor AI application behavior using observability platforms
Seniority
Senior, hands-on IC