Principal Software Engineer
Core
Design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems to ensure quality, reliability, and safety before and after production deployment.
Role type
Senior IC AI Test / Automation Engineer
Builds
Automated test suites, evaluation systems, and CI/CD validation pipelines for AI systems
Domain
Artificial Intelligence / Machine Learning / Quality Assurance
Deliverable
production ML models
Required skills
Python, test automation frameworks (PyTest, Playwright, Selenium, Cypress), CI/CD tools (GitHub Actions, Jenkins, GitLab CI), cloud platforms (AWS, Azure, GCP), containers (Docker, Kubernetes), LLM ecosystems (OpenAI, Anthropic, Bedrock), RAG architectures, vector databases (Pinecone, Weaviate), agent frameworks (LangChain, LlamaIndex, AutoGen), non-deterministic testing, LLM-as-a-judge, BLEU/ROUGE/scoring, adversarial testing, bias/fairness validation, jailbreak detection
Preferred skills
AI testing & observability tools (LangSmith, TruLens, Arize, Weights & Biases), evaluation tools (DeepEval, Ragas, PromptFoo, Giskard), monitoring (Prometheus, Grafana, OpenTelemetry), shift-left testing, chaos/resilience testing
Technologies
Python, Bash, TypeScript, Go, PyTest, Playwright, Selenium, Cypress, GitHub Actions, Jenkins, GitLab CI, AWS, Azure, GCP, Docker, Kubernetes, OpenAI, Anthropic, Bedrock, Pinecone, Weaviate, LangChain, LlamaIndex, AutoGen, LangSmith, TruLens, Arize, Weights & Biases, DeepEval, Ragas, PromptFoo, Giskard, Prometheus, Grafana, OpenTelemetry
Responsibilities
Build and maintain AI test automation frameworks for pre-qualification and continuous validation; Develop comprehensive test suites including unit, integration, E2E, functional, regression, performance, and safety testing; Validate AI system behavior including non-deterministic LLM outputs, hallucinations, and edge cases; Design and manage evaluation systems with golden datasets and benchmarking pipelines; Automate testing within CI/CD pipelines for model updates and prompt changes; Implement observability and telemetry for traceability and audit readiness; Collaborate cross-functionally with ML, MLOps, Product, and Security teams; Track and report quality KPIs including test coverage and defect leakage; Drive root-cause analysis and continuous improvement across the AI testing lifecycle
Seniority
Senior, hands-on IC
