Senior Product Software Engineer - AI Quality Engineering
Core
Architect, build, and maintain evaluation systems and reliability practices for enterprise-scale generative AI applications (RAG, LLM orchestration) to ensure trustworthiness, safety, and production readiness in tax and accounting workflows.
Role type
Senior IC AI Quality Engineer (LLMOps)
Builds
Automated evaluation harnesses, regression suites, quality gates, and observability dashboards for non-deterministic AI systems.
Domain
Enterprise AI / Generative AI / Tax & Accounting
Deliverable
production ML models
Required skills
Python test automation, RAG system validation, LLM evaluation metrics, CI/CD pipeline integration, observability, root-cause analysis, automated testing frameworks, API validation, cloud-native delivery
Preferred skills
LLM observability platforms (LangSmith, Langfuse), vector database testing, LLM frameworks (LangChain, LlamaIndex), Azure AI services, containerization (Docker, Kubernetes), performance testing tools, LLMOps practices
Technologies
Python, Pytest, LangChain, LangSmith, Azure OpenAI, Pinecone, Elasticsearch, Docker, Kubernetes, Grafana, Prometheus, Datadog
Responsibilities
Design automated regression suites for AI quality drift; define measurable quality gates for AI releases; lead root-cause analysis for AI failures; build observability dashboards for model behavior; mentor engineers in AI evaluation methods; validate AI system architecture and service integrations.
Seniority
Senior, hands-on IC
