Principal Software Engineer, Applied AI – Splunk Observability Cloud
Core
Architecting agentic investigation systems and evaluation frameworks for Splunk Observability Cloud to enable autonomous detection and code-level remediation of production incidents.
Role type
Principal Software Engineer (Applied AI)
Builds
Intelligent AI SRE experience including AI Assistant and Troubleshooting Agent for metrics, traces, logs, and infrastructure.
Domain
Observability, SRE, Applied AI, Telemetry
Deliverable
production ML models
Required skills
Production-grade LLM system design, Agentic system architecture with tool use, AI/ML evaluation framework engineering, Retrieval and context pipeline engineering, Latency optimization, Cost management, Multi-tenancy, Security guardrails
Preferred skills
Observability/SRE domain expertise, Kubernetes, Open-source contributions in AI/agent frameworks, Customer-zero deployment experience
Technologies
LLMs, Agentic frameworks, Telemetry databases, Kubernetes, OpenTelemetry, MCP
Responsibilities
Define technical trajectory for agentic investigation, Architect critical evaluation systems and retrieval layers, Establish production standards for latency, cost, and security, Mentor team members and partner with chief architects, Drive long-term technical direction for AI systems.
Seniority
Principal, strategy & mentorship