CareerPlan

Principal Software Engineer, Applied AI – Splunk Observability Cloud

Seattle, Washington, USFull-time$241k–$323k2026-11-18 → 2026-11-20

Core

Architecting agentic investigation systems and evaluation frameworks for Splunk Observability Cloud to enable autonomous detection and code-level remediation of production incidents.

Role type

Principal Software Engineer (Applied AI)

Builds

Intelligent AI SRE experience including AI Assistant and Troubleshooting Agent for metrics, traces, logs, and infrastructure.

Domain

Observability, SRE, Applied AI, Telemetry

Deliverable

production ML models

Required skills

Production-grade LLM system design, Agentic system architecture with tool use, AI/ML evaluation framework engineering, Retrieval and context pipeline engineering, Latency optimization, Cost management, Multi-tenancy, Security guardrails

Preferred skills

Observability/SRE domain expertise, Kubernetes, Open-source contributions in AI/agent frameworks, Customer-zero deployment experience

Technologies

LLMs, Agentic frameworks, Telemetry databases, Kubernetes, OpenTelemetry, MCP

Responsibilities

Define technical trajectory for agentic investigation, Architect critical evaluation systems and retrieval layers, Establish production standards for latency, cost, and security, Mentor team members and partner with chief architects, Drive long-term technical direction for AI systems.

Seniority

Principal, strategy & mentorship