Principal Software Engineer - Observability & Telemetry Data
Core
Architect and own the end-to-end design of Smartsheet's unified telemetry data platform, enabling collection, modeling, storage, and analysis of metrics, logs, traces, and AI/agentic telemetry at global scale.
Role type
Principal Software Engineer (Observability & Telemetry Data)
Builds
A durable, portable, company-owned telemetry data platform and analytics layer for reliability, cost, and incident metrics.
Domain
Cloud Infrastructure, Observability, Data Engineering, AI/ML Telemetry
Deliverable
production ML models | product features | dashboards & analysis | infrastructure
Required skills
Distributed systems architecture, Telemetry data engineering, OpenTelemetry, Databricks, OpenTelemetry, Pipeline engineering, AWS, Kubernetes, Go/Java/Python/Scala, SQL, Infrastructure as Code, Architectural influence
Preferred skills
LLM/Agentic system instrumentation, Data reliability engineering, Telemetry cost engineering, Regulated environment experience
Technologies
Databricks, MLflow, OpenTelemetry, FluentBit, Kinesis, Kafka, AWS (EKS, ECS, EC2, Lambda, CloudWatch), Terraform, Flux, ArgoCD, Delta Lake, Apache Iceberg, Spark
Responsibilities
Architect the telemetry data platform and define semantic conventions; Lead migration to OpenTelemetry-based instrumentation; Integrate observability with Databricks and MLflow for AI telemetry; Build analytics layer for reliability and cost metrics; Engineer high-volume collection and routing pipelines; Set telemetry cost architecture and governance; Partner with Data and AI Platform teams; Mentor engineers on telemetry design; Participate in production support and on-call rotation.
Seniority
Principal, hands-on IC with strategic influence
