Senior Data Engineer, Observability Engineering
Core
Design, build, and operate scalable Databricks-based data pipelines and lakehouse solutions to ingest, transform, and govern high-volume telemetry, infrastructure, and security data for enterprise observability and operational intelligence.
Role type
Senior individual contributor data engineer (observability & security data)
Builds
Enterprise observability lakehouse, resilient ingestion frameworks, and trusted data products for security analytics
Domain
Healthcare technology, observability, infrastructure security, and data engineering
Deliverable
production ML models | product features | dashboards & analysis
Required skills
Python, PySpark, SQL, distributed data processing, Databricks, Delta Lake, Unity Catalog, Structured Streaming, data governance, CI/CD, cloud security principles
Preferred skills
OCSF framework familiarity, Databricks Asset Bundles, GitHub Actions, Azure DevOps, near real-time ingestion, observability platform support
Technologies
Databricks, PySpark, Delta Lake, Unity Catalog, Structured Streaming, Azure DevOps, GitHub Actions
Responsibilities
Design and deploy scalable Databricks pipelines for observability and security data; optimize PySpark workloads and streaming jobs; partner with security and infrastructure teams to onboard data sources and establish governance; contribute to engineering standards, code reviews, and platform optimization; promote best practices for data governance and cost-efficient operations
Seniority
Senior, hands-on IC