Senior Platform Data Engineer
Core
Building and maintaining the shared data platform, retrieval infrastructure, and feature pipelines that enable clinical AI and generative applications at scale.
Role type
Senior Platform Data Engineer (AI Infrastructure)
Builds
Real-time clinical data feeds, reusable feature tables, RAG retrieval infrastructure, and vector database systems for healthcare AI.
Domain
Healthcare / Clinical Data Engineering / Generative AI Infrastructure
Deliverable
production ML models | infrastructure
Required skills
Databricks (Delta Live Tables, PySpark, Unity Catalog, Feature Store), real-time streaming (Kafka, Spark Structured Streaming), SQL, Python (pandas), vector database administration, embedding model tuning, data governance, clinical data modeling.
Preferred skills
Epic data extraction (SDE, FHIR, epic-ws), healthcare domain knowledge, chunking strategies for clinical documents.
Technologies
Databricks, Delta Live Tables, PySpark, Unity Catalog, Feature Store, Kafka, Spark Structured Streaming, Pinecone, Weaviate, Qdrant, Databricks Vector Search, SQL, pandas, Epic SDE, FHIR.
Responsibilities
Stream clinical data from EHR sources into Databricks; curate shared clinical feature tables; design and operate RAG infrastructure and document ingestion pipelines; manage embedding models and vector databases; build retrieval pipelines with hybrid search and reranking; establish data quality gates and monitor retrieval metrics; administer Databricks workspaces and access controls.
Seniority
Senior, hands-on IC
