Staff AI & Data Engineer - Databricks
Core
Design and build production-grade data pipelines, medallion architectures, and AI-ready data products on Databricks to support enterprise analytics and AI agents.
Role type
Staff AI & Data Engineer
Builds
Scalable batch/streaming data pipelines, Bronze/Silver/Gold medallion architectures, AI-ready data products, semantic layers, and agent-ready tools.
Domain
Cloud Data Engineering & Generative AI
Deliverable
production ML models
Required skills
Databricks, Apache Spark, Delta Lake, SQL, Python, AWS (EMR, Bedrock), CI/CD, Git, Unity Catalog, Dimensional Modeling, Data Quality, Observability, AI Governance, MCP, Agent Tool Calling
Preferred skills
Databricks Data Engineer Professional certification, Databricks AI/BI/Genie/MLflow, Real Estate/Finance/Operations domain experience, Metadata-driven ingestion frameworks
Technologies
Databricks, Apache Spark, Delta Lake, AWS, EMR, Bedrock, Unity Catalog, MLflow, Git, CI/CD, MCP
Responsibilities
Design and build production-grade data pipelines on Databricks; Ingest data from enterprise applications, databases, APIs, SaaS platforms, files, and other sources; Build and maintain Bronze, Silver, and Gold medallion architectures; Implement deduplication, historical tracking, schema evolution, and data lineage; Optimize pipelines for reliability, performance, scalability, and cost; Develop a configuration-driven source onboarding framework and Lakeflow pipeline templates; Automate source registration, deployment, monitoring, lineage, and operational controls; Establish data quality expectations, anomaly detection, monitoring, and quarantine processes; Set up job and pipeline monitoring, alerts, audit logging, and operational dashboards; Support automated deployment and promotion through Databricks Asset Bundles and CI/CD; Develop conformed dimensions and shared-key strategies across domains; Create integrated Gold-layer products and reusable data contracts; Build AI-ready Gold data products with ownership, lineage, definitions, and quality information; Develop semantic context, business glossaries, ontologies, and certified KPIs in Unity Catalog; Create governed Unity Catalog functions and agent-ready tools and endpoints; Implement tracing, evaluation, testing, monitoring, and governance for AI and agent workflows; Support development of AI agents that use trusted enterprise data.
