Databricks Data Specialist - R01569707
Core
Design, develop, and optimize scalable data engineering and analytics solutions on the Databricks Lakehouse Platform, including building batch and real-time data pipelines and implementing Lakehouse architectures.
Role type
Senior IC Databricks Data Engineer
Builds
End-to-end data pipelines, Lakehouse architectures (Bronze, Silver, Gold layers), Delta Lake solutions, and real-time data processing solutions.
Domain
Data Engineering / Cloud Data Platforms
Required skills
Databricks Platform, Delta Lake, Delta Live Tables (DLT), Unity Catalog, Databricks Workflows, PySpark, Apache Spark, Structured Streaming, Auto Loader, SQL, Lakehouse Data Modeling
Preferred skills
Azure Data Factory, Azure Synapse Analytics, AWS Glue, Apache Airflow, DBT, Apache Kafka, Power BI, BigQuery
Technologies
Databricks, PySpark, Apache Spark, Delta Lake, Unity Catalog, Structured Streaming, Auto Loader, Azure Data Factory, AWS Glue, Apache Airflow, Apache Kafka, Power BI, BigQuery
Responsibilities
Design and maintain end-to-end data pipelines using Databricks and PySpark; Build and implement Lakehouse architectures utilizing Bronze, Silver, and Gold data layers; Develop and manage Delta Lake solutions with ACID transactions; Create, monitor, and optimize Delta Live Tables (DLT) pipelines; Implement scalable data ingestion processes using Auto Loader; Develop and manage real-time data processing solutions using Structured Streaming; Orchestrate, schedule, and monitor data workflows using Databricks Workflows; Design and implement Lakehouse data models to support reporting and analytics; Establish and enforce data governance, security, and access controls using Unity Catalog; Optimize Spark jobs, SQL queries, and overall platform performance.
Seniority
Senior, hands-on IC