Data Engineer (Databricks) | SR
Core
Design, build, and evolve scalable, high-performance data architectures and reliable data pipelines for batch and streaming environments.
Role type
Senior IC data engineer (Databricks/Lakehouse)
Builds
Robust Lakehouse architectures, optimized workloads, and secure, well-governed data ecosystems.
Domain
Cloud data platforms and data engineering
Deliverable
production ML models
Required skills
Databricks, PySpark, Spark SQL, Delta Lake, Delta Live Tables (DLT), Unity Catalog, Azure Data Factory (ADF), Python, Apache Spark, Airflow, Kafka, dbt, SQL, NoSQL databases, Git, Databricks Workflows
Preferred skills
REST APIs, microservices, automation, observability, monitoring, production pipeline support, technical leadership
Technologies
Databricks, Apache Spark, PySpark, Delta Lake, Unity Catalog, Azure Data Factory, Airflow, Kafka, dbt, Git
Responsibilities
Define and evolve data architectures using Databricks; Build, maintain, and optimize scalable, reliable data pipelines; Design and implement Lakehouse architectures and Medallion structures; Develop batch and streaming data processing solutions; Build and orchestrate data ingestion and integration processes; Implement data governance, security, lineage, and access control practices; Develop REST APIs, integrations, microservices, and automation solutions; Establish CI/CD processes, version control, and automated deployments; Optimize Spark performance, compute resource allocation, and processing costs; Monitor and maintain production data pipelines; Act as a technical reference for the team, mentoring professionals and promoting engineering standards and best practices.