Databricks Engineer - Senior/ Lead
Core
Design, build, and optimize scalable data pipelines and analytics solutions on the Azure cloud platform using Databricks.
Role type
Senior/Lead Data Engineer (Azure Databricks)
Builds
Scalable ETL/ELT workflows, Delta Lake tables, and data pipelines for analytics and ML use cases.
Domain
Cloud Data Engineering (Azure)
Deliverable
production ML models
Required skills
Azure Databricks, Python, SQL, Apache Spark/PySpark, Delta Lake, Azure Data Lake Storage (ADLS Gen2), Git, Azure Data Factory, CI/CD pipelines, distributed computing concepts, streaming data (Spark Structured Streaming, Event Hub, Kafka), data modeling, cloud security and RBAC.
Preferred skills
Experience with large datasets, modern data engineering practices, data quality validation and monitoring. (via careerplan.io/jobs/744000153721909-databricks-engineer-senior-lead-at-blend360)
Responsibilities
Design and maintain scalable data pipelines; Implement ETL/ELT workflows; Optimize Spark jobs for performance, cost, and scalability; Work with structured and semi-structured data; Build and manage Delta Lake tables; Integrate Databricks with Azure Data Lake Storage; Develop complex queries and transformations; Collaborate with data scientists, analysts, and stakeholders; Ensure data quality, validation, and monitoring; Follow best practices for security, access control, and governance.
Seniority
Senior/Lead, hands-on IC