Middle Data Engineer (Azure Databricks)
Core
Design and develop modern data pipelines and Lakehouse architectures on Azure Databricks to enable advanced analytics and business intelligence.
Role type
Middle Data Engineer (Azure Databricks)
Builds
Scalable cloud-based data architectures, data pipelines, and analytical layers.
Domain
Cloud Data Engineering / Big Data
Deliverable
production ML models | product features
Required skills
Azure Databricks, PySpark, Spark SQL, Delta Lake, Lakehouse architectures, Azure Data Factory, SQL Server, distributed data processing, big data concepts
Preferred skills
Spark optimization techniques, structured streaming in Databricks, CI/CD pipelines (Azure DevOps), Power BI, migrating from traditional ETL to cloud
Technologies
Azure Databricks, Spark, Azure Data Factory, Delta Lake, SQL Server, Azure DevOps, Power BI
Responsibilities
Design, develop, and maintain data pipelines using Azure Databricks; Build and optimize data transformations using PySpark and SQL; Implement and maintain Lakehouse architectures using Delta Lake; Develop ETL/ELT pipelines orchestrated through Azure Data Factory; Integrate data from multiple sources into the data platform; Design and maintain data models and data warehouse structures; Ensure data quality, scalability, and performance of large-scale data processing pipelines; Collaborate with BI teams to support Power BI and reporting platforms; Support and evolve existing SQL Server data platforms and ETL solutions.
Seniority
Middle, hands-on IC
