Senior Lead Data Engineer - Databricks

Description

We're on the lookout for exceptional individuals to join our team as Senior Lead Data Engineers – Databricks. As part of our team, you'll play a key role in designing and delivering modern, scalable data platforms for clients across multiple industries, leveraging Azure and Databricks technologies.

You will lead the design, development, and optimization of end-to-end data engineering solutions built on modern Lakehouse architectures. Working closely with clients and cross-functional teams, you'll translate business requirements into scalable, high-performance data platforms while mentoring engineers and driving technical excellence.

• Responsibilities
• Lead the design, development, and maintenance of scalable data engineering solutions using Azure and Databricks.
• Design and implement modern Lakehouse architectures following industry best practices.
• Build and optimize large-scale batch and streaming data pipelines.
• Develop ETL/ELT pipelines using Databricks, PySpark, SQL, and Azure Data Factory.
• Design, implement, and optimize Delta Lake solutions for reliable and performant data processing.
• Collaborate with clients to gather requirements and translate business needs into technical solutions.
• Optimize Spark workloads for performance, scalability, and cost efficiency.
• Design and maintain enterprise-grade Data Warehouses, Data Lakes, and Lakehouses.
• Implement data governance, security, and metadata management using Azure services such as Purview, Key Vault, and Microsoft Entra ID (Azure Active Directory).
• Establish CI/CD pipelines and DevOps practices for data engineering workloads.
• Mentor and coach junior data engineers while promoting engineering best practices.
• Participate in architecture discussions, solution design, and technical leadership initiatives.
• Contribute to internal knowledge sharing and continuous improvement.

•
• Requirements
• Required
• 10+ years of professional experience in Data Engineering.
• 4+ years of hands-on experience with Databricks.
• Strong experience developing data pipelines using PySpark and Apache Spark.
• Strong experience designing and implementing Lakehouse architectures.
• Experience working with Delta Lake and Medallion Architecture.
• Strong knowledge of SQL and Python.
• Experience with Azure Data Factory.
• Experience working with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Experience with Azure Synapse Analytics.
• Experience with Data Warehousing concepts and dimensional modeling.
• Strong understanding of ETL and ELT design patterns.
• Experience optimizing Spark jobs, partitioning strategies, joins, caching, and workload performance.
• Experience with Git and Azure DevOps CI/CD pipelines.
• Experience implementing secure data platforms using Azure Key Vault, Microsoft Entra ID, and Microsoft Purview.
• Experience building scalable enterprise data platforms in Azure.
• Experience working directly with clients and gathering technical requirements.
• Experience mentoring engineers and leading technical initiatives.

•
• Strongly Preferred
• Experience with Snowflake.
• Experience building real-time streaming solutions using Structured Streaming, Kafka, or Event Hubs.
• Experience with Databricks Workflows.
• Experience with Delta Live Tables.
• Experience with Unity Catalog.
• Experience with Databricks SQL.
• Experience with MLflow.
• Experience with dbt.
• Experience with Airflow or other orchestration platforms.
• Experience working in Agile environments.

•
• Nice to Have
• MSc in Computer Science or a related field.
• Experience with DataOps practices.
• Experience with MLOps.
• Experience with Azure Machine Learning.
• Experience with Azure Cognitive Services.
• Experience with Docker and Kubernetes.
• Experience working with Microsoft Fabric.
• Experience with Power BI.

•
• Technical Skills
• Core Technologies
• Azure Databricks
• Apache Spark
• PySpark
• SQL
• Python
• Delta Lake
• Azure Data Factory
• Azure Synapse Analytics
• ADLS Gen2
• Snowflake

• Data Engineering
• Data Warehousing
• Data Lakes
• Lakehouse Architecture
• Medallion Architecture
• Dimensional Modeling
• ETL / ELT
• Batch Processing
• Streaming Data Pipelines
• Data Governance
• Performance Optimization

• Azure Ecosystem
• Azure DevOps
• Microsoft Purview
• Azure Key Vault
• Microsoft Entra ID
• Azure Functions
• Event Hubs

• Nice-to-Have Technologies
• Kafka
• Airflow
• dbt
• MLflow
• Docker
• Kubernetes
• Power BI

• Salary paid in USD
• Six-month career advancing opportunities
• Supportive and friendly work environment
• Premium medical insurance [employee +family]
• English language development courses
• Interest-free loans paid over 2.5 years
• Technical development courses
• Planned overtime program (POP)
• Employment referral program
• Premium location in Maadi
• Social insurance