Data Engineer
Core
Design, develop, and maintain scalable ETL/ELT data pipelines and large-scale data-processing solutions using Azure Databricks and Apache Spark.
Role type
Senior Data Engineer (Cloud/Big Data)
Builds
Production-grade data pipelines, batch processing solutions, and data ingestion workflows for structured and semi-structured data.
Domain
Cloud Data Engineering (Azure)
Deliverable
production ML models
Required skills
Azure Databricks, PySpark, Apache Spark, SQL, Python, ETL/ELT, Delta Lake, Azure Data Factory, CI/CD, Git, Data Quality, Performance Optimization
Preferred skills
None stated
Technologies
Azure Databricks, PySpark, Apache Spark, Delta Lake, Azure Data Lake Storage, Azure Data Factory, Git
Responsibilities
Design and maintain scalable ETL/ELT pipelines; Build and optimize batch data-processing solutions; Develop data ingestion and transformation workflows; Implement reliable data-processing frameworks using Delta Lake; Perform data cleansing, validation, and quality checks; Optimize Spark jobs, notebooks, and clusters for performance; Troubleshoot pipeline failures and performance bottlenecks; Implement incremental processing and partitioning techniques; Develop reusable data-engineering components; Implement version control and automated deployment; Collaborate with data architects and stakeholders; Maintain technical documentation; Ensure compliance with security and governance standards.
Seniority
Mid-Senior (4-6 years experience)