Python Developer - Databricks and Azure
Core
Senior Data Engineer designing and implementing scalable data and application solutions using Python, Spark, and Azure ML.
Role type
Senior hands-on IC data engineer
Builds
Production-ready ETL/ELT pipelines, data assets, and application solutions
Domain
Cloud data engineering (Azure, Databricks)
Deliverable
production ML models
Required skills
Python, Scala, SQL, Apache Spark, Databricks, Azure Cloud services, ETL/ELT pipeline design, CI/CD implementation, data governance, security, logging, monitoring, alerting, Agile methodologies
Preferred skills
Java, Spring application development, Unity Catalog, Jenkins, GitHub Actions, Azure DevOps
Technologies
Azure, Databricks, Spark, Python, SQL, Scala, Java, Spring, Unity Catalog, Jenkins, GitHub Actions, Azure DevOps
Responsibilities
Design and implement complex ETL/ELT pipelines for large-scale datasets; Optimize Python and Spark job performance and cost efficiency; Partner with data scientists and stakeholders to solve business challenges; Implement data governance, security, and compliance measures; Create and maintain DevOps CI/CD pipelines; Establish monitoring, logging, and alerting for production workloads; Support agile pods with project planning and sprint execution; Evaluate candidates through technical interviews.
Seniority
Senior, hands-on IC