Senior Data Engineer with Python and Databricks
Core
Design, implement, and maintain scalable data pipelines for ingestion, processing, and transformation using cloud services to support data scientists and analysts.
Role type
Senior Data Engineer
Builds
Data pipelines and data solutions in Azure
Domain
Cloud data engineering, Data Warehousing
Deliverable
production ML models
Required skills
Python, SQL, Azure Data Factory, Azure Synapse, Databricks/Spark, Polars, Pandas, ETL/ELT, Data Modeling (Star Schema/Kimball), Relational/NoSQL databases, Event-Driven Architecture
Preferred skills
Data monitoring and observability tools
Technologies
Azure, Data Factory, Synapse, Databricks, Spark, Polars, Pandas, Fabric
Responsibilities
Create and maintain data solutions in Azure; Implement data validation and cleansing procedures; Improve scalability, efficiency, and cost-effectiveness of pipelines; Monitor and resolve data pipeline problems.
Seniority
Senior, hands-on IC