Senior Data Engineer with Python and Databricks
Core
Design, implement, and maintain scalable data pipelines for ingestion, processing, and transformation using cloud services.
Role type
Senior Data Engineer
Builds
Data pipelines and data solutions in Azure
Domain
Cloud data engineering, Azure ecosystem
Required skills
Python, SQL, Azure Data Factory, Azure Synapse, Databricks/Spark, Polars/Pandas, ETL/ELT, Relational databases, NoSQL databases, Data modeling (Kimball, Star Schema), Event-Driven Architecture
Preferred skills
Data monitoring and observability tools
Responsibilities
Collaborate with data scientists and analysts to define data needs, design and build data pipelines, implement validation and cleansing procedures, optimize pipeline scalability and cost, monitor and resolve pipeline issues.