Senior Data Engineer / Databricks Expert
Core
Building a global analytics environment for a pharmaceutical organization by ingesting, processing, and organizing operational data from laboratories, manufacturing systems, and supply chains into reliable datasets.
Role type
Senior Data Engineer (Databricks Expert)
Builds
Scalable data pipelines, transformation layers, and a centralized lakehouse architecture for downstream analytics.
Domain
Pharmaceutical industry, data engineering, lakehouse architecture
Required skills
Databricks, Python, PySpark, SQL, Apache Spark, Delta Lake, Lakehouse architecture, Unity Catalog, CI/CD, Git
Preferred skills
DBT, Structured Streaming, Terraform, Machine Learning workflows, Pharmaceutical data
Technologies
Databricks, Apache Spark, Python, SQL, Delta Lake, Unity Catalog, DBT, Azure/AWS/GCP
Responsibilities
Design and maintain data pipelines in Databricks using Python, PySpark, and SQL; Build and maintain Delta Lake tables following bronze, silver, and gold architecture; Integrate data from multiple source systems into consistent datasets; Implement incremental processing, change data capture, and schema evolution; Develop and orchestrate production workflows with scheduling and monitoring; Optimize Spark workloads and compute usage; Implement automated data quality checks and validation; Support data access management, governance, and lineage using Unity Catalog; Maintain reusable code, technical documentation, and Git-based workflows.
Seniority
Senior, hands-on IC