Senior Engineer – Data Engineering (Databricks) - Contractor
Core
Design and deliver scalable data platforms and advanced data pipelines using Databricks and cloud-based lakehouse architectures.
Role type
Senior Data Engineer (Databricks)
Builds
Modern, scalable data platforms and distributed data processing solutions
Domain
Data Engineering / Cloud Data Platforms
Deliverable
production ML models
Required skills
Databricks, Delta Lake, Delta Live Tables, Unity Catalog, Apache Spark, Python, PySpark, SQL, YAML configuration, Data Orchestration (Airflow, dbt), CI/CD, Infrastructure as Code (Terraform)
Preferred skills
MLflow, prompt engineering, NoSQL databases, container management, Databricks certifications
Technologies
Databricks, Delta Lake, Delta Live Tables, Unity Catalog, Apache Spark, Apache Airflow, dbt, Amazon EMR, Amazon Redshift, Snowflake, Google BigQuery, Terraform, MLflow
Responsibilities
Design and implement scalable data engineering solutions using Databricks; Build and maintain advanced data pipelines leveraging Delta Lake and Delta Live Tables; Design and optimize YAML configuration-driven data pipelines for batch, micro-batch, and real-time streaming; Implement efficient data ingestion and transformation workflows using Databricks Auto Loader; Develop complex job scheduling and workflow orchestration solutions; Collaborate with solution architects to design enterprise data platform architectures; Develop and implement data quality, pipeline auditing, monitoring, and observability frameworks; Optimize data processing performance, pipeline reliability, and resource utilization; Implement DevOps and CI/CD practices using Databricks Asset Bundles; Contribute to infrastructure automation using Infrastructure as Code (IaC) tools; Explore and integrate emerging technologies in data engineering and AI.
Seniority
Senior, hands-on IC