CareerPlan

Senior Engineer – Data Engineering (Databricks) - Contractor

IndiaRemoteContract2026-11-16
Remote · from IndiaFreelance / independent contractor

Core

Design and deliver scalable data platforms and advanced data pipelines using Databricks and cloud-based lakehouse architectures.

Role type

Senior Data Engineer (Databricks)

Builds

Modern, scalable data platforms and distributed data processing solutions

Domain

Data Engineering / Cloud Data Platforms

Deliverable

production ML models

Required skills

Databricks, Delta Lake, Delta Live Tables, Unity Catalog, Apache Spark, Python, PySpark, SQL, YAML configuration, Data Orchestration (Airflow, dbt), CI/CD, Infrastructure as Code (Terraform)

Preferred skills

MLflow, prompt engineering, NoSQL databases, container management, Databricks certifications

Technologies

Databricks, Delta Lake, Delta Live Tables, Unity Catalog, Apache Spark, Apache Airflow, dbt, Amazon EMR, Amazon Redshift, Snowflake, Google BigQuery, Terraform, MLflow

Responsibilities

Design and implement scalable data engineering solutions using Databricks; Build and maintain advanced data pipelines leveraging Delta Lake and Delta Live Tables; Design and optimize YAML configuration-driven data pipelines for batch, micro-batch, and real-time streaming; Implement efficient data ingestion and transformation workflows using Databricks Auto Loader; Develop complex job scheduling and workflow orchestration solutions; Collaborate with solution architects to design enterprise data platform architectures; Develop and implement data quality, pipeline auditing, monitoring, and observability frameworks; Optimize data processing performance, pipeline reliability, and resource utilization; Implement DevOps and CI/CD practices using Databricks Asset Bundles; Contribute to infrastructure automation using Infrastructure as Code (IaC) tools; Explore and integrate emerging technologies in data engineering and AI.

Seniority

Senior, hands-on IC