Senior Databricks Engineer
Core
Design, develop, test, and deploy scalable batch and real-time data engineering solutions on a greenfield Databricks platform, including reverse engineering existing AWS-based data processing.
Role type
Senior Databricks Engineer (Data Engineering)
Builds
Scalable data pipelines, Bronze/Silver/Gold data layers, and Lakeflow Declarative Pipelines for customer data models.
Domain
Cloud Data Engineering / Big Data
Required skills
Databricks, Apache Spark, PySpark, Delta Lake, Unity Catalog, AWS Glue, Amazon S3, Amazon Redshift, Structured Streaming, CI/CD, Data Modeling, Performance Tuning, Reverse Engineering
Preferred skills
AI-assisted development tools, Automation, Mentoring
Technologies
Databricks, Apache Spark, PySpark, Delta Lake, Unity Catalog, AWS Glue, Amazon S3, Amazon Redshift, Lakeflow Declarative Pipelines
Responsibilities
Design and implement scalable ingestion solutions using Databricks Auto Loader; Develop and operate Lakeflow Declarative Pipelines supporting batch and streaming workloads; Implement robust transformation logic using Apache Spark and Delta Lake; Design and implement Unity Catalog governance structures; Perform performance tuning and optimization of Spark workloads; Provide hands-on knowledge transfer and mentoring to client engineers.
Seniority
Senior, hands-on IC