Principal Consultant- Databricks Developer
Core
Design and build scalable ETL pipelines and real-time data processing frameworks using Databricks, Unity Catalog, Apache Spark, and Kafka to meet functional and non-functional data requirements.
Role type
Principal Consultant - Databricks Developer (Data Engineering)
Builds
Scalable ETL pipelines, real-time data streams, and data governance frameworks for global enterprises.
Domain
Data Engineering / Cloud Computing
Deliverable
production ML models | product features | dashboards & analysis | infrastructure
Required skills
Databricks Platform, Apache Spark, Apache Kafka, Unity Catalog, Python, Data Engineering, Cloud Computing, Data Governance, Data Orchestration, Data Lineage
Preferred skills
Agile Methodology, Client Relations, Commercial Sales, Risk Management, Storytelling
Technologies
Databricks, Unity Catalog, Apache Spark, Apache Kafka, Python, Snowflake
Responsibilities
Develop and maintain scalable ETL pipelines using Databricks with a focus on Unity Catalog; Implement data processing frameworks using Apache Spark for large-scale data transformation; Integrate real-time data streams using Apache Kafka and Databricks; Design and enforce data governance policies, access controls, and security protocols within Unity Catalog; Monitor data pipeline performance, troubleshoot issues, and implement optimizations; Write efficient Python scripts for data extraction, transformation, and loading.
Seniority
Principal, hands-on IC with strategic client delivery
