IN_Manager_Databricks Engineer _D&A _Advisory _Mumbai
Core
Lead the development and optimization of scalable data processing systems and ETL/ELT pipelines using Databricks to drive strategic insights for clients.
Role type
Manager, Data Engineering & Analytics
Builds
Production data pipelines, optimized Spark jobs, and data warehousing solutions for enterprise clients
Domain
Financial Services, Data Engineering, Cloud Data Platforms
Deliverable
production ML models | product features | dashboards & analysis
Required skills
PySpark, Databricks, SQL, Unity Catalog, Delta Tables, DLT, ETL/ELT design, Cloud storage integration (Azure/AWS), Data governance, Performance tuning, Git, Airflow/Azure Data Factory
Preferred skills
Informatica PowerCenter, Advanced data architecture patterns
Technologies
Databricks, Apache Spark, PySpark, Unity Catalog, Delta Lake, Azure Data Lake, AWS S3, Airflow, Azure Data Factory, Git
Responsibilities
Design and maintain scalable ETL/ELT pipelines using PySpark on Databricks; Optimize Spark jobs for performance and cost-efficiency; Ingest and transform data from structured and unstructured cloud sources; Collaborate with stakeholders to define data requirements and drive user adoption; Implement data quality, integrity, and governance best practices; Lead code reviews, performance tuning, and documentation of data solutions.
Seniority
Manager, hands-on IC with team leadership
