Databricks Data Engineer - Manager
Core
Build and optimize large-scale data engineering solutions on Azure and AWS using Databricks to deliver data and insights for financial services clients.
Role type
Manager-level Data Engineer (Databricks)
Builds
Production data pipelines, Delta Lake implementations, and Unity Catalog RBAC configurations for State Street's Global Technology Services.
Domain
Financial services / Cloud Data Engineering
Deliverable
production ML models | product features
Required skills
Databricks Platform (Notebooks, Clusters, Jobs, Delta Lake), Unity Catalog, RBAC configuration, Azure Cloud Services, CI/CD (Azure DevOps, Harness), Scala, Python, PySpark, SQL, Apache Spark optimization, Debugging and code optimization.
Preferred skills
AWS, Docker, Kubernetes, Helm framework.
Technologies
Databricks, Delta Lake, Unity Catalog, Azure Data Factory, Azure Blob Storage, Azure DevOps, Harness, PySpark, Scala, Python, SQL, Apache Spark, AWS, Docker, Kubernetes, Helm.
Responsibilities
Design and implement data engineering solutions on Databricks and Azure; optimize PySpark/SQL code for performance; configure Unity Catalog and RBAC; manage CI/CD pipelines; lead technical execution for data projects.
Seniority
Manager, hands-on IC with leadership responsibilities
