Senior Databricks Data Engineer - Assistant Manager
Core
Design, develop, and deploy scalable ETL pipelines and high-throughput data ingestion (streaming, batch) on cloud infrastructure to process high-volume data in near real-time for State Street Global Technology Services.
Role type
Senior Databricks Data Engineer (Assistant Manager)
Builds
End-to-end data lakehouse solutions, configurable data pipelines, and scalable ETL workflows.
Domain
Financial Services / Cloud Data Engineering
Deliverable
production ML models | product features | dashboards & analysis
Required skills
Databricks SQL, Scala/PySpark, Azure services, Microservices/API/Event-Driven Architecture, Data Lakehouse architecture, Cloud-based development (Azure/AWS), ETL pipeline design, Query optimization, Resource management, Root cause analysis, Software development lifecycle management.
Preferred skills
Financial industry experience, Agile development methodology, Cross-functional collaboration.
Technologies
Databricks, Azure, AWS, Scala, PySpark, SQL, Microservices
Responsibilities
Design and develop custom high-throughput data pipelines; Build and test end-to-end data ingestion pipelines (streaming, events, batch); Optimize and tune Databricks job performance; Address data and environment issues through impact analysis and maintenance; Manage all aspects of the software development lifecycle including design, coding, testing, deployment, and documentation.
Seniority
Senior, hands-on IC with management responsibilities
