Senior Data Engineer - Databricks | PySpark | Spark Streaming
Core
Design, develop, and maintain data pipelines and backend services for real-time decisioning, reporting, and data collection.
Role type
Senior Data Engineer
Builds
Data pipelines, backend services, and data governance tools
Domain
Financial services / Cloud data engineering
Deliverable
production ML models
Required skills
PySpark, Spark Streaming, Databricks, Delta Lake, SQL, Python, Scala, Azure, CI/CD, Kafka, Event Hubs, Apache Flink, Kubernetes, Docker, Helm Charts, dbt, Data Vault
Preferred skills
Data modelling, system design, event-driven architectures
Responsibilities
Design and develop data pipelines for real-time decisioning; define data requirements and develop data models; manage and transform data across the platform; produce high-quality, well-tested, and secure code; develop software to improve data governance and security; develop processes to ensure Data Security and Data Quality
Seniority
Senior, hands-on IC