Senior Data Engineer @ DS STREAM
Core
Building and maintaining high-availability data pipelines and hubs using Azure Databricks, Spark Structured Streaming, and Medallion Architecture to ingest, transform, and integrate large-scale data sources.
Role type
Senior Data Engineer (Large Scale Data Pipelines)
Builds
Scalable data ingestion and transformation pipelines within an Azure/GCP ecosystem
Domain
Big Data, Cloud Data Engineering (Azure, GCP, Databricks)
Deliverable
production ML models | product features | dashboards & analysis | infrastructure
Required skills
Python, SQL, PySpark, Spark Structured Streaming, Azure Databricks, Unity Catalog, Airflow, GCP (BigQuery, Pub/Sub), Docker, CI/CD, API integrations
Preferred skills
FastAPI, Kubernetes, AWS, mentoring engineers, leading teams
Technologies
Azure, GCP, Databricks, Unity Catalog, Spark, Airflow, BigQuery, VertexAI, FastAPI, Docker, Kubernetes
Responsibilities
Develop Databricks pipelines in PySpark/SQL, handle Spark Structured Streaming pipelines, design and manage objects in Unity Catalog, ingest data from APIs, build pipelines using version control tools
Seniority
Senior, hands-on IC
