Data Engineer - Spark Developer
Core
Design, develop, test, deploy, maintain, and improve big data pipelines and architectures using Apache Spark on Azure Databricks.
Role type
Data Engineer (Spark Developer)
Builds
Data pipelines and big data architectures on Azure
Domain
Cloud Data Engineering (Azure)
Deliverable
production ML models | product features | infrastructure
Required skills
Apache Spark, Python or Scala, Azure Data Factory, Azure Service Bus, Spark query tuning, cloud platforms (Azure/AWS/GCP), big data architecture design
Preferred skills
DAG operations analysis, cost estimation for big data processing, architecture documentation
Technologies
Apache Spark, Azure Databricks, Azure Data Factory, Azure Service Bus
Responsibilities
Design and develop big data architectures; Code using Apache Spark on Azure Databricks; Design, develop, test, deploy, maintain, and improve data pipelines
Seniority
Mid-level, hands-on IC
