Manager 1, Software Development & Engineering
Core
Design, develop, and optimize large-scale streaming data pipelines and systems using Apache Spark to enable advanced analytics and machine learning.
Role type
Senior Data Engineering Manager
Builds
High-performance data pipelines, streaming jobs, and automated operational tooling for complex data processing.
Domain
Big Data Engineering / Cloud Infrastructure
Deliverable
production ML models
Required skills
Apache Spark, Scala, Java, Cloud platforms (AWS EMR, Azure Databricks, GCP Dataproc), CI/CD, Infrastructure as Code (Terraform, CloudFormation), Data pipeline architecture, Performance optimization
Preferred skills
Real-time streaming (Kafka, Spark Streaming), Docker, Kubernetes, Databricks, Performance tuning
Technologies
Apache Spark, Kafka, Docker, Kubernetes, Terraform, CloudFormation, AWS EMR, Azure Databricks, GCP Dataproc
Responsibilities
Design and implement scalable streaming jobs in Cloud and on-prem environments; Optimize Spark jobs for performance and cost efficiency; Develop and maintain automated tooling for operational monitoring and maintenance of complex pipelines; Implement data validation, monitoring, and lineage tracking; Research and adopt new technologies to improve Spark-based data engineering processes.
Seniority
Senior, hands-on IC with management responsibilities