Data Engineer with using Apache Spark, Python, Scala.
Core
Design, develop, and maintain robust data infrastructure and pipelines for business requirements.
Role type
Data Engineer
Builds
Data processing pipelines, ingestion/extraction processes, streaming and batch workflows
Domain
Data Engineering / Big Data
Deliverable
production ML models | product features | infrastructure
Required skills
Apache Spark, Python, Scala, Kafka, Docker, Kubernetes, pytest, JUnit, Cucumber, Lettuce, SQL, ETL, data warehousing, performance tuning
Responsibilities
Design and optimize data processing pipelines; Develop and maintain data ingestion and extraction processes; Implement performance tuning for Spark jobs and SQL queries; Collaborate with DevOps to deploy infrastructure on NetApp S3 and Kubernetes; Containerize applications and orchestrate deployments; Develop and maintain unit tests using BDD tools; Implement data governance, security, and compliance best practices.
