Sr. Platform Engineer, Kubernetes
Core
Architecting and managing Kubernetes clusters and cloud services to enable efficient, scalable execution of large-scale Apache Spark data processing workloads.
Role type
Senior Platform Engineer (Data Infrastructure)
Builds
Robust, cost-efficient environments for data engineers and scientists running Spark jobs on Kubernetes/AWS
Domain
Cloud Infrastructure, Big Data Processing, Data Platform Engineering
Deliverable
production ML models | infrastructure
Required skills
Apache Spark architecture, Kubernetes, Terraform, Ansible, Python, Scala, SQL, Linux Bash scripting, Prometheus, Grafana, Delta Lake, Apache Iceberg, Unity Catalog, Databricks, Snowflake, Networking protocols, IAM/VPC/EC2 setup
Preferred skills
Experience with Alluxio caching layers, open-source community contributions, data lake governance
Technologies
Kubernetes, AWS (EKS), Docker, Apache Spark, Apache Flyte, Terraform, Ansible, Prometheus, Grafana, Apache Iceberg, Unity Catalog, Databricks, Snowflake, Delta Lake, Alluxio, Linux, Bash, Python, Scala, Java
Responsibilities
Architect and manage Kubernetes clusters and cloud services for Spark workloads; Package Spark workloads and integrate with orchestration systems; Deploy infrastructure via IaC tools; Troubleshoot job failures and optimize Spark configurations for cost; Build automation tools; Implement monitoring, logging, and alerting systems; Develop and optimize data catalog platforms; Automate workflows and incident resolution; Collaborate with data teams to define requirements and ensure delivery; Create and maintain infrastructure documentation.
Seniority
Senior, hands-on IC