Principal Machine Learning Engineer, Accelerated Apache Spark
Core
Design and implement machine learning solutions for performance prediction and optimization of GPU-accelerated enterprise Apache Spark workloads.
Role type
Principal Machine Learning Engineer (System Optimization)
Builds
GPU-accelerated Apache Spark workloads and AI-based optimization agents
Domain
Data Processing / High-Performance Computing / Machine Learning Systems
Deliverable
production ML models
Required skills
Large-scale data processing (Apache Spark), Python, PyTorch, TensorFlow, XGBoost, Reinforcement Learning, Adaptive/Online ML, Feature Engineering, Technical Leadership, CUDA/NVIDIA GPU architecture
Preferred skills
Scala, Java, C++, LLM/GenAI, Deep understanding of Apache Spark internals
Technologies
Apache Spark, NVIDIA GPUs, CUDA, Python, PyTorch, TensorFlow, XGBoost, numpy, pandas, scikit-learn, scipy
Responsibilities
Design ML solutions for performance prediction and optimization of GPU-accelerated Spark workloads; Develop advanced algorithms and adaptive systems to improve Spark performance on GPUs; Develop AI-based agents for system issue resolution and application optimization; Provide technical mentorship and leadership in data science and machine learning
Seniority
Principal, hands-on IC with strategic leadership
