Senior Databricks Engineer, Apache Spark and AWS - Vice President
Core
Senior Databricks Engineer leading the modernization of a legacy Hadoop platform to Databricks on AWS, focusing on performance optimization and pipeline simplification.
Role type
Senior hands-on IC Databricks/Spark Engineer
Builds
Cloud-native data processing pipelines on Databricks and AWS
Domain
Big Data / Cloud Data Engineering
Deliverable
production ML models
Required skills
Apache Spark (JavaSpark/PySpark), Databricks, Delta Lake, AWS services, SQL, distributed systems architecture, performance tuning, pipeline orchestration
Preferred skills
Legacy Hadoop modernization, large-scale batch optimization, data modeling
Technologies
Databricks, Apache Spark, AWS, Delta Lake, Java, Python
Responsibilities
Refactor and modernize existing Spark pipelines to Databricks native architectures; Build and optimize solutions using Databricks features including Delta Lake and Auto scaling; Analyze and improve Spark job performance; Develop unit, integration, and data validation tests; Contribute to low and mid level architecture and design; Write clean, modular, and testable code
Seniority
Senior, hands-on IC