Big Data / PySpark Engineering Lead - Vice President
Core
Lead the strategic migration of legacy data systems to a modern Data Lakehouse, designing scalable batch and real-time data processing pipelines.
Role type
Senior IC Big Data Engineering Lead (PySpark/Spark)
Builds
Scalable distributed data processing frameworks, modern Data Lakehouse architectures, and automated ETL/ELT pipelines.
Domain
Big Data, Data Engineering, Cloud Data Platforms
Required skills
PySpark, Python, SQL, Hadoop, Kafka, NoSQL databases, Unix shell scripting, CI/CD pipelines, Data Lineage tools, Schema evolution
Preferred skills
Reverse engineering legacy code, Change management, Automation mindset, AI tools familiarity
Technologies
Spark, Flink, Kafka, Starburst/Trino, Parquet, Avro, Iceberg, MongoDB, HBase, Git, Bitbucket
Responsibilities
Design and implement scalable, fault-tolerant batch and real-time data processing pipelines; Lead the strategic migration of data and logic from legacy platforms to a modern Data Lakehouse; Re-engineer existing stored procedures and complex legacy ETL jobs into distributed processing frameworks; Build and maintain CI/CD pipelines for automated testing and deployment; Provide technical mentorship and conduct code reviews for junior and mid-level engineers.
Seniority
Senior, hands-on IC with leadership responsibilities