Senior Data Engineer
Core
Senior Data Engineer specializing in optimizing production Apache Spark pipelines and distributed processing architectures.
Role type
Senior IC Data Engineer (Scala & Apache Spark)
Builds
Production data processing pipelines and distributed computing systems
Domain
Data Engineering / Big Data / Cloud Data Warehousing
Deliverable
production ML models | product features | infrastructure
Required skills
Scala, Apache Spark, Catalyst optimizer, DAG analysis, performance tuning, Snowflake, distributed processing, memory management, data skew correction
Preferred skills
Advanced partitioning strategies, broadcast joins, hierarchical data traversal
Technologies
Scala, Apache Spark, Snowflake
Responsibilities
Optimize production Spark pipelines, refactor code for performance, design efficient large-scale distributed processing, reduce costs and latency
Seniority
Senior, hands-on IC
