Senior Data Engineer
Core
Design, develop, and operate scalable real-time data pipelines and a next-generation data platform using streaming technologies and lakehouse formats.
Role type
Senior Data Engineer (Real-time Streaming & Lakehouse)
Builds
Real-time data products, scalable data pipelines, and observability tools for engineering teams.
Domain
Data Engineering, Real-time Streaming, Cloud Infrastructure
Required skills
Apache Spark Structured Streaming, Apache Iceberg, AWS EMR, Change Data Capture (CDC), Python, SQL, Apache Airflow, Kubernetes
Preferred skills
Snowflake, Grafana, Loki, Scala, Java, Delta Lake, Apache Hudi
Technologies
Apache Spark, Apache Iceberg, AWS EMR, Kafka, Estuary Flow, Snowflake, Grafana, Loki, Apache Airflow, Kubernetes
Responsibilities
Design and maintain robust Apache Spark Structured Streaming applications; Optimize AWS EMR clusters and manage resource allocation; Maintain and optimize Apache Iceberg tables at scale; Onboard new business domains via CDC sources and Kafka topics; Investigate and resolve complex production incidents; Develop monitoring capabilities and diagnostic tools; Improve CI/CD pipelines and developer workflows.
Seniority
Senior, hands-on IC
