Big Data Lead
Core
Lead data integration, ELT, and data engineering solutions for large data volumes in distributed or cloud-based environments.
Role type
Senior IC data engineering lead
Builds
Resilient data pipelines, data lakes, data warehouses, and distributed compute platforms
Domain
Cloud data engineering and automation
Deliverable
production ML models
Required skills
SQL, Python/Java/Scala/Spark SQL, Apache Airflow, AWS Lambda, KAFKA, containerization, CI/CD, IaC, data modeling, metadata management, data governance
Preferred skills
Experience in regulated environments (financial services, healthcare), cloud-native foundations, AI coding assistants
Technologies
AWS, Apache Airflow, Azure Data Factory, AWS Step Functions, Kafka, Spark, Python, Java, Scala
Responsibilities
Develop and support data integration, ELT, and data engineering solutions; Process large data volumes in distributed or cloud-based environments; Design resilient data pipelines incorporating logging, monitoring, retry logic, idempotency, and failure recovery; Lead data modeling, metadata management, data governance, and master data management practices (via careerplan.io/jobs/45-004-66-07D-big-data-lead-at-sequoia-connect)
Seniority
Senior, hands-on IC
