Data Engineering Operations Lead
Core
Lead Data Engineering operations, ensuring reliability and efficiency of data ingestion pipelines, monitoring, and incident resolution for a global credit reference agency.
Role type
Senior IC data engineering operations lead
Builds
Scalable data ingestion pipelines, data lakes, warehouses, and real-time streaming systems
Domain
Finance / Credit Risk / Big Data
Deliverable
production ML models | infrastructure
Required skills
Apache Spark (PySpark, SparkSQL), Python, SQL, Apache Airflow, GCP Data Stack (BigQuery, Dataflow, Pub/Sub, Cloud Composer, Dataproc), CI/CD, Git, Data Modeling, ETL/ELT, Incident Management, Team Leadership
Preferred skills
Kafka, R, Sparklyr, MLOps, MLflow, Harness
Technologies
Apache Spark, Python, SQL, Apache Airflow, Google Cloud Platform (GCP), BigQuery, Dataflow, Pub/Sub, Cloud Composer, Dataproc, Kafka, Delta Lake
Responsibilities
Own and manage daily Data Engineering operations including pipeline monitoring and incident resolution; Oversee end-to-end pipeline health and resolve failures; Organize and prioritize team workload; Maintain and improve ingestion pipelines into data lakes and warehouses; Act as escalation point for pipeline failures and drive root cause analysis; Define and enforce operational standards like runbooks and alerting; Coordinate with upstream providers and downstream consumers; Mentor junior engineers on operational best practices; Drive continuous improvement in pipeline reliability and observability.
Seniority
Senior, hands-on IC with leadership responsibilities
