Data Pipeline Engineer
Core
Design, develop, and maintain batch and real-time data pipelines for ingestion, transformation, and delivery of enterprise data.
Role type
Data Pipeline Engineer
Builds
Scalable ETL/ELT solutions, data warehouses, lakehouses, and storage solutions
Domain
Cloud data engineering and analytics
Deliverable
production ML models | product features
Required skills
SQL, Python or Scala, ETL/ELT pipeline development, data modeling, orchestration tools, big-data and streaming tools, cloud data warehouses, data quality frameworks, CI/CD, infrastructure-as-code
Preferred skills
Near Real Time event streams, Azure SQL Managed Instances, Power BI Gateways, DataFlows
Technologies
Airflow, Azure Data Factory, Spark, Kafka, Synapse, OneLake, Lakehouse, Azure Event Hubs, Azure Synapse
Responsibilities
Troubleshoot and resolve pipeline failures, performance bottlenecks, and data quality issues; Design and optimize data models and storage solutions; Support migration of legacy data environments to modern cloud-based architectures; Apply data security approaches and ensure compliance with governance requirements
Seniority
Mid-level, hands-on IC
