Middle Data Engineer
Core
Building scalable and reliable data pipelines to ingest data from various sources, ensuring data quality, consistency, and integrity for a platform serving millions of users.
Role type
Middle Data Engineer
Builds
Data pipelines, ETL/ELT processes, Data Lake infrastructure, and data warehouse views
Domain
Technology / Data Engineering / Cloud Infrastructure
Deliverable
production ML models
Required skills
Python, PySpark, SQL, Apache Airflow, GCP services (PubSub, Dataproc, Composer, CloudStorage), JSON, Parquet, Postgres, MSSQL
Preferred skills
Tableau or other BI tools
Technologies
PySpark, GCP, Apache Airflow, BigQuery, Postgres, MSSQL, JSON, Parquet
Responsibilities
Create and maintain scalable data ingestion pipelines from internal systems, APIs, and external platforms; Work with structured and semi-structured data to ensure quality and integrity; Develop and maintain ETL/ELT processes; Process large datasets in Spark (Data Lake); Optimize PySpark pipelines; Work with JSON and Parquet data structures; Create data warehouse views in BigQuery; Work with Postgres and MSSQL databases.
