D&A - Senior Data Engineer
Core
Design, develop, and maintain AWS and Databricks-based data platforms, bridging Data Engineering, DBA, and Analytics Engineering.
Role type
Senior Data Engineer (Cloud & Lakehouse)
Builds
Scalable ETL/ELT pipelines, Medallion architecture data layers, and optimized data models for BI tools.
Domain
Cloud Data Engineering, Lakehouse Architecture
Deliverable
production ML models
Required skills
PySpark, SQL (Window Functions, CTEs), AWS ecosystem, Snowflake, Databricks, Delta Lake, Data Modeling, Performance Tuning, Data Governance
Preferred skills
Delta Live Tables, dbt, MLOps, Apache Airflow
Technologies
AWS (S3, IAM, EC2), Databricks, Snowflake, Delta Lake, Unity Catalog, Power BI, Tableau
Responsibilities
Develop and maintain complex ETL/ELT pipelines using Python/PySpark on Databricks and Snowflake; Implement and manage data layers following the Medallion architecture using Delta Lake; Set up and manage Databricks SQL Warehouses and perform query optimization; Leverage AWS cloud services to manage data storage, compute resources, and secure data movement; Act as a custodian for the data platform managing indexing, clustering, vacuuming, and performance tuning; Design data models (Star Schema) in the Gold layer for BI tools; Manage metadata, permissions, and lineage using Unity Catalog; Implement automated Data Quality tests in CI/CD pipelines.
Seniority
Mid-Level, hands-on IC
