Senior Data Engineer
Core
Design, develop, and maintain scalable data pipelines and ETL/ELT processes for large-scale, diverse datasets.
Role type
Senior Data Engineer
Builds
Data pipelines, ETL/ELT processes, data models, and architectures
Domain
Cloud data engineering (AWS)
Required skills
SQL, PySpark, Python, Apache Airflow, AWS Glue, Kafka, Redshift, Git, Terraform, Ansible, CI/CD, distributed systems design, API development
Preferred skills
DBT, Data Mesh principles, real-time (Lambda) and batch (Kappa) processing architectures, serverless architectures, building scalable data platforms
Technologies
AWS (Lambda, S3, CloudWatch, SNS/SQS, Kinesis), Apache Airflow, AWS Glue, Kafka, Redshift, Terraform, Ansible, FastAPI, GraphQL, OpenAPI
Responsibilities
Lead design and development of data pipelines; Optimize data ingestion and transformation; Drive best practices in version control and infrastructure as code; Collaborate with cross-functional teams to design data models; Monitor and debug ETL pipelines; Mentor mid-level and junior engineers
Seniority
Senior, hands-on IC with mentorship responsibilities