Data Engineer - QA
Core
Design, develop, and ensure the quality, reliability, and accuracy of data pipelines and the Data Platform component of the ThunderSoft Cloud Platform.
Role type
Data Engineer with a focus on Data Quality and QA
Builds
Robust data quality frameworks, automated tests, and monitoring solutions for data pipelines
Domain
Cloud data engineering (Databricks / AWS)
Deliverable
production ML models | product features
Required skills
Python, data validation frameworks, ETL/ELT pipelines, Databricks, Spark, SQL, CI/CD integration, data observability
Preferred skills
AWS cloud environments, agile methodology
Technologies
Databricks, Spark, AWS, Python
Responsibilities
Define and maintain data quality rules and validation frameworks; Design and integrate automated tests into ETL/ELT pipelines; Implement data observability and monitoring solutions; Analyze and troubleshoot data quality issues in production; Validate data transformations against business logic; Collaborate with teams to define data contracts and acceptance criteria; Ensure production reliability and respond to incidents; Promote QA best practices including test automation and shift-left quality.
Seniority
Mid-level (4+ years experience)
