Data QA engineer
Core
Build systematic data testing processes and ensure data quality across all processing stages from sources to final analytical warehouses.
Role type
Middle Data QA Engineer
Builds
Corporate data platform on Databricks, integrating data from SAP, 1C, and other systems for master data preparation and management reporting.
Domain
Security systems industry / Data Warehouse & Analytics
Required skills
SQL, Python, ETL/ELT processes, Data Warehouse architecture, Data Quality principles, Data Validation, Data Reconciliation, Data Governance, Regression testing, Automated testing
Preferred skills
Databricks, PySpark, Spark, Python-based testing frameworks (pytest, Great Expectations, dbt tests, Soda), SAP integration testing, CI/CD integration, BI solutions (Power BI), Building testing processes from scratch
Technologies
Databricks, Python, SQL, PySpark, SAP, 1C, Power BI
Responsibilities
Build and develop data testing processes in the Data Warehouse & Analytics team; Test ETL/ELT processes and verify data transfer/transformation from SAP, 1C, and other sources to Databricks; Verify data compliance with business requirements from source systems to final analytical warehouses; Conduct Data Validation and Source-to-Target Reconciliation; Check business logic of transformations, processing rules, and data enrichment; Detect duplicates, missing records, incorrect values, and other data anomalies; Develop and maintain test cases, test scenarios, and data quality rules; Build and develop automated data testing, including regression checks; Verify data correctness after changes in data pipelines, data structures, and business logic; Participate in testing master data preparation processes for SAP implementation; Analyze causes of data discrepancies and collaborate with Data Engineers, Business Analysts, and related teams to resolve them; Participate in improving Data Quality approaches and increasing data processing reliability.
Seniority
Middle, hands-on IC