QA Data Engineering Lead
Core
Lead QA strategy and testing for complex data pipelines and large-scale datasets on Databricks platforms.
Role type
Senior IC QA Data Engineering Lead
Builds
Data pipelines, ETL processes, and data warehouse solutions on Databricks
Domain
Financial services, Data Engineering
Deliverable
production ML models
Required skills
Databricks, advanced SQL, Python, PySpark, Pandas, ETL processes, data warehouse concepts, data validation, Jira, Agile/Scrum
Preferred skills
Azure or AWS environments, Medallion Architecture (Bronze/Silver/Gold layers), test automation
Technologies
Databricks, SQL, Python, PySpark, Pandas, Jira, Azure, AWS
Responsibilities
Own overall data QA strategy and testing governance; Lead a team of Database QA Engineers; Develop comprehensive testing plans for complex data pipelines; Review business and technical requirements to identify testable conditions; Provide assessments of data quality and release readiness; Drive continuous improvement across QA processes; Design, develop, and execute ETL test cases; Validate data integrity across Medallion Architecture layers; Investigate pipeline failures and data inconsistencies.
Seniority
Senior, hands-on IC