Job
Core
Senior Data Engineer specializing in Data Quality and QA, building trusted, reliable, and production-ready data products within a Databricks Lakehouse environment.
Role type
Senior IC data engineer (data quality & testing)
Builds
Production-grade PySpark and SQL pipelines, automated testing frameworks, data observability solutions, and reusable data quality libraries.
Domain
Financial services / Data Engineering
Deliverable
production ML models
Required skills
Python, PySpark, SQL, Databricks (Unity Catalog, Delta Lake, Asset Bundles, Workflows), automated testing frameworks, data quality tools (Great Expectations, Soda Core, dbt), data observability, CI/CD (GitHub Actions), data governance (GDPR, PII masking), performance testing.
Preferred skills
Watermark-based incremental ingestion, CDC pipelines, dbt, AWS (S3, IAM), API testing (Postman, REST Assured), data catalog platforms (OpenMetadata, Collibra), Power BI, financial services domain experience.
Responsibilities
Design and execute test strategies for Databricks pipelines and data products; build automated testing frameworks for data quality validation; validate data accuracy, completeness, and lineage; implement automated data observability and anomaly detection; build and maintain production-grade PySpark/SQL pipelines; develop reusable data quality frameworks; evaluate and recommend data quality tooling; mentor team members on testing strategies and engineering standards.
Seniority
Senior, hands-on IC