Data Architect - Data Engineering 4D
Core
Build and operate a governed, metadata-driven data platform on Azure Databricks/Unity Catalog for ingestion, archival, and controlled distribution to downstream systems.
Role type
Senior DataOps Engineer (Data Engineering)
Builds
A central data repository with automated ingestion pipelines, retention policies, and routing rules for multiple downstream systems.
Domain
Cloud Data Engineering / Big Data
Required skills
Azure Databricks, Unity Catalog, PySpark, SQL, Structured Streaming, Delta Lake, Metadata-driven code generation, CI/CD, Data Governance
Preferred skills
Delta Live Tables, Snowflake, Iceberg, Terraform
Technologies
Azure Databricks, Unity Catalog, Azure Data Factory, Azure DevOps, GitHub Actions, Delta Lake, PySpark, SQL
Responsibilities
Build Auto Loader-based ingestion pipelines; Design metadata-driven pipeline generation; Implement partition-based retention and archive extraction; Maintain dataset routing-rules engine; Own Unity Catalog governance; Manage Databricks Asset Bundles (DAB) for IaC; Monitor pipeline health and data quality; Collaborate on data contracts and access requirements.
Seniority
Senior, hands-on IC