Software Engineer III - Metadata Management & Data Discovery
Core
Design, build, and operationalize scalable metadata harvesting solutions across cloud, multi-cloud, and legacy on-premises platforms to enable automated data discovery, cataloging, and governance.
Role type
Senior Software Engineer (Metadata Management & Data Discovery)
Builds
Enterprise metadata harvesting pipelines, connectors, and integrations for Data Compass / Collibra
Domain
Financial Services / Data Governance & Metadata Management
Deliverable
production ML models | product features | infrastructure
Required skills
Python, SQL, API integration patterns, Databricks Unity Catalog, Delta Lake, Spark/PySpark, Azure ADLS Gen2, Azure Synapse, Azure Data Factory, Collibra EDGE, Snowflake, AWS, Google Cloud Platform, legacy data platforms (Oracle, SQL Server, DB2, Teradata, mainframe), data governance concepts (lineage, classification, stewardship), CI/CD, Git, monitoring, automated testing
Preferred skills
Graduate degree in computer science or engineering, experience with Purview, Alation, or Informatica, knowledge of Entra ID, Key Vault, private endpoints, VNets, NSGs, RBAC
Technologies
Collibra, Data Compass, Databricks, Unity Catalog, Delta Lake, Spark, PySpark, Azure ADLS, Azure Synapse, Azure Data Factory, Snowflake, AWS, Google Cloud, Oracle, SQL Server, DB2, Teradata
Responsibilities
Lead technical design and implementation of enterprise metadata harvesting solutions; Build automated harvesting pipelines and connectors; Engineer metadata extraction for Databricks Unity Catalog; Develop scalable harvesting patterns for Azure data platforms; Extend capabilities across Snowflake, AWS, and GCP; Design reconciliation controls; Define asset and attribute mapping rules; Partner with security and infrastructure teams; Support environment promotion and operational readiness; Build monitoring and support capabilities; Create technical documentation and architecture diagrams; Mentor engineers on metadata harvesting practices.
Seniority
Senior, hands-on IC