Data engineer Databricks
Core
Ensure robustness of the data contract registry (Aurora and Python FastAPI) and manage data ecosystems by delivering metadata to the data lake.
Role type
Data Engineer
Builds
Data pipelines, metadata delivery to data lake, automated governance features
Domain
Cloud data engineering (AWS S3, AWS Glue, Unity Catalog)
Deliverable
production ML models | infrastructure
Required skills
Python, FastAPI, Aurora, AWS S3, AWS Glue, Unity Catalog, data pipeline design, metadata management
Preferred skills
Data governance, team mentoring, proactive system monitoring
Technologies
Aurora, Python, FastAPI, AWS S3, AWS Glue, Unity Catalog
Responsibilities
Monitor systems continuously and take proactive management actions; Manage data ecosystems and deliver metadata to the data lake; Develop automated governance features (access and metadata management) in collaboration with Staff Engineers; Define best practices and advise on data pipeline design; Support team tools.