Staff, Backend Engineer - Catalog
Core
Lead development of DataHub's Platform framework to connect diverse data systems and power metadata collection for enterprise AI infrastructure.
Role type
Staff, Backend Engineer (Platform Framework)
Builds
Scalable ingestion systems, APIs for connector ecosystem, event-driven architectures, schema mapping, and versioning systems for AI assets.
Domain
AI & Data Context Platform / Enterprise SaaS
Deliverable
production ML models | product features | infrastructure
Required skills
Distributed systems, Python, API design, High-scale data processing, Systems architecture, Online traffic handling, Monitoring/alerting, Data architecture, LLM development
Preferred skills
Java/Scala/Kotlin/C#/Go, TypeScript/Node.js, Metadata/ETL frameworks (Airflow, Airbyte, dbt), Open-source contributions, LLM integration, Fine-tuning LLMs
Technologies
AWS, Kubernetes, Docker, CI/CD pipelines, Microservice Architecture
Responsibilities
Design and build scalable, fault-tolerant ingestion systems for enterprise-scale metadata; Develop clean, intuitive APIs for the connector ecosystem; Implement event-driven architectures for real-time metadata processing; Create schema mapping between diverse systems and DataHub's unified model; Build versioning systems for AI assets (training data, model weights, embeddings); Set up monitoring and alerting for services.
Seniority
Staff, hands-on IC
