Senior Platform Engineer
Core
Lead development of DataHub's ingestion framework to connect diverse data systems and power metadata collection for enterprise-scale AI and data assets.
Role type
Senior Python Platform Engineer (Ingestion Systems)
Builds
Scalable, fault-tolerant ingestion systems, clean APIs for the connector ecosystem, event-driven architectures for real-time metadata processing, and versioning systems for AI assets.
Domain
Enterprise SaaS, AI & Data Infrastructure, Metadata Management
Deliverable
production ML models | infrastructure
Required skills
Python (advanced), distributed systems design, API design, high-scale data processing, integration frameworks, systems architecture
Preferred skills
Metadata/ETL frameworks (Airflow, Airbyte, dbt), open-source contributions, early-stage startup experience
Technologies
Python, distributed systems, event-driven architectures
Responsibilities
Design and build scalable ingestion systems for enterprise metadata; Develop intuitive APIs for the connector ecosystem; Implement event-driven architectures for real-time processing; Create schema mapping between diverse systems and DataHub's unified model; Build versioning systems for AI assets (training data, model weights, embeddings).
Seniority
Senior, hands-on IC