Data Engineer
Core
Design, build, test, and maintain scalable data pipelines and models on a Databricks lakehouse on Azure, integrating agentic AI into data workflows.
Role type
Senior IC Data Engineer (AI-first data platform)
Builds
Scalable batch and real-time data pipelines, data models, and agentic AI workflows on Azure Databricks
Domain
Cloud data engineering, AI/ML infrastructure, Big Data
Deliverable
production ML models
Required skills
PySpark, Python, Databricks (Unity Catalog, Workflows, Asset Bundles), Kafka, SQL, CI/CD, automated testing, agentic AI tools
Preferred skills
Azure Data Factory, event streaming architecture, LLM-powered agent design
Technologies
Databricks, Azure, Unity Catalog, Kafka, PySpark, Python, GitHub Actions, Azure DevOps
Responsibilities
Design and develop data pipelines in Python (PySpark); Ingest and integrate diverse data sources into the lakehouse; Introduce event streaming and real-time ingestion; Maintain data models and codebase; Write automated tests for pipeline logic and data quality; Monitor and operate the cloud data platform; Collaborate with engineers, analysts, and business stakeholders
Seniority
Senior, hands-on IC