Lead Data Engineer
Core
Architecting and building a unified data platform on Databricks to power AI tools and insight delivery for accounting firms.
Role type
Lead Data Engineer
Builds
Unified data platform, data pipelines, MLOps frameworks, and query endpoints for the Karbon application.
Domain
Accounting / AI / Data Engineering
Deliverable
production ML models | infrastructure
Required skills
Databricks, dbt, Python, SQL, batch and streaming data technologies, Terraform, API development, data governance
Preferred skills
Azure cloud services, MLOps (MLflow, feature stores), DevOps
Technologies
Databricks, dbt, Azure, Snowflake, Terraform, Datadog
Responsibilities
Design and implement Medallion Architecture (Bronze, Silver, Gold) on Databricks; build resilient batch and real-time data pipelines; implement data replication to Snowflake; establish data quality checks and monitoring; partner with AI team to design MLOps frameworks; enforce security and RBAC models; mentor junior engineers.
Seniority
Senior, hands-on IC with leadership responsibilities
