Senior Data Engineer
Core
Design, build, and maintain enterprise-grade Databricks platform architecture, ensuring reliability, security, and cost efficiency for AI/ML workloads.
Role type
Senior Data Engineer (Platform & Infrastructure)
Builds
Self-service Databricks platform, reference architectures, and AI/ML infrastructure patterns
Domain
Cloud Data Engineering / AI Infrastructure
Deliverable
infrastructure
Required skills
Databricks administration, Infrastructure-as-Code (Terraform), CI/CD, Python, PySpark, SQL, AWS fundamentals, Platform security & governance, FinOps, Distributed systems design
Preferred skills
Databricks Certified Data Engineer Professional, AWS Certified Solutions Architect, Compliance-driven platform experience, Mentoring experience, Platform migration experience
Technologies
Databricks, Unity Catalog, Lakeflow, MLflow, Model Serving, Terraform, GitHub Actions, AWS (S3, IAM, VPC, KMS), PySpark, Delta Lake, Auto Loader
Responsibilities
Own workspace and account-level Databricks architecture including Unity Catalog and compute policies; Design and review infrastructure-as-code and CI/CD pipelines; Manage platform reliability, incident management, and disaster recovery; Implement FinOps activities for cost optimization; Enforce platform security and governance controls; Enable Databricks capabilities for AI/ML workloads; Build and maintain reference implementations for PySpark and SQL; Lead code reviews and mentor engineers.
Seniority
Senior, hands-on IC with mentorship responsibilities