Databricks SME
Core
Lead data management and optimization for data at rest and streaming, designing ETL pipelines and Spark-based solutions.
Role type
Senior Data Engineer (Databricks SME)
Builds
ETL pipelines, real-time data processing solutions, optimized data storage and retrieval systems
Domain
Big Data, Cloud Data Platforms
Deliverable
production ML models
Required skills
Python, Scala, Java, Apache Spark, Databricks, SQL, NoSQL, data streaming technologies, data pipeline orchestration
Preferred skills
AI/ML concepts, Cribl, Airflow, data security and compliance principles
Responsibilities
Provide technical leadership in data management, design and maintain ETL pipelines, optimize data processing using Databricks and Spark, manage streaming data sources, implement data transformation and enrichment, work on data optimization initiatives, ensure data security and compliance, build documentation for data processes
Seniority
Senior, hands-on IC