Data Engineer with DataBricks
Core
Build and maintain scalable data processing solutions, batch and streaming pipelines, and optimize big data workloads on cloud-based data platforms.
Role type
Senior Data Engineer (Databricks/Spark)
Builds
Scalable data pipelines, ETL/ELT processes, data lakes, and data warehousing solutions
Domain
Financial services, Banking, Energy
Deliverable
production ML models
Required skills
Python, Scala, Apache Spark, Spark SQL, Delta Lake, Databricks Workflows, SQL query optimization, ETL/ELT frameworks, Data Warehousing concepts, Cloud platforms (Azure/AWS/GCP Databricks)
Preferred skills
None stated
Technologies
Python, Scala, Apache Spark, Databricks, Delta Lake, Azure, AWS, GCP
Responsibilities
Design, develop, and maintain scalable data pipelines; Build and optimize ETL/ELT processes on Databricks Lakehouse Platform; Develop high-performance data processing solutions for large-scale datasets; Work with structured, semi-structured, and unstructured data sources; Collaborate with Data Architects, Business Analysts, and Data Scientists; Perform performance tuning and optimization of Spark workloads; Design and maintain data models, data lakes, and data warehousing solutions; Implement data quality, governance, and security standards; Troubleshoot production issues and support data platform operations
Seniority
Senior, hands-on IC