Databricks Data Engineer/Architect @ Spyrosoft
Core
Designing and maintaining robust data pipelines and distributed data processing systems using Databricks to modernize enterprise data infrastructure and provide reliable insights.
Role type
Senior Data Engineer / Architect
Builds
High-performance enterprise data platform managing large-scale structured and unstructured datasets
Domain
Big Data, Data Engineering, Cloud Data Platforms
Deliverable
production ML models | product features | infrastructure
Required skills
Python, SQL, Databricks (Unity Catalog, Delta Live Tables), ETL, Data Warehousing, Distributed Data Processing, Public Cloud (Azure/AWS/GCP), Data Mesh, Relational and Non-Relational Storage Design
Preferred skills
Data Governance, Security Frameworks, DevOps, CI/CD, Agile Methodologies
Technologies
Databricks, Unity Catalog, Delta Live Tables, PySpark, Azure, AWS, GCP
Responsibilities
Design and maintain robust data pipelines and distributed data processing systems using Databricks; Implement and manage data governance and security frameworks via Unity Catalog; Develop sophisticated data models (Relational and Non-Relational) to support complex analytical requirements; Improve the performance and reliability of Big Data workflows and ETL processes; Work within an Agile environment, integrating DevOps and CI/CD principles into the data lifecycle; Act as a subject matter expert, guiding the team through complex big data challenges and architectural decisions.
