Data Engineer bij BeFrank
Core
Building scalable data solutions and pipelines that transform raw data from various sources into data products delivering business value.
Role type
Senior Data Engineer
Builds
Production-grade data pipelines and data products for a pension company's digital platform.
Domain
Financial Services / Data Engineering
Deliverable
production ML models | product features
Required skills
Python, PySpark, Delta Lake, Medallion Architecture, CI/CD, Unit/Integration/BDD Testing, Data Quality Validation, SQL, NoSQL, REST APIs, SFTP, Azure Data Stack (Synapse, Data Factory, Functions)
Preferred skills
Experience with partitioning, schema evolution, time travel, merge/upsert patterns
Technologies
Python, PySpark, Delta Lake, Azure Synapse, Azure Data Factory, Azure Functions, Azure DevOps, S3, SQL, NoSQL
Responsibilities
Design and implement scalable data pipelines using the medallion architecture; Connect diverse source systems (databases, APIs, storage); Write automated tests and integrate into CI/CD; Design and implement data quality validations; Collaborate with analysts on data models; Contribute to team knowledge sharing and code reviews.
Seniority
Senior, hands-on IC
