Senior Data Engineer
Core
Develop and optimise data pipelines and ETL/ELT workflows in Azure Databricks and Azure Data Factory for the Australian Bureau of Agricultural and Resource Economics and Sciences (ABARES).
Role type
Senior Data Engineer
Builds
Scalable data pipelines and structured datasets for reporting and analytics
Domain
Agriculture, Forestry, and Fisheries (Public Sector)
Deliverable
production ML models | product features | dashboards & analysis
Required skills
Python, SQL, Azure Databricks, Azure Data Factory, Azure DevOps, Git, Kimball dimensional modelling, Medallion architecture
Preferred skills
Data transformation automation, query optimisation, CI/CD pipeline management, data lake structuring, data modelling best practices
Technologies
Azure Databricks, Azure Data Factory, Azure DevOps, Git
Responsibilities
Develop and maintain data pipelines using Python and SQL; Design and implement ETL/ELT workflows; Apply Kimball and Medallion architecture; Collaborate with stakeholders to translate requirements; Implement CI/CD pipelines; Monitor and optimise Databricks jobs; Ensure data governance and security compliance; Contribute to code quality through peer reviews.
Seniority
Senior, hands-on IC
