Azure Data Integration Engineer
Core
Design, develop, and maintain scalable Azure-based pipelines and ETL/ELT solutions to ingest, condition, curate, and transform multi-source data into standardized enterprise data products for the VA's ATLAS platform.
Role type
Senior Azure Data Integration Engineer (Cloud Data Architecture & BI)
Builds
Scalable data pipelines, data lakes, warehouses, lakehouses, and Power BI dashboards for the VA's multi-cloud enterprise data management platform.
Domain
Healthcare / Public Sector / Cloud Data Engineering
Required skills
Azure Data Factory, Synapse Analytics, Azure Data Lake, Databricks, Apache Spark, SQL (T-SQL), Power BI (DAX, M), Python, Kafka, Event Hub, Azure DevOps, Terraform, Bicep, Row-Level Security, CI/CD
Preferred skills
Azure Data Engineer Associate (DP-203), Power BI Data Analyst Associate (PL-300), CosmosDB, Delta Lake, NoSQL, Machine Learning integration, Looker, Tableau, Azure Government Cloud
Technologies
Azure Data Factory, Synapse Analytics, Azure Data Lake, Databricks, Apache Spark, Kafka, Event Hub, Stream Analytics, Power BI, Azure DevOps, GitHub Actions, Terraform, Bicep, Collibra, Pyramid Analytics
Responsibilities
Design and implement scalable, secure data pipelines using Azure Data Factory; Develop and optimize ETL processes for Azure Data Lake, SQL, and Synapse; Automate data ingestion from APIs, databases, and IoT streams; Build and maintain data lakes and lakehouses; Optimize storage using Parquet and Delta Lake; Implement batch and real-time streaming solutions; Integrate data from SQL Server, CosmosDB, SAP, and third-party APIs; Implement data security best practices (RBAC, Azure AD, Key Vault); Work with Purview and Data Catalog for governance; Monitor and troubleshoot pipeline performance; Optimize query performance in Synapse and SQL; Automate data quality checks and alerting; Design and develop Power BI dashboards and reports; Automate deployments with CI/CD pipelines.
Seniority
Senior, hands-on IC