Databricks Data Engineer
Core
Design, build, and operate scalable lakehouse solutions on Databricks to ingest, curate, and transform multi-source data into enterprise data products for the VA's ATLAS platform.
Role type
Senior Databricks Data Engineer
Builds
Scalable batch and streaming data pipelines, Delta Lake tables, and real-time ingestion solutions for the VA's multi-cloud enterprise data platform.
Domain
Healthcare / Government / Big Data
Deliverable
production ML models
Required skills
Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, Unity Catalog, Structured Streaming, Python, SQL, ETL/ELT, CI/CD, Git, Cloud platforms (AWS/Azure/GCP), Data governance
Preferred skills
Databricks certification, MLflow, Machine Learning, AI, NLP, Kafka, Hadoop, Hive, Amazon EMR, NoSQL, Data quality frameworks, Visualization tools (Plotly, Seaborn, ggplot2)
Technologies
Databricks, Apache Spark, Delta Lake, Unity Catalog, Kafka, MLflow, Git, AWS, Azure, Google Cloud
Responsibilities
Design and maintain scalable batch and streaming data pipelines; Build and manage Delta Lake tables using medallion architecture; Develop real-time data ingestion solutions; Configure and manage Databricks clusters and workflows; Implement data governance and security using Unity Catalog; Optimize data workflows for performance and cost; Integrate with CI/CD pipelines; Collaborate on data models for ML/AI use cases; Monitor and troubleshoot data processing jobs; Document data processes and pipelines.
Seniority
Senior, hands-on IC