Databricks Engineer
Core
Design, build, and maintain scalable ETL pipelines and troubleshoot production workloads for large-scale data projects within the Department of Homeland Security.
Role type
Senior Databricks Data Engineer
Builds
Scalable ETL pipelines, data jobs, and optimized compute clusters for DHS mission goals
Domain
Government / Data Engineering & Security
Deliverable
production ML models | product features
Required skills
Databricks, PySpark, SQL, Delta Lake, batch processing, stream processing, autoloader, compute optimization, CI/CD, Agile project management, data governance, Unity Catalog, Terraform
Preferred skills
Kafka, Event Hub, Azure DevOps, Jenkins, GitHub Actions, Scrum Master Certification, commercial cloud infrastructure management
Technologies
Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Terraform, Kafka, Event Hub, Azure DevOps, Jenkins, GitHub Actions
Responsibilities
Design and maintain scalable ETL pipelines; troubleshoot Databricks job failures and cluster performance; implement CI/CD practices; lead Agile project management activities; perform code reviews and data quality validation; ensure alignment with data governance and security standards
Seniority
Senior, hands-on IC
