Senior Data Platform Engineer

Salary: $145,600 - 156,000 per year
Requirements:
We need 5+ years of experience in data engineering, platform engineering, DevOps, software engineering, or data-platform operations. We need strong production experience with Apache Airflow and hands-on experience with Astronomer or a comparable managed Airflow platform. We need strong Python development skills, including DAG design and CI/CD support. We need experience delivering pipelines across multiple controlled environments using Git-based workflows. We need strong experience with Azure Data Factory and Microsoft SQL Server, including advanced SQL troubleshooting. We need integration experience with databases, REST APIs, files, cloud storage, and enterprise applications. We need experience with automated testing for data pipelines and orchestration code, along with logging, monitoring, alerting, retry, and failure-recovery patterns. We need the ability to diagnose complex production issues across application, orchestration, database, network, and infrastructure layers. We need a working understanding of secrets management, identity, access control, and secure configuration. We prefer a bachelors degree in computer science, data engineering, software engineering, information systems, or a related field, or equivalent professional experience. Preferred technologies include Azure DevOps Pipelines, GitHub Actions, GitLab CI, Jenkins, Docker, Kubernetes, Terraform, Azure Key Vault, managed identities, ADLS Gen2, Databricks, Spark, PySpark, Delta Lake, Kafka, dbt, VaultSpeed or Data Vault 2.0, OpenTelemetry, Azure Monitor, and Grafana or Prometheus. Experience in manufacturing, ERP, finance, supply-chain, or operational data is a plus.
Responsibilities:
We own the technical implementation and operational support of Astronomer and Apache Airflow. We develop, maintain, test, and troubleshoot Airflow DAGs. We set standards for DAG structure, naming, dependencies, retries, scheduling, alerting, and error handling. We build and maintain CI/CD pipelines for Airflow DAGs, plugins, Python packages, configuration, and related components. We automate deployments across development, QA, UAT, and production environments. We implement automated validation, unit testing, integration testing, and deployment checks for data pipelines. We establish source-control, branching, pull-request, code-review, and release-management practices. We manage Airflow connections, variables, secrets, pools, queues, executors, and environment settings. We integrate Airflow with Azure Data Factory, Microsoft SQL Server, APIs, file systems, cloud storage, and other platforms. We develop and support Azure Data Factory pipelines, datasets, linked services, triggers, parameters, and integration runtimes. We orchestrate ADF pipelines from Airflow when needed. We develop and optimize SQL Server extraction, transformation, and loading processes. We troubleshoot SQL queries, stored procedures, connectivity, performance, and pipeline-related failures. We build reusable operators, hooks, sensors, libraries, and pipeline templates. We implement pipeline monitoring, logging, alerting, dashboards, and operational metrics. We improve pipeline reliability, recoverability, scalability, and performance. We define retry, restart, backfill, catch-up, and failure-recovery procedures. We implement secure secret management and remove credentials from code and configuration. We support network connectivity, identity, access control, certificates, and private endpoints with infrastructure and security partners. We maintain technical documentation, deployment procedures, support runbooks, and architectural diagrams. We mentor data engineers on Airflow, Python, testing, CI/CD, and production support practices. We participate in evaluating and potentially adopting Databricks and other modern data-platform technologies.
Technologies:
Airflow Azure CI/CD Cloud Data Vault Databricks DevOps Docker ERP Git GitHub GitLab Grafana Support Jenkins Kafka Kubernetes Network OpenTelemetry Prometheus Python PySpark REST SQL Security Spark Terraform dbt AI
More:
We are hiring for a remote Senior Data Platform Engineer on a contract-to-hire basis. This role supports enterprise data pipelines with a focus on Astronomer, Apache Airflow, automation, testing, observability, and operational reliability. The position pays $70.00 to $75.00 per hour on a W2 basis, and W2 consultants are eligible for medical, dental, and vision coverage, a 401(k) with company match, and life insurance. We are a strategic consulting firm with nearly 40 years of experience helping organizations achieve stronger outcomes through technology, business advisory, and life sciences solutions. We value transparency, inclusion, and equal opportunity in our hiring practices, and we offer a referral program as part of our talent network.
last updated 30 week of 2026