Lead Software Engineer Databricks

Salary: $61,000 - 101,000 per year
Requirements:
We expect formal software engineering training or certification, plus 5+ years of hands-on experience. We are looking for advanced experience in software and data engineering, with substantial production delivery using Apache Spark on Databricks and/or AWS EMR. We require deep practical expertise across Databricks features such as Delta Lake, Unity Catalog, Workflows, Repos/notebooks, and SQL Warehouses, including cluster setup and optimization. We need proven ability to architect, build, and support dependable batch and streaming ETL/ELT pipelines, including schema design and evolution, service-level expectations, and reliability practices. We value strong Spark performance-tuning skills, including diagnosing bottlenecks and improving scalability, cost efficiency, and runtime. We expect strong programming ability in Python and/or Java for data processing, platform tooling, and automation. We require strong SQL and analytical data modeling expertise, including dimensional and star schema design, alongside Lakehouse best practices. We are seeking demonstrated experience leading the use of approved AI-assisted development tools, including clear validation standards for correctness, performance, and security. We require a strong responsible-AI and security-first mindset, including data sensitivity awareness, secure input/output handling, secrets management, encryption, network controls, and resiliency practices. We prefer experience with Delta Live Tables and advanced Databricks governance, including catalogs, grants, and auditing. We prefer AWS networking knowledge, including VPCs, subnets, routing, security groups, and data egress controls. We prefer experience using Terraform for infrastructure deployment. We prefer cost-optimization experience covering autoscaling, spot versus on-demand choices, auto-termination, storage layout, and compaction. We prefer familiarity with Airflow, Genie, Streamlit, and React. We prefer observability experience for data systems, including freshness, completeness, lineage, SLAs, and alerting. We prefer leadership experience in code quality, reviews, testing strategy, CI/CD, and technical mentorship, along with strong stakeholder communication.
Responsibilities:
We will expect you to lead the design and delivery of high-throughput, low-latency data pipelines on Databricks using Apache Spark. We will rely on you to shape and mature Lakehouse patterns with Delta Lake so our data platforms remain performant and maintainable at scale. You will own our Databricks cluster strategy and configuration, including runtime selection, autoscaling, sizing, Spark settings, init scripts, cluster policies, pools, and instance profiles. You will orchestrate and automate pipelines and jobs through Databricks Workflows, integrating AWS eventing and orchestration services where needed. You will design secure ingestion and transformation frameworks, including table design, ingestion tasks, and Airflow DAGs that produce trusted datasets. We will expect you to enforce data quality, lineage, and governance using Unity Catalog and/or AWS Glue Catalog, embedding validation directly into pipelines. You will drive Spark and Databricks performance engineering and tuning to improve cost efficiency and throughput. You will build and maintain reusable libraries, frameworks, and APIs in Python and/or Java, with strong unit, integration, and data validation coverage. You will implement CI/CD for data projects using Git-based workflows, Terraform-based infrastructure deployments, environment promotion, and automated releases. You will champion engineering standards, code reviews, and approved AI-assisted engineering practices, while ensuring secure coding, peer review, automated testing, and reuse of proven patterns. You will promote the effective use of approved AI-assisted development tools across the team to improve code quality, delivery speed, and operational outcomes. You will use SDLC tooling, including approved AI-assisted development and automation capabilities, to increase the value delivered through automation.
Technologies:
AI Airflow AWS AWS Glue Architect CI/CD Databricks ETL Git Support Java Marketing Network Python React SQL Security Spark Terraform Unity Cloud GameDev Liquid REST
More:
We are JPMorgan Chase, one of the worlds oldest financial institutions, delivering innovative financial solutions to consumers, small businesses, and major corporate, institutional, and government clients under the J.P. Morgan and Chase brands. Our history spans more than 200 years, and we are a leader in investment banking, consumer and small business banking, commercial banking, financial transaction processing, and asset management. We offer a competitive total rewards package, including base salary, and eligible roles may also receive commission and discretionary incentive compensation. We provide a broad range of benefits and programs such as comprehensive health coverage, wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, and financial coaching. We value diversity and inclusion, and we are an equal opportunity employer committed to providing reasonable accommodations where needed. This role sits within our Corporate Functions team, where our professionals support finance, risk, human resources, marketing, and other essential areas that help our businesses, clients, customers, and employees succeed.
last updated 28 week of 2026