Lead Software Engineer Databricks
Core
Lead the design and delivery of high-throughput, low-latency data pipelines on Databricks using Apache Spark to shape and mature Lakehouse patterns.
Role type
Senior IC data engineering lead (Databricks/Spark)
Builds
Dependable batch and streaming ETL/ELT pipelines, reusable libraries, frameworks, and APIs
Domain
Financial services data engineering on Databricks Lakehouse
Deliverable
production ML models | product features
Required skills
Apache Spark, Databricks (Delta Lake, Unity Catalog, Workflows), Python, Java, SQL, data modeling, CI/CD, Terraform, AWS networking, performance tuning
Preferred skills
Delta Live Tables, Airflow, observability, cost optimization, leadership in code quality and mentorship (via careerplan.io/jobs/gf21fca7-lead-software-engineer-databricks-at-j-p-morgan)
Technologies
Databricks, Apache Spark, AWS, Airflow, Terraform, Unity Catalog, Git, React, Streamlit
Responsibilities
Design and deliver high-throughput data pipelines; own Databricks cluster strategy and configuration; orchestrate and automate pipelines via Databricks Workflows; design secure ingestion and transformation frameworks; enforce data quality, lineage, and governance; drive Spark performance engineering; build and maintain reusable libraries and APIs; implement CI/CD for data projects; champion engineering standards and AI-assisted development practices.