Senior Data Engineer - Databricks/Spark/AWS
Core
Design, build, maintain, and debug secure, high-quality production data pipelines on Databricks using PySpark for workforce analytics.
Role type
Senior IC data engineer (lakehouse)
Builds
End-to-end workforce analytics pipelines using medallion lakehouse patterns
Domain
Financial services / Data engineering
Deliverable
production ML models
Required skills
PySpark, Databricks, Delta Lake, AWS (S3, EMR, Glue, Lambda, Athena), Python, SQL, CI/CD, Airflow, system design
Preferred skills
Java, Scala, Oracle, Generative AI frameworks, Databricks certifications, broader AWS expertise
Technologies
PySpark, Databricks, Delta Lake, AWS, Airflow, Git, Jenkins, CloudWatch, Tableau, Alteryx, Sigma
Responsibilities
Design and build production data pipelines on Databricks using PySpark; Tune PySpark jobs and clusters for performance and cost efficiency; Develop scalable frameworks for workforce analytics pipelines; Implement data quality checks using Delta Live Tables; Establish monitoring and alerting for ingestion issues; Automate remediation of operational issues; Deliver curated datasets to BI partners; Partner with stakeholders to shape solutions and create architecture artifacts
Seniority
Senior, hands-on IC (via careerplan.io/jobs/gb8ebbaa-senior-data-engineer-databrickssparkaws-at-j-p-morgan)
