Lead Software Engineer
Core
Lead the design and delivery of scalable PySpark and SQL pipelines that transform market risk data into analytics-ready datasets.
Role type
Senior IC data engineering lead (market risk)
Builds
Production data pipelines, data models, and analytical data products for interactive reporting and business decision-making
Domain
Financial services / Market risk / Data engineering
Deliverable
production ML models
Required skills
Python, PySpark, SQL, distributed data processing, data modeling, data quality, data lineage, cloud data platforms, automated testing, code review, technical leadership
Preferred skills
Databricks, Delta Lake, Unity Catalog, LLMs, agentic techniques, infrastructure as code, data observability, performance optimization
Technologies
PySpark, SQL, Databricks, Delta Lake, Amazon S3
Responsibilities
Design and optimize data pipelines; define reusable engineering patterns; build data models for reporting; partner with stakeholders to clarify requirements; guide architecture across batch processing and lakehouse storage; establish AI-assisted development practices; mentor engineers; resolve performance and operational issues; communicate technical risks and recommendations
Seniority
Senior, hands-on IC with leadership responsibilities