Lead Software Engineer - Data & AI Platform
Core
Design, build, and maintain scalable data pipelines, ETL/ELT workflows, and production-grade data platforms supporting Agentic AI, LLMs, and RAG architectures.
Role type
Lead Software Engineer - Data & AI Platform
Builds
Scalable data pipelines, data cataloging, data quality frameworks, semantic/metrics layers, and Agentic Autonomous Lakehouse capabilities.
Domain
Financial services, Data Engineering, Artificial Intelligence (LLMs, RAG, Agentic AI)
Required skills
System design, application development, testing, operational stability, Python, Java, SQL, Apache Spark, Apache Flink, data modeling (star schema, snowflake), query optimization, Apache Airflow, AWS data services (S3, Glue, Redshift, Athena, EMR), Kubernetes, Apache Iceberg, Unity Catalog, OpenMetadata, CI/CD, security, Agentic AI, LLMs, RAG architectures, MCP, vector databases, embedding-based retrieval systems
Preferred skills
Data mesh or data product architectures, Infrastructure as Code (Terraform), containerized deployments (Docker, Kubernetes), data observability, quality, and metadata management tools, semantic layers, metrics stores, BI platforms (Tableau, dbt Metrics)
Technologies
Apache Spark, Apache Flink, Apache Airflow, AWS, Databricks, Kubernetes, Apache Iceberg, Unity Catalog, OpenMetadata, Terraform, Docker, Kafka, dbt, Tableau, MCP
Responsibilities
Design and build scalable data pipelines and ETL/ELT workflows for batch and real-time processing. Develop data platform capabilities including data cataloging, data quality frameworks, and semantic/metrics layers with governance. Implement data modeling strategies to support analytics and reporting. Partner with analytics teams and stakeholders to turn data needs into production-ready solutions. Produce secure, high-quality production code and review/debug code written by others. Identify opportunities to automate remediation of recurring issues to strengthen operational stability. Lead evaluation sessions with vendors and internal teams to assess architectural designs. Lead development of the Agentic Autonomous Lakehouse capability. Lead communities of practice across Software Engineering to promote emerging technologies.
Seniority
Lead, hands-on IC with mentorship responsibilities
