Scala - Spark Developer
Core
Develop and maintain large-scale regulatory reporting data pipelines using Scala and Apache Spark on Databricks and Snowflake.
Role type
Mid-level data engineer (Scala/Spark)
Builds
Production data pipelines for regulatory reporting
Domain
Financial services / Regulatory reporting
Deliverable
production ML models | product features
Required skills
Scala, Apache Spark, Snowflake, SQL, Python, BDD testing, Unix/Linux shell scripting, batch job scheduling (Autosys), data validation, DAG optimization
Preferred skills
Trade lifecycle concepts, Equities/Options asset classes, Front Office data, Generative AI use cases, large-scale data environments
Technologies
Scala, Apache Spark, Databricks, Snowflake, Autosys, Python, BDD frameworks
Responsibilities
Develop and enhance Spark-based data pipelines, implement business logic for regulatory reporting, develop automation utilities and operational controls, work with datasets stored in Snowflake, support batch workflow setup and maintenance, write and maintain BDD test cases and unit tests, participate in code reviews and design discussions, provide basic support during production issues
Seniority
Mid-level, hands-on IC
