Software Engineer, Ray Data

About Anyscale

Anyscale is commercializing Ray, an open-source project for scalable machine learning. Anyscale is backed by Andreessen Horowitz, NEA, and Addition with $250+ million raised to date. About the role: Ray aims to provide a universal API for building distributed applications. Data is a core element connecting different stages in Ray’s ecosystem. About the Ray Data team: The Ray Data team develops and maintains the Ray Datasets library, which is used in production by companies like Amazon and Alibaba. Ray Datasets is a Python library built on top of Apache Arrow and Ray Core. The Ray Data team interacts with Ray Core components and works with Ray’s ML libraries including Train, RLlib, and Serve. A snapshot of projects you will work on: Performance of Ray Datasets at large scale, integration with ML training and data sources, stability and stress testing infrastructure, lead future work integrating streaming workloads into Ray, differentiate Data operations in Anyscale hosted Ray service. As part of this role, you will: Develop high quality open source software to simplify distributed programming (Ray), identify, implement, and evaluate architectural improvements to Ray core and Datasets, improve the testing process for Ray to make releases as smooth as possible, communicate your work to a broader audience through talks, tutorials, and blog posts. We'd love to hear from you if have: At least 5 years of relevant work experience, solid background in algorithms, data structures, system design, experience in building scalable and fault-tolerant distributed systems, experience with data processing, database internals including Spark or Dask (streaming is a plus).