Scalable Documentation

Scalable is a Python library for running complex workflows on HPC systems efficiently and with minimal manual intervention. It uses a dask backend and a range of custom programs to achieve this. The figure below shows the general architecture of Scalable.

These questions can help answering if Scalable would be useful for you:

• Is your workflow ran on a HPC system and takes a significant amount of time?

• Does your workflow involve pipelines, where outputs from certain functions or models are passed as inputs to other functions or models?

• Do you want the hardware allocation to be done automatically?

Scalable could be useful if one of more of the above questions are affirmative. To incorporate the ability to run functions under different environments, docker containers can be used. A Dockerfile with multiple targets can be used to make multiple containers, each with different installed libraries and models. When adding workers to cluster, it can be specified how many workers of each type should be added.

Contents:

Getting Started

• Getting Started with Scalable

API

• Worker Management
• Caching
• Submitting Functions

How-tos

• How to add custom hash functions for cacheable
• How to add custom containers for scalable as workers
• How to convert R code to Python code using rpy2 and run it through Scalable

Demos

• Scalable Example Workflow
• A Complete Process of Adding & Scaling a New Application

Common Issues

• Common Issues