Submitting Functions

class scalable.ScalableClient(cluster, *args, **kwargs)

Bases: Client

Client for submitting tasks to a Dask cluster. Inherits the dask client object.

Parameters:

cluster (Cluster) – The cluster object to connect to for submitting tasks.

__init__(cluster, *args, **kwargs)

scalable.ScalableClient.submit(self, func, *args, tag=None, n=1, **kwargs)

Submit a function to be ran by workers in the cluster.

Parameters:

• func (function) – Function to be scheduled for execution.

• *args (tuple) – Optional positional arguments to pass to the function.

• tag (str (optional)) – User-defined tag for the container that can run func. If not provided, func is assigned to be ran on a random container.

• n (int (default 1)) – Number of workers needed to run this task. Meant to be used with tag. Multiple workers can be useful for application level distributed computing.

• **kwargs (dict (optional)) – Optional key-value pairs to be passed to the function.

Examples

>>> c = client.submit(add, a, b)

Returns:

Returns the future object that runs the function.

Return type:

Future

Raises:

• TypeError – If ‘func’ is not callable, a TypeError is raised.

• ValueError – If ‘allow_other_workers’is True and ‘workers’ is None, a ValueError is raised.

scalable.ScalableClient.map(self, func, *parameters, tag, n, **kwargs)

Map a function on multiple sets of arguments to run the function multiple times with different inputs.

Parameters:

• func (function) – Function to be scheduled for execution.

• parameters (list of lists) – Lists of parameters to be passed to the function. The first list should have the first parameter values, the second list should have the second parameter values, and so on. The lists should be of the same length.

• tag (str (optional)) – User-defined tag for the container that can run func. If not provided, func is assigned to be ran on a random container.

• n (int (default 1)) – Number of workers needed to run this task. Meant to be used with tag. Multiple workers can be useful for application level distributed computing.

• *args (tuple) – Positional arguments to pass to dask client’s map method.

• **kwargs (dict) – Keyword arguments to pass to dask client’s map method.

Examples

>>> def add(a, b): ...
>>> L = client.map(add, [[1, 2, 3], [4, 5, 6]])

Returns:

Returns a list of future objects, each for a separate run of the function with the given parameters.

Return type:

List of futures

scalable.ScalableClient.get_versions(self, check=False, packages=None)

Return version info for the scheduler, all workers and myself

Parameters:

• check (bool) – Raise ValueError if all required & optional packages do not match. Default is False.

• packages (list) – Extra package names to check.

Examples

>>> c.get_versions()

>>> c.get_versions(packages=['sklearn', 'geopandas'])

scalable.ScalableClient.cancel(self, futures, *args, **kwargs)

Cancel running futures This stops future tasks from being scheduled if they have not yet run and deletes them if they have already run. After calling, this result and all dependent results will no longer be accessible

Parameters:

• futures (future | future, list) – One or more futures to cancel (as a list).

• *args (tuple) – Positional arguments to pass to dask client’s cancel method.

• **kwargs (dict) – Keyword arguments to pass to dask client’s cancel method.

scalable.ScalableClient.close(self, timeout=<no_default>)

Close this client

Clients will also close automatically when your Python session ends

Parameters:

timeout (number) – Time in seconds after which to raise a dask.distributed.TimeoutError