{% extends "cctbx/header.html" %} {% load static %} {% block content %}

##
How to run any method with multiprocessing

This tutorial describes how to convert (almost) any method to run with multiprocessing

### Setting up a very simple method with multiprocessing

Although setting up multiprocessing can seem intimidating, it is actually amazingly easy to do in most cases. We are going to set up an example where we first run without multiprocessing and then we convert the method to run with multiprocessing.

The key to easy multiprocessing is setting up your function in a certain way. We are going to separate the function that does something from the list of inputs to that function (which are going to change for the N jobs that are run).

First, let's set up some imports:

    from __future__ import print_function  # so we can use print function easily
    import sys        # import sys
    from libtbx import group_args #  group_args function we will use to return items

We have imported the `print_function` here so that this method will work in both python 2 and python 3.

Here is our function:

    def run_something(value):  # simple function
      return value * 2    # just returns 2 * the input value

And here is our list of inputs to the function that we would like to run in parallel:

    iteration_list = [5,7,9]  # list of anything

Let's first set up this calculation running one by one through the values in the values in `iteration_list`. It is a good idea to start this way even if you know from the start that you want to run with multiprocessing because once you have set it up in the way we have it below it is very easy to convert.

    def run_one_by_one(iteration_list): #
      result_list = []                             # initialize result list
      for i in range(len(iteration_list)):         # iterate over input values
        result = run_something(iteration_list[i])  # get result
        result_list.append(result)  # save result
      return result_list   # return the list of results

We can run `run_one_by_one` and get a list of results:

    result_list = run_one_by_one(iteration_list)  # get results one by one
    print(result_list)  # prints list of values  [10, 14, 18]

Now let's convert the `run_one_by_one` method to run with multiprocessing.

    def run_in_parallel(iteration_list, nproc = 4): #
      from libtbx.easy_mp import simple_parallel  # import the simple_parallel code
      result_list = simple_parallel(      # run in parallel
        iteration_list = iteration_list,  # our list of values to run with
        function = run_something,         # the method we are running
        nproc = nproc )                   # how many processors
      return result_list   # return the list of results

Now we can run `run_in_parallel` and get a list of results:

    result_list = run_in_parallel(iteration_list, nproc = 4)  # run in parallel
    print(result_list)  # prints list of values  [10, 14, 18]

Wow that was easy. All we did was replace the explicit iteration over our iteration_list of values and calls to run_something with a call to simple_parallel with these as arguments and specifying how many processors to use.

Bottom line on this: set up your calculation as an iteration over some list of values where the results are appended to a list and the list of result is returned. Then you can use simple_parallel without changing anything else at all.

### Advanced multiprocessing made easy

You can set up multiprocessing in a more complicated situation pretty easily as well. Suppose we want to run a method that is going to do something with a really big input (like a map) that is going to be the same for every job, and some inputs that are going to be different for each job (like a different set of coordinates each time).

In this example the key features are the big object (which we do not want to copy and we do not want to write to disk and we may not be able to pickle) and some list of sets of inputs, one set of inputs for each job.

This is going to look a lot like the easy example, but it will have some new features.

Here is the function we want to run. We are going to call our fixed big object `big_object`. (It can have any name, and there can be any number of such inputs, or none of them). We are going to call this function with an `info` object (a `group_args` object) that is going to be different for each job and that contains values of `info.value` and `info.index`. (Once again, this info object can contain anything you want.) In this case, we are going to multiply `info.value` by two and add the value of `big_object` at index `info.index`:

    def run_advanced(info, big_object = None,  #
         log = sys.stdout):   #  we can specify the log in this method if we want
      output_value = info.value * 2 + big_object[info.index]   # our result
      print("Value: %s Index: %s Output value: %s" %(info.value, info.index, output_value), file = log)
      return group_args( #
        group_args_type = 'Result from one job',  #
        input_info = info,   #
        output_value = output_value,)   #

In this example function we have added two new features:

- A log stream. If you add the keyword `log = sys.stdout` to your function and have print statements in your function that print to `log` then the `simple_parallel` procedure will handle all the text that is printed.
- We are returning a `group_args` object and not a simple value. The use of `group_args` for a return value allows returning multiple items in a simple way and allows saving and printing output text.

Note that we are returning the input `info` object as an element of the result `group_args` object. We don't have to do that, but it shows that you can return something that will identify which output came from which inputs.

Here is how we are going to set up to run this without multiprocessing. This time we are going to make our `iteration_list` be a list of the `info` `group_args` objects that specify what inputs are to be used in each job, and set our `big_object` to be a little array:

    iteration_list = []   # initialize
    from libtbx import group_args
    for i in [5,7,9]:  # our values to vary for each job
      iteration_list.append(   # a list of info objects
        group_args(   # info object (group_args)
          group_args_type = 'value of info for one job',   # title
          value = i,   # value of value
          index = 2)   # value of index
        )   #

    big_object = [0,1,2,3]  # just some supposedly big object

... and we can run it like this:

    def advanced_run_as_is(iteration_list,  #
         big_object = None,  #
         log = sys.stdout): # run in usual way
      result_list = []   # initialize
      for i in range(len(iteration_list)):     #  iterate through jobs
        result = run_advanced(iteration_list[i],
          big_object = big_object,
          log = log)  # run job
        result_list.append(result)   #
      return result_list    # return list of results

We can run this and print out each result as a `group_args` object. Notice how convenient it is that each `result` records both the inputs and the output.

The use of the `log` keyword allows us to capture the log output if we want to.

    result_list = advanced_run_as_is( #
       iteration_list, big_object = big_object,
       log = sys.stdout) #
    for result in result_list:  # run through results
      print("\nOne result:\n%s" %str(result))  # print this result (it is a group_args object)

Now we can run the same job in parallel with just tiny changes:

    def advanced_run_in_parallel(iteration_list,  #
          big_object = None, nproc = 4, log = sys.stdout): # run in parallel w
      from libtbx.easy_mp import simple_parallel  #
      result_list = simple_parallel(  #
        iteration_list = iteration_list, # list of varying inputs
        big_object = big_object, # any number of keyword arguments allowed
        function = run_advanced,  # function to run
        nproc = 3,   # number of processors
        verbose = False,   # non-verbose output
        log = log,
        )
      return result_list

With `verbose = False` the output from the first job run (actually the first batch of jobs if more than one job is in each batch as will be the case if there are many more jobs than processors used) is sent to the log stream (here that is `sys.stdout`) and all the other output is ignored. If we had set instead set `verbose = True` then all the output would be printed (hard to read but useful for debugging).

We can now run and print out each result as a `group_args` object:

    result_list = advanced_run_in_parallel(  #
        iteration_list, #
        big_object = big_object,  #
        log = sys.stdout) #
    for result in result_list:  # run through results
      print("\nOne result:\n%s" %str(result))  # print this result (it is a group_args object)

{% endblock %}