10.4. Scheduling processes across hosts — Open MPI 5.0.1 documentation

# 10.4. Scheduling processes across hosts

Open MPI provides many options for scheduling application processes across hosts, including oversubscribing processes to processors. This section describes how to define that mapping.

## 10.4.1. Scheduling overview

If you are not oversubscribing your hosts (i.e., trying to run more processes than slots available on that host), scheduling is pretty simple and occurs either on a by-slot or by-node round robin schedule. If you’re oversubscribing, the issue gets much more complicated — keep reading.

The more complete answer is: Open MPI schedules processes to nodes by asking two questions from each application on the `mpirun` command line:

1. How many processes should be launched?

2. Where should those processes be launched?

The “how many” question is directly answered with the `-n` switch to `mpirun`. If `-n` is not specified on the `mpirun` command line, its value is the sum of the slots on all the nodes.

The “where” question is a little more complicated, and depends on three factors:

1. The final node list (e.g., after `-hostname` / `--host` exclusionary or inclusionary processing)

2. The scheduling policy (which applies to all applications in a single job)

3. The default and maximum number of slots on each host

Open MPI currently supports two scheduling policies: by slot and by node:

1. By slot: This is the default scheduling policy, but can also be explicitly requested by using either the `--map-by slot` option to `mpirun` or by setting the MCA parameter `rmaps_default_mapping_policy` to the string `slot`.

   In this mode, Open MPI will schedule processes on a node until all of its default slots are exhausted before proceeding to the next node. In MPI terms, this means that Open MPI tries to maximize the number of adjacent ranks in `MPI_COMM_WORLD` on the same host without oversubscribing that host.

   For example:

       shell$ cat my-hosts
       node0 slots=2 max_slots=20
       node1 slots=2 max_slots=20
       shell$ mpirun --hostfile my-hosts -n 8 --map-by slot hello | sort
       Hello World I am rank 0 of 8 running on node0
       Hello World I am rank 1 of 8 running on node0
       Hello World I am rank 2 of 8 running on node1
       Hello World I am rank 3 of 8 running on node1
       Hello World I am rank 4 of 8 running on node0
       Hello World I am rank 5 of 8 running on node0
       Hello World I am rank 6 of 8 running on node1
       Hello World I am rank 7 of 8 running on node1

2. By node: This policy can be requested either by using the `--map-by node` option to `mpirun` or by setting the MCA parameter `rmaps_default_mapping_policy` to the string “node”.

   In this mode, Open MPI will schedule a single process on each node in a round-robin fashion (looping back to the beginning of the node list as necessary) until all processes have been scheduled. Nodes are skipped once their default slot counts are exhausted.

   For example:

       shell$ cat my-hosts
       node0 slots=2 max_slots=20
       node1 slots=2 max_slots=20
       shell$ mpirun --hostname my-hosts -n 8 --map-by node hello | sort
       Hello World I am rank 0 of 8 running on node0
       Hello World I am rank 1 of 8 running on node1
       Hello World I am rank 2 of 8 running on node0
       Hello World I am rank 3 of 8 running on node1
       Hello World I am rank 4 of 8 running on node0
       Hello World I am rank 5 of 8 running on node1
       Hello World I am rank 6 of 8 running on node0
       Hello World I am rank 7 of 8 running on node1

In both policies, if the default slot count is exhausted on all nodes while there are still processes to be scheduled, Open MPI will trigger an oversubscription condition.

If `:OVERSUBSCRIBE` is added as a modifier to the `--map-by` option (e.g., `mpirun --map-by node:OVERSUBSCRIBE ...` — see this section for more details), Open MPI will continue to loop through the list of nodes again and try to schedule one more process to each node until all processes are scheduled. Nodes are skipped in this process if their maximum slot count is exhausted. If the maximum slot count is exhausted on all nodes while there are still processes to be scheduled, Open MPI will abort without launching any processes.

If `:OVERSUBSCRIBE` is not specified and an oversubscription condition occurs, Open MPI will abort without launching any processes.

## 10.4.2. Scheduling with the –hostfile option

The `--hostfile` option to `mpirun` takes a filename that lists hosts on which to launch MPI processes.

Important

The hosts listed in a hostfile have nothing to do with which network interfaces are used for MPI communication. They are only used to specify on which hosts to launch MPI processes.

Hostfiles are simple text files with hosts specified, one per line. Each host can also specify a default and maximum number of slots to be used on that host (i.e., the maximum number of processes that will be launched on that node). Comments are also supported, and blank lines are ignored. For example:

    # This is an example hostfile.  Comments begin with #.
    #
    # Since no slots are specified, the number of slots defaults to the
    # number of processor cores available on the machine.
    foo.example.com

    # We want to allow launching a maximum of 2 processes on this host
    # (e.g., potentially because it has two processor cores):
    bar.example.com slots=2

Slots are discussed in much more detail in this section.

Hostfiles works in two different ways:

1. Exclusionary: If a list of hosts to run on has been provided by another source (e.g., by a hostfile or a batch scheduler such as Slurm, PBS/Torque, SGE, etc.), the hosts provided by the hostfile must be in the already-provided host list. If the hostfile-specified nodes are not in the already-provided host list, `mpirun` will abort without launching anything.

   In this case, hostfiles act like an exclusionary filter — they limit the scope of where processes will be scheduled from the original list of hosts to produce a final list of hosts.

   For example, say that a scheduler job contains hosts `node01` through `node04`. If you run:

       shell$ cat my_hosts
       node03
       shell$ mpirun -n 1 --hostfile my_hosts hostname

   This will run a single copy of `hostname` on the host `node03`.

   However, presuming your job was allocated only to `node03` and you run the following:

       shell$ cat my_hosts
       node17
       shell$ mpirun -n 1 --hostfile my_hosts hostname

   This is an error (because `node17` is not allocated to your job), and `mpirun` will abort.

   Finally, note that in exclusionary mode, processes will only be executed on the hostfile-specified hosts, If this ends up causing an oversubscription situation, `mpirun` will abort by default.

2. Inclusionary: If a list of hosts has not been provided by another source, then the hosts provided by the `--hostfile` option will be used as the original and final host list.

   In this case, `--hostfile` acts as an inclusionary agent; all `--hostfile`-supplied hosts become available for scheduling processes. For example (assume that you are not in a scheduling environment where a list of nodes is being transparently supplied):

       shell$ cat my_hosts
       node01.example.com slots=1
       node02.example.com slots=1
       node03.example.com slots=1
       shell$ mpirun -n 3 --hostfile my_hosts hostname

   This will launch a single copy of `hostname` on the hosts `node01.example.com`, `node02.example.com`, and `node03.example.com`.

Note, too, that `--hostfile` is essentially a per-application switch. Hence, if you specify multiple applications (as in an MPMD job), `--hostfile` can be specified multiple times:

    shell$ cat hostfile_1
    node01.example.com
    shell$ cat hostfile_2
    node02.example.com
    shell$ mpirun -n 1 --hostfile hostfile_1 hostname : -n 1 --hostfile hostfile_2 uptime
    node01.example.com
     06:11:45 up 1 day,  2:32,  0 users,  load average: 21.65, 20.85, 19.84

Notice that `hostname` was launched on `node01.example.com` and `uptime` was launched on `node02.example.com`.

## 10.4.3. Scheduling with the –host option

The `--host` option to `mpirun` takes a comma-delimited list of hosts on which to run. For example:

    shell$ mpirun -n 3 --host a,b,c hostname

Will launch one copy of `hostname` on each of hosts `a`, `b`, and `c`. Specifically: each host defaults to 1 slot, unless specified by the `:N` suffix. For example:

    shell$ mpirun --host a,b:2,c:3 hostname

Will launch one copy of `hostname` on `a`, two copies of `hostname` on `b`, and three copies of `hostname` and `c`.

Slots are discussed in much more detail in this section.

Important

The hosts specified by the `--host` option have nothing to do with which network interfaces are used for MPI communication. They are only used to specify on which hosts to launch MPI processes.

`--host` works in two different ways:

1. Exclusionary: If a list of hosts to run on has been provided by another source (e.g., by a hostfile or a batch scheduler such as Slurm, PBS/Torque, SGE, etc.), the hosts provided by the `--host` option must be in the already-provided host list. If the `--host`-specified nodes are not in the already-provided host list, `mpirun` will abort without launching anything.

   In this case, the `--host` option acts like an exclusionary filter — it limits the scope of where processes will be scheduled from the original list of hosts to produce a final list of hosts.

   For example, say that the hostfile `my_hosts` contains the hosts `node1` through `node4`. If you run:

       shell$ mpirun -n 1 --hostfile my_hosts --host node3 hostname

   This will run a single copy of `hostname` on the host `node3`. However, if you run:

       shell$ mpirun -n 1 --hostfile my_hosts --host node17 hostname

   This is an error (because `node17` is not listed in `my_hosts`); `mpirun` will abort.

   Finally, note that in exclusionary mode, processes will only be executed on the `--host`-specified hosts. If this ends up causing an oversubscription situation, `mpirun` will abort by default.

2. Inclusionary: If a list of hosts has not been provided by another source, then the hosts provided by the `--host` option will be used as the original and final host list.

   In this case, `--host` acts as an inclusionary agent; all `--host`-supplied hosts become available for scheduling processes. For example (assume that you are not in a scheduling environment where a list of nodes is being transparently supplied):

       shell$ mpirun -n 3 --host a,b,c hostname

   This will launch a single copy of `hostname` on the hosts `a`, `b`, and `c`.

Note, too, that `--host` is essentially a per-application switch. Hence, if you specify multiple applications (as in an MPMD job), `--host` can be specified multiple times:

    shell$ mpirun -n 1 --host a hostname : -n 1 --host b uptime

This will launch `hostname` on host `a` and `uptime` on host `b`.

## 10.4.4. Process Slots

Slots are Open MPI’s representation of how many processes can be launched on a given host.

Open MPI maintains the number of slots for each host in a given parallel job, and — by default — will not let you launch more processes on a host than it has slots.

Important

It is common to set the number of slots on a host to be less than or equal to the number of processor cores on that host.

But it is important to realize that Open MPI’s concept of slots is actually unrelated to the number of physical processor cores on a host.

Specifically: the number of slots on a host can be less than, equal to, or more than the number of processor cores on a host.

If you wish to run more processes on a host than it has slots, see the section on oversubscription.

### 10.4.4.1. Calculating the number of slots

The number of slots on a host depends on a few factors:

1. If the host is specified by a job scheduler (e.g., Slurm, PBS/Torque, etc.), the job scheduler specifies the number of slots for that host.

2. If the host is specified in a hostfile:

   1. If the `slots` parameter is specified, that value is used for the number of slots on that host.

   2. Otherwise:

      1. If `--map-by :HWTCPUS` was specified, the number of slots defaults to the number of hardware threads on that host.

      2. Otherwise, the number of slots defaults to the number of processor cores on that host.

3. If the host is specified via the `--host` command line option:

   1. If the `:N` suffix is specified, `N` is used for the number of slots on that host.

   2. Otherwise, the number of slots defaults to 1.

   3. If the same host name is specified multiple times, the slots value for that host is increased by `N` if `:N` is specified, or increased by 1 if `:N` is not specified.

Caution

The exact scheme used to determine the number of slots has varied between different major versions of Open MPI. The scheme described above is relevant for Open MPI v5.0.x.

Max slot counts, however, are rarely specified by schedulers. The max slot count for each node will default to “infinite” if it is not provided (meaning that Open MPI will oversubscribe the node if you ask it to — see more on oversubscribing in this section).

Here are some examples, all from unscheduled environments:

1. Use a hostfile and specify the `slots` parameter.

       shell$ cat my-hostfile
       node01.example.come slots=4
       shell$ mpirun --hostfile my-hostfile hostname
       node01
       node01
       node01
       node01

   This launched 4 processes because `slots=4` was specified in the hostfile.

2. Use a hostfile and do not specify the `slots` parameter (assume that `node01.example.com` has 2 processor cores):

       shell$ cat my-hostfile
       node01.example.come
       shell$ mpirun --hostfile my-hostfile hostname
       node01
       node01

   This launched 2 processes because `slots` was not specified, and `node02` has 2 processor cores.

3. Use `--host`:

       shell$ mpirun --host node01.example.com hostname
       node01

   This launched 1 processes because `--host` with no `:N` suffix increments the slot count for that host by 1.

4. Use `--host` with a `:N` suffix:

       shell$ mpirun --host node01.example.com:2 hostname
       node01
       node01

   This launched 2 processes because `:2` was specified on the command line.

5. Use `--host` with a `:N` suffix, and mention the host multiple times:

       shell$ mpirun --host node01.example.com:2,node01.example.com hostname
       node01
       node01
       node01

   This launched 3 processes because `:2` was specified on the command line, and then `node01.example.com` was specified an additional time, incrementing the slot count for that host to 3.

## 10.4.5. Oversubscribing nodes

Running more MPI processes than processors are available on a node is called oversubscribing the node. Open MPI can oversubscribe nodes, but it very much matters how you do it.

Specifically: it is critical that Open MPI knows that you are oversubscribing the node, or severe performance degradation can result.

Important

Here is a good general rule to follow: never specify a number of slots that is more than the available number of processors.

For example, if you want to run 4 processes on a host with 2 processor cores, then indicate that you only have 2 slots but want to run 4 processes. For example:

    # In a hostfile, the number of slots will default to the number of
    # processor cores on the host
    shell$ cat my-hostfile
    localhost
    shell$ mpirun -n 4 --hostfile my-hostfile a.out

Specifically: we strongly suggest that you do NOT have a hostfile that contains `slots=4` (because there are only two available processor cores).

That being said, the above command will fail, because you are trying to run 4 processes but there are only 2 slots available. You must specifically tell Open MPI that it is ok to oversubscribe via `--map-by :OVERSUBSCRIBE`:

    shell$ cat my-hostfile
    # For the purposes of this example, explicitly tell Open MPI
    # that we have 2 slots on the host.
    localhost slots=2
    shell$ mpirun -n 4 --hostfile my-hostfile --map-by :OVERSUBSCRIBE a.out

The reason you should tell Open MPI whether you’re oversubscribing or not (i.e., never specify a `slots` value more than the number of processor cores available) is because Open MPI basically runs its message passing progression engine in two modes: aggressive and degraded.

1. Degraded: When Open MPI thinks that it is in an oversubscribed mode (i.e., more processes are running than there are processor cores available), MPI processes will automatically run in degraded mode and frequently yield the processor to its peers, thereby allowing all processes to make progress.

   Note

   Be sure to see this FAQ entry that describes how degraded mode affects processor and memory affinity.

2. Aggressive: When Open MPI thinks that it is in an exactly- or under-subscribed mode (i.e., the number of running processes is equal to or less than the number of available processor cores), MPI processes will automatically run in aggressive mode, meaning that they will never voluntarily give up the processor to other processes. With some network transports, this means that Open MPI will spin in tight loops attempting to make message passing progress, effectively causing other processes to not get any CPU cycles (and therefore never make any progress).

For example, on a node with a two processor cores:

    shell$ cat my-hostfile
    localhost slots=4
    shell$ mpirun -n 4 --hostfile my-hostfile a.out

This would cause all 4 MPI processes to run in aggressive mode because Open MPI thinks that there are 4 available processor cores to use. This is actually a lie (there are only 2 processor core — not 4), and can cause extremely bad performance.

## 10.4.6. Forcing aggressive or degraded performance mode

The MCA parameter `mpi_yield_when_idle` controls whether an MPI process runs in Aggressive or Degraded performance mode. Setting it to 0 forces Aggressive mode; setting it to 1 forces Degraded mode (see this FAQ entry to see how to set MCA parameters).

Note that this value only affects the behavior of MPI processes when they are blocking in MPI library calls. It does not affect behavior of non-MPI processes, nor does it affect the behavior of a process that is not inside an MPI library call.

Open MPI normally sets this parameter automatically and users are cautioned against setting this parameter unless you are really, absolutely, positively sure of what you are doing.