13.8. Internal frameworks — Open MPI 5.0.0 documentation

# 13.8. Internal frameworks

The Modular Component Architecture (MCA) is the backbone of Open MPI – most services and functionality are implemented through MCA components.

## 13.8.1. MPI layer frameworks

Here is a list of all the component frameworks in the MPI layer of Open MPI:

- `bml`: BTL management layer

- `coll`: MPI collective algorithms

- `fbtl`: point to point file byte transfer layer: abstraction for individual read: collective read and write operations for MPI I/O

- `fcoll`: collective file system functions for MPI I/O

- `fs`: file system functions for MPI I/O

- `hook`: Generic hooks into Open MPI

- `io`: MPI I/O

- `mtl`: Matching transport layer, used for MPI point-to-point messages on some types of networks

- `op`: Back end computations for intrinsic MPI_Op operators

- `osc`: MPI one-sided communications

- `pml`: MPI point-to-point management layer

- `sharedfp`: shared file pointer operations for MPI I/O

- `topo`: MPI topology routines

- `vprotocol`: Protocols for the “v” PML

## 13.8.2. OpenSHMEM component frameworks

- `atomic`: OpenSHMEM atomic operations

- `memheap`: OpenSHMEM memory allocators that support the PGAS memory model

- `scoll`: OpenSHMEM collective operations

- `spml`: OpenSHMEM “pml-like” layer: supports one-sided, point-to-point operations

- `sshmem`: OpenSHMEM shared memory backing facility

## 13.8.3. Miscellaneous frameworks

- `allocator`: Memory allocator

- `backtrace`: Debugging call stack backtrace support

- `btl`: Point-to-point Byte Transfer Layer

- `dl`: Dynamic loading library interface

- `hwloc`: Hardware locality (hwloc) versioning support

- `if`: OS IP interface support

- `installdirs`: Installation directory relocation services

- `memchecker`: Run-time memory checking

- `memcpy`: Memory copy support

- `memory`: Memory management hooks

- `mpool`: Memory pooling

- `patcher`: Symbol patcher hooks

- `pmix`: Process management interface (exascale)

- `rcache`: Memory registration cache

- `reachable`: Network reachability determination

- `shmem`: Shared memory support (NOT related to OpenSHMEM)

- `smsc`: Shared memory single-copy support

- `threads`: OS and userspace thread support

- `timer`: High-resolution timers

## 13.8.4. Framework notes

Each framework typically has one or more components that are used at run-time. For example, the `btl` framework is used by the MPI layer to send bytes across different types underlying networks. The `tcp` `btl`, for example, sends messages across TCP-based networks; the `ucx` `pml` sends messages across InfiniBand-based networks.

## 13.8.5. MCA parameter notes

Each component typically has some tunable parameters that can be changed at run-time. Use the ompi_info(1) command to check a component to see what its tunable parameters are. For example:

    shell$ ompi_info --param btl tcp

shows some of the parameters (and default values) for the `tcp` `btl` component (use `--all` or `--level 9` to show all the parameters).

Note that `ompi_info` (without `--all` or a specified level) only shows a small number a component’s MCA parameters by default. Each MCA parameter has a “level” value from 1 to 9, corresponding to the MPI-3 MPI_T tool interface levels. See the LEVELS section in the ompi_info(1) man page for an explanation of the levels and how they correspond to Open MPI’s code.

Here’s rules of thumb to keep in mind when using Open MPI’s levels:

- Levels 1-3:

  - These levels should contain only a few MCA parameters.

  - Generally, only put MCA parameters in these levels that matter to users who just need to run Open MPI applications (and don’t know/care anything about MPI). Examples (these are not comprehensive):

    - Selection of which network interfaces to use.

    - Selection of which MCA components to use.

    - Selective disabling of warning messages (e.g., show warning message XYZ unless a specific MCA parameter is set, which disables showing that warning message).

    - Enabling additional stderr logging verbosity. This allows a user to run with this logging enabled, and then use that output to get technical assistance.

- Levels 4-6:

  - These levels should contain any other MCA parameters that are useful to expose to end users.

  - There is an expectation that “power users” will utilize these MCA parameters — e.g., those who are trying to tune the system and extract more performance.

  - Here’s some examples of MCA parameters suitable for these levels (these are not comprehensive):

    - When you could have hard-coded a constant size of a resource (e.g., a resource pool size or buffer length), make it an MCA parameter instead.

    - When there are multiple different algorithms available for a particular operation, code them all up and provide an MCA parameter to let the user select between them.

- Levels 7-9:

  - Put any other MCA parameters here.

  - It’s ok for these MCA parameters to be esoteric and only relevant to deep magic / the internals of Open MPI.

  - There is little expectation of users using these MCA parameters.

See this section for details on how to set MCA parameters at run time.