# 4. Exceptions to the PMIx Standard

Exceptions to the base PMIx Standard are listed here. These exceptions are not indicative of any intent to stray from the Standard, but instead represent the difference between the pace of development of the library versus the normal Standard’s process. Accordingly, it is expected that the exceptions listed below will make their way into a future release of the PMIx Standard and then be removed from the list of exceptions in some future OpenPMIx release.

## 4.1. Extensions

OpenPMIx v5.0.3rc1 is based on the PMIx 4.2 Standard. In addition to Extensions to the Standard, this release includes the conversion of all support macros to PMIx function APIs — e.g., the `PMIX_LOAD_PROCID` macro is now the `PMIx_Load_procid()` function — in accordance with planned changes to the Standard. The macro versions have been retained as deprecated (without warnings) for backward compatibility.

## 4.2. Qualified Values

OpenPMIx has introduced the concept of `qualified values` to allow users to specify a value combined with one or more qualifiers.

## 4.3. Scheduler Integration APIs

- Allow a scheduler to direct the resource manager to execute a session-related operation, or allow the resource manager to report a session-related action (e.g., session terminated) to the scheduler:

      pmix_status_t PMIx_Session_control(uint32_t sessionID,
                                         const pmix_info_t *directives, size_t ndirs,
                                         pmix_info_cbfunc_t cbfunc, void *cbdata);

## 4.4. Tool APIs

- Check if the tool is connected to a PMIx server:

      bool PMIx_tool_is_connected(void);

- Allow the tool to register a server function pointer module so it can service client requests:

      pmix_status_t PMIx_tool_set_server_module(pmix_server_module_t *mod);

## 4.5. Utility APIs

- Load a key:

      void PMIx_Load_key(pmix_key_t key, const char *src);

- Check a key:

      bool PMIx_Check_key(const char *key, const char *str);

- Check to see if a key is a “reserved” key:

      bool PMIx_Check_reserved_key(const char *key);

- Load a string into a `pmix_nspace_t` struct:

      void PMIx_Load_nspace(pmix_nspace_t nspace, const char *str);

- Check two `nspace` structs for equality:

      bool PMIx_Check_nspace(const char *key1, const char *key2);

- Check if a namespace is invalid:

      bool PMIx_Nspace_invalid(const char *nspace);

- Load a process ID struct:

      void PMIx_Load_procid(pmix_proc_t *p,
                            const char *ns,
                            pmix_rank_t rk);

- Transfer a process ID struct (non-destructive):

      void PMIx_Xfer_procid(pmix_proc_t *dst,
                            const pmix_proc_t *src);

- Check two proc IDs for equality:

      bool PMIx_Check_procid(const pmix_proc_t *a,
                             const pmix_proc_t *b);

- Check two ranks for equality:

      bool PMIx_Check_rank(pmix_rank_t a,
                           pmix_rank_t b);

- Check if proc ID is invalid:

      bool PMIx_Procid_invalid(const pmix_proc_t *p);

## 4.6. Argv Handling

Functions for handling of argv arrays (`NULL`-terminated array of strings)

- Count the number of entries

    int PMIx_Argv_count(char **a);

- Append a string to the array

    pmix_status_t PMIx_Argv_append_nosize(char ***argv, const char *arg);

- Prepend a string to the array

    pmix_status_t PMIx_Argv_prepend_nosize(char ***argv, const char *arg);

- Append a string to the array, but only if it doesn’t already appear on the array (ignore if it does)

    pmix_status_t PMIx_Argv_append_unique_nosize(char ***argv, const char *arg);

- Free an array, including each string on the array

    void PMIx_Argv_free(char **argv);

- Split a string into an argv array, dividing the string on each occurrence of the specified delimiter character. Retain empty entries in the array when more than one copy of the delimiter character appears in a sequence.

    char **PMIx_Argv_split_inter(const char *src_string,
                                 int delimiter,
                                 bool include_empty);

- Split a string into an argv array, dividing the string on each occurrence of the specified delimiter character. Retain empty entries in the array when more than one copy of the delimiter character appears in a sequence. Acts as a wrapper to `PMIx_Argv_split_inter` with `include_empty` set to `true`

    char **PMIx_Argv_split_with_empty(const char *src_string, int delimiter);

- Split a string into an argv array, dividing the string on each occurrence of the specified delimiter character. Discard empty entries in the array when more than one copy of the delimiter character appears in a sequence. Acts as a wrapper to `PMIx_Argv_split_inter` with `include_empty` set to `false`

    char **PMIx_Argv_split(const char *src_string, int delimiter);

- Join all the elements of an argv array into a single newly-allocated string, with the specified delimiter character at the join points.

    char *PMIx_Argv_join(char **argv, int delimiter);

- Copy a `NULL`-terminated argv array.

    char **PMIx_Argv_copy(char **argv);

- Set environment variable:

      pmix_status_t PMIx_Setenv(const char *name,
                                const char *value,
                                bool overwrite,
                                char ***env);

## 4.7. Value Struct Functions

- Initialize a value struct:

      void PMIx_Value_construct(pmix_value_t *val);

- Free memory stored inside a value struct:

      void PMIx_Value_destruct(pmix_value_t *val);

- Create and initialize an array of value structs:

      pmix_value_t* PMIx_Value_create(size_t n);

- Free memory stored inside an array of coord structs (does not free the struct memory itself):

      void PMIx_Value_free(pmix_value_t *v, size_t n);

- Check the given value struct to determine if it includes a boolean value (includes strings for `true` and `false`, including abbreviations such as `t` or `f`), and if so, then its value. A value type of `PMIX_UNDEF` is taken to imply a boolean `true`.

      pmix_boolean_t PMIx_Value_true(const pmix_value_t *v);

- Compare the contents of two `pmix_value_t` structures:

      pmix_value_cmp_t PMIx_Value_compare(pmix_value_t *v1,
                                          pmix_value_t *v2);

- Get the size of the contents of a `pmix_value_t` structure:

      pmix_status_t PMIx_Value_get_size(const pmix_value_t *val,
                                        size_t *size);

## 4.8. Data Array Functions

- Construct a data array object, allocating the memory for the indicated number of the specified data type. Memory for the provided data array object must have previously been allocated or statically declared:

      void PMIx_Data_array_construct(pmix_data_array_t *p,
                                     size_t num, pmix_data_type_t type);

- Initialize the fields of a data array object without allocating any memory for the included array:

      void PMIx_Data_array_init(pmix_data_array_t *p,
                                pmix_data_type_t type);

- Destroy a data array object, releasing all memory included in it:

      void PMIx_Data_array_destruct(pmix_data_array_t *d);

- Create and initialize a `pmix_data_array_t` structure, allocating the memory for the indicated number of the specified data type as well as the `pmix_data_array_t` object itself:

      pmix_data_array_t* PMIx_Data_array_create(size_t n, pmix_data_type_t type);

- Free memory stored inside a `pmix_data_array_t` structure (does not free the provided `pmix_data_array_t` object itself):

      void PMIx_Data_array_free(pmix_data_array_t *p);

## 4.9. Info Struct Functions

- Initialize an info struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Info_construct(pmix_info_t *p);

- Free memory stored inside an info struct:

      void PMIx_Info_destruct(pmix_info_t *p);

- Create and initialize an array of info structs:

      pmix_info_t* PMIx_Info_create(size_t n);

- Free memory stored inside an array of coord structs (does not free the struct memory itself):

      void PMIx_Info_free(pmix_info_t *p, size_t n);

- Check the given info struct to determine if it includes a boolean value (includes strings for `true` and `false`, including abbreviations such as `t` or `f`), and if so, then its value. A value type of `PMIX_UNDEF` is taken to imply a boolean `true` as the presence of the key defaults to indicating `true`.

      pmix_boolean_t PMIx_Info_true(const pmix_info_t *p);

- Mark the info struct as required:

      void PMIx_Info_required(pmix_info_t *p);

- Mark the info struct as optional:

      void PMIx_Info_optional(pmix_info_t *p);

- Check if the info struct is required:

      bool PMIx_Info_is_required(const pmix_info_t *p);

- Check if the info struct is optional:

      bool PMIx_Info_is_optional(const pmix_info_t *p);

- Mark the info struct as processed:

      void PMIx_Info_processed(pmix_info_t *p);

- Check if the info struct has been processed:

      bool PMIx_Info_was_processed(const pmix_info_t *p);

- Mark the info struct as the end of an array:

      void PMIx_Info_set_end(pmix_info_t *p);

- Check if the info struct is the end of an array:

      bool PMIx_Info_is_end(const pmix_info_t *p);

- Mark the info as a qualifier:

      void PMIx_Info_qualifier(pmix_info_t *p);

- Check if the info struct is a qualifier:

      bool PMIx_Info_is_qualifier(const pmix_info_t *p);

- Mark the info struct as persistent — do not release its contents:

      void PMIx_Info_persistent(pmix_info_t *p);

- Check if the info struct is persistent:

      bool PMIx_Info_is_persistent(const pmix_info_t *p);

- Get the size of a `pmix_info_t` structure:

      pmix_status_t PMIx_Info_get_size(const pmix_info_t *val,
                                       size_t *size);

## 4.10. Coordinate Struct Functions

- Initialize a coord struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Coord_construct(pmix_coord_t *m);

- Free memory stored inside a coord struct:

      void PMIx_Coord_destruct(pmix_coord_t *m);

- Create and initialize an array of coord structs:

      pmix_coord_t* PMIx_Coord_create(size_t dims,
                                      size_t number);

- Free memory stored inside an array of coord structs (does not free the struct memory itself):

      void PMIx_Coord_free(pmix_coord_t *m, size_t number);

## 4.11. Topology Functions

- Initialize a topology struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Topology_construct(pmix_topology_t *t);

- Create and initialize an array of topology structs:

      pmix_topology_t* PMIx_Topology_create(size_t n);

- Free memory stored inside an array of topology structs (does not free the struct memory itself):

      void PMIx_Topology_free(pmix_topology_t *t, size_t n);

## 4.12. Cpuset Functions

- Initialize a cpuset struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Cpuset_construct(pmix_cpuset_t *cpuset);

- Free memory stored inside a cpuset struct:

      void PMIx_Cpuset_destruct(pmix_cpuset_t *cpuset);

- Create and initialize an array of cpuset structs:

      pmix_cpuset_t* PMIx_Cpuset_create(size_t n);

- Free memory stored inside an array of cpuset structs (does not free the struct memory itself):

      void PMIx_Cpuset_free(pmix_cpuset_t *c, size_t n);

## 4.13. Geometry Functions

- Initialize a geometry struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Geometry_construct(pmix_geometry_t *g);

- Free memory stored inside a cpuset struct:

      void PMIx_Geometry_destruct(pmix_geometry_t *g);

- Create and initialize an array of cpuset structs:

      pmix_geometry_t* PMIx_Geometry_create(size_t n);

- Free memory stored inside an array of cpuset structs (does not free the struct memory itself):

      void PMIx_Geometry_free(pmix_geometry_t *g, size_t n);

## 4.14. Device Distance Functions

- Initialize a device distance struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Device_distance_construct(pmix_device_distance_t *d);

- Free memory stored inside a device distance struct:

      void PMIx_Device_distance_destruct(pmix_device_distance_t *d);

- Create and initialize an array of device distance structs:

      pmix_device_distance_t* PMIx_Device_distance_create(size_t n);

- Free memory stored inside an array of device distance structs (does not free the struct memory itself):

      void PMIx_Device_distance_free(pmix_device_distance_t *d, size_t n);

## 4.15. Byte Object Functions

- Initialize a byte object struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Byte_object_construct(pmix_byte_object_t *b);

- Free memory stored inside a byte object struct:

      void PMIx_Byte_object_destruct(pmix_byte_object_t *g);

- Create and initialize an array of byte object structs:

      pmix_byte_object_t* PMIx_Byte_object_create(size_t n);

- Free memory stored inside an array of byte object structs (does not free the struct memory itself):

      void PMIx_Byte_object_free(pmix_byte_object_t *g, size_t n);

- Load a byte object:

      void PMIx_Byte_object_load(pmix_byte_object_t *b,
                                 char *d, size_t sz);

## 4.16. Endpoint Functions

- Initialize an endpoint struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Endpoint_construct(pmix_endpoint_t *e);

- Free memory stored inside an endpoint struct:

      void PMIx_Endpoint_destruct(pmix_endpoint_t *e);

- Create and initialize an array of endpoint structs:

      pmix_endpoint_t* PMIx_Endpoint_create(size_t n);

- Free memory stored inside an array of endpoint structs (does not free the struct memory itself):

      void PMIx_Endpoint_free(pmix_endpoint_t *e, size_t n);

## 4.17. Envar Functions

- Initialize an envar struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Envar_construct(pmix_envar_t *e);

- Free memory stored inside an envar struct:

      void PMIx_Envar_destruct(pmix_envar_t *e);

- Create and initialize an array of envar structs:

      pmix_envar_t* PMIx_Envar_create(size_t n);

- Free memory stored inside an array of envar structs (does not free the struct memory itself):

      void PMIx_Envar_free(pmix_envar_t *e, size_t n);

- Load an envar struct:

      void PMIx_Envar_load(pmix_envar_t *e,
                           char *var,
                           char *value,
                           char separator);

## 4.18. Data Buffer Functions

- Initialize a data buffer struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Data_buffer_construct(pmix_data_buffer_t *b);

- Free memory stored inside a data buffer struct:

      void PMIx_Data_buffer_destruct(pmix_data_buffer_t *b);

- Create a data buffer struct:

      pmix_data_buffer_t* PMIx_Data_buffer_create(void);

- Free memory stored inside a data buffer struct:

      void PMIx_Data_buffer_release(pmix_data_buffer_t *b);

- Load a data buffer struct:

      void PMIx_Data_buffer_load(pmix_data_buffer_t *b,
                                 char *bytes, size_t sz);

- Unload a data buffer struct:

      void PMIx_Data_buffer_unload(pmix_data_buffer_t *b,
                                   char **bytes, size_t *sz);

## 4.19. Proc Struct Functions

- Initialize a proc struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Proc_construct(pmix_proc_t *p);

- Clear memory inside a proc struct:

      void PMIx_Proc_destruct(pmix_proc_t *p);

- Create and initialize an array of proc structs:

      pmix_proc_t* PMIx_Proc_create(size_t n);

- Free memory stored inside an array of proc structs (does not free the struct memory itself):

      void PMIx_Proc_free(pmix_proc_t *p, size_t n);

- Load a proc struct:

      void PMIx_Proc_load(pmix_proc_t *p,
                          char *nspace, pmix_rank_t rank);

- Construct a multicluster `nspace` struct from cluster and `nspace` values:

      void PMIx_Multicluster_nspace_construct(pmix_nspace_t target,
                                              pmix_nspace_t cluster,
                                              pmix_nspace_t nspace);

- Parse a multicluster nspace struct to separate out the cluster and `nspace` portions:

      void PMIx_Multicluster_nspace_parse(pmix_nspace_t target,
                                          pmix_nspace_t cluster,
                                          pmix_nspace_t nspace);

## 4.20. Proc Info Functions

- Initialize a proc info struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Proc_info_construct(pmix_proc_info_t *p);

- Clear memory inside a proc info struct:

      void PMIx_Proc_info_destruct(pmix_proc_info_t *p);

- Create and initialize an array of proc info structs:

      pmix_proc_info_t* PMIx_Proc_info_create(size_t n);

- Free memory stored inside an array of proc info structs (does not free the struct memory itself):

      void PMIx_Proc_info_free(pmix_proc_info_t *p, size_t n);

## 4.21. Proc Stats Functions

- Initialize a proc stats struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Proc_stats_construct(pmix_proc_stats_t *p);

- Clear memory inside a proc stats struct:

      void PMIx_Proc_stats_destruct(pmix_proc_stats_t *p);

- Create and initialize an array of proc stats structs:

      pmix_proc_stats_t* PMIx_Proc_stats_create(size_t n);

- Free memory stored inside an array of proc stats structs (does not free the struct memory itself):

      void PMIx_Proc_stats_free(pmix_proc_stats_t *p, size_t n);

## 4.22. Disk Stats Functions

- Initialize a disk stats struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Disk_stats_construct(pmix_disk_stats_t *p);

- Clear memory inside a disk stats struct:

      void PMIx_Disk_stats_destruct(pmix_disk_stats_t *p);

- Create and initialize an array of disk stats structs:

      pmix_disk_stats_t* PMIx_Disk_stats_create(size_t n);

- Free memory stored inside an array of disk stats structs (does not free the struct memory itself):

      void PMIx_Disk_stats_free(pmix_disk_stats_t *p, size_t n);

## 4.23. Net Stats Functions

- Initialize a net stats struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Net_stats_construct(pmix_net_stats_t *p);

- Clear memory inside a net stats struct:

      void PMIx_Net_stats_destruct(pmix_net_stats_t *p);

- Create and initialize an array of net stats structs:

      pmix_net_stats_t* PMIx_Net_stats_create(size_t n);

- Free memory stored inside an array of net stats structs (does not free the struct memory itself):

      void PMIx_Net_stats_free(pmix_net_stats_t *p, size_t n);

## 4.24. Process Data Functions

- Initialize a pdata struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_Pdata_construct(pmix_pdata_t *p);

- Clear memory inside a pdata struct:

      void PMIx_Pdata_destruct(pmix_pdata_t *p);

- Create and initialize an array of pdata structs:

      pmix_pdata_t* PMIx_Pdata_create(size_t n);

- Free memory stored inside an array of pdata structs (does not free the struct memory itself):

      void PMIx_Pdata_free(pmix_pdata_t *p, size_t n);

## 4.25. App Struct Functions

- Initialize a `pmix_app_t` struct. Memory for the provided object must have previously been allocated or statically declared:

      void PMIx_App_construct(pmix_app_t *p);

- Clear memory inside an app struct:

      void PMIx_App_destruct(pmix_app_t *p);

- Create and initialize an array of app structs:

      pmix_app_t* PMIx_App_create(size_t n);

- Create and initialize an array of `pmix_info_t` structs in the provided `pmix_app_t` object:

      void PMIx_App_info_create(pmix_app_t *p, size_t n);

- Free memory stored inside an array of app structs (does not free the struct memory itself):

      void PMIx_App_free(pmix_app_t *p, size_t n);

- Free memory stored inside a `pmix_app_t` object

      void PMIx_App_release(pmix_app_t *p);

## 4.26. PMIx Info List Functions

Constructing arrays of `pmix_info_t` for passing to an API can be tedious since the `pmix_info_t` itself is not a “list object”. Since this is a very frequent operation, a set of APIs has been provided that opaquely manipulates internal PMIx list structures for this purpose. The user only need provide a `void*` pointer to act as the caddy for the internal list object. The base functions for these operations are in the Standard, but the following functions have been added here:

- Retrieve the next `pmix_info_t` from the provided list, given the current pointer. Passing a `NULL` to the `prev` parameter will return the first object on the list. A `NULL` is returned upon reaching the end of the list:

  >     pmix_info_t* PMIx_Info_list_get_info(void *ptr, void *prev, void **next);

- Insert a pmix_info_t` struct into the provided list. This directly copies the contents of the provided `pmix_info_t` struct, preserving any included pointers. The object on the list is subsequently marked as `persistent` to avoid free’ing any objects pointed to in the struct:

  >     pmix_status_t PMIx_Info_list_insert(void *ptr, pmix_info_t *info);

- Prepend a value onto the provided list:

  >     pmix_status_t PMIx_Info_list_prepend(void *ptr,
  >                                          const char *key,
  >                                          const void *value,
  >                                          pmix_data_type_t type);

## 4.27. Pretty-Print Functions

The following pretty-print support APIs have been added:

- Print a `pmix_value_cmp_t` value

      const char* PMIx_Value_comparison_string(pmix_value_cmp_t cmp);

- Print the contents of a `pmix_app_t` struct. Note that the returned string must be free’d by the caller:

  >     char* PMIx_App_string(const pmix_app_t *app);

The following pretty-print support APIs have been slightly modified to add a `const` qualifier to their input parameter:

>     const char* PMIx_Get_attribute_string(const char *attribute);
>     const char* PMIx_Get_attribute_name(const char *attrstring);
>     char* PMIx_Info_string(const pmix_info_t *info);
>     char* PMIx_Value_string(const pmix_value_t *value);

> This is not expected to cause any issues for users.

The following function has been added to return the `pmix_status_t` corresponding to the string name of the constant:

>     pmix_status_t PMIx_Error_code(const char *errname);

## 4.28. Constants

- `PMIX_DATA_BUFFER`: data type for packing/unpacking of `pmix_data_buffer_t` objects

- `PMIX_DISK_STATS`: data type for packing/unpacking of `pmix_disk_stats_t` objects

- `PMIX_NET_STATS`: data type for packing/unpacking of `pmix_net_stats_t` objects

- `PMIX_NODE_STATS`: data type for packing/unpacking of `pmix_node_stats_t` objects

- `PMIX_PROC_STATS`: data type for packing/unpacking of `pmix_proc_stats_t` objects

- `PMIX_ERR_JOB_EXE_NOT_FOUND`: specified executable not found

- `PMIX_ERR_JOB_INSUFFICIENT_RESOURCES`: insufficient resources to spawn job

- `PMIX_ERR_JOB_SYS_OP_FAILED`: system library operation failed

- `PMIX_ERR_JOB_WDIR_NOT_FOUND`: specified working directory not found

- `PMIX_READY_FOR_DEBUG`: event indicating job/proc is ready for debug (accompanied by `PMIX_BREAKPOINT` indicating where proc is waiting)

- `PMIX_ERR_PROC_REQUESTED_ABORT`: process called `PMIx_Abort`

- `PMIX_ERR_PROC_KILLED_BY_CMD`: process was terminated by RTE command

- `PMIX_ERR_PROC_FAILED_TO_START`: process failed to start

- `PMIX_ERR_PROC_ABORTED_BY_SIG`: process aborted by signal (e.g., segmentation fault)

- `PMIX_ERR_PROC_SENSOR_BOUND_EXCEEDED`: process terminated due to exceeding a sensor boundary

- `PMIX_ERR_EXIT_NONZERO_TERM`: process exited normally, but with a non-zero status

- `PMIX_INFO_QUALIFIER` (value: 0x00000008): Info is a qualifier to the primary value

- `PMIX_INFO_PERSISTENT` (value: 0x00000010): Do not release included value

Note

OpenPMIx version v5.0.3rc1 renamed the `PMIX_DEBUG_WAIT_FOR_NOTIFY` to `PMIX_READY_FOR_DEBUG`. The prior name is retained as deprecated for backward compatibility.

## 4.29. Attributes

Attribute | Type | Description
--- | --- | ---
`PMIX_EXTERNAL_AUX_EVENT_BASE` `"pmix.evaux"` | `(void*)` | event base to be used for auxiliary functions (e.g., capturing signals) that would otherwise interfere with the host
`PMIX_CONNECT_TO_SCHEDULER` `"pmix.cnct.sched"` | `(bool)` | Connect to the system scheduler
`PMIX_BIND_PROGRESS_THREAD` `"pmix.bind.pt"` | `(char*)` | Comma-delimited ranges of CPUs that the internal PMIx progress thread shall be bound to
`PMIX_BIND_REQUIRED` `"pmix.bind.reqd"` | `(bool)` | Return error if the internal PMIx progress thread cannot be bound
`PMIX_COLOCATE_PROCS` `"pmix.colproc"` | `(pmix_data_array_t*)` | Array of `pmix_proc_t` identifying the procs with which the new job’s procs are to be colocated
`PMIX_COLOCATE_NPERPROC` `"pmix.colnum.proc"` | `(uint16_t)` | Number of procs to colocate with each identified proc
`PMIX_COLOCATE_NPERNODE` `"pmix.colnum.node"` | `(uint16_t)` | Number of procs to colocate on the node of each identified proc
`PMIX_EVENT_ONESHOT` `pmix.evone` | `(bool)` | when registering, indicate that this event handler is to be deleted after being invoked
`PMIX_GROUP_ADD_MEMBERS` `pmix.grp.add` | `(pmix_data_array_t*)` | Array of `pmix_proc_t` identifying procs that are not included in the membership specified in the procs array passed to the `PMIx_Group_construct[_nb]()` call, but are to be included in the final group. The identified procs will be sent an invitation to join the group during the construction procedure. This is used when some members of the proposed group do not know the full membership and therefore cannot include all members in the call to construct.
`PMIX_GROUP_LOCAL_CID` `pmix.grp.lclid` | `(size_t)` | Local context ID for the specified process member of a group
`PMIX_GROUP_INFO` `pmix.grp.info` | `pmix_data_array_t` | Array of pmix_info_t containing data that is to be shared across all members of a group during group construction
`PMIX_IOF_TAG_DETAILED_OUTPUT` `pmix.iof.tagdet` | `(bool)` | Tag output with the [local jobid,rank][hostname:pid] and channel it comes from
`PMIX_IOF_TAG_FULLNAME_OUTPUT` `pmix.iof.tagfull` | `(bool)` | Tag output with the [nspace,rank] and channel it comes from
`PMIX_LOG_AGG` `pmix.log.agg` | `(bool)` | Whether to aggregate and prevent duplicate logging messages based on key value pairs.
`PMIX_LOG_KEY` `pmix.log.key` | `(char*)` | key to a logging message
`PMIX_LOG_VAL` `pmix.log.val` | `(char*)` | value to a logging message
`PMIX_MYSERVER_URI` `pmix.mysrvr.uri` | `(char*)` | URI of this proc’s listener socket
`PMIX_QUALIFIED_VALUE` `pmix.qual.val` | `(pmix_data_array_t*)` | Value being provided consists of the primary key-value pair in first position, followed by one or more key-value qualifiers to be used when subsequently retrieving the primary value
`PMIX_WDIR_USER_SPECIFIED` `pmix.wdir.user` | `(bool)` | User specified the working directory
`PMIX_RUNTIME_OPTIONS` `pmix.runopt` | `(char*)` | Environment-specific runtime directives that control job behavior
`PMIX_ABORT_NON_ZERO_TERM` `pmix.abnz` | `(bool)` | Abort the spawned job if any process terminates with non-zero status
`PMIX_DO_NOT_LAUNCH` `pmix.dnl` | `(bool)` | Execute all procedures to prepare the requested job for launch, but do not launch it. Typically combined with the PMIX_DISPLAY_MAP or PMIX_DISPLAY_MAP_DETAILED for debugging purposes.
`PMIX_SHOW_LAUNCH_PROGRESS` `pmix.showprog` | `(bool)` | Provide periodic progress reports on job launch procedure (e.g., after every 100 processes have been spawned)
`PMIX_AGGREGATE_HELP` `pmix.agg.help` | `(bool)` | Aggregate help messages, reporting each unique help message once accompanied by the number of processes that reported it
`PMIX_REPORT_CHILD_SEP` `pmix.rptchildsep` | `(bool)` | Report the exit status of any child jobs spawned by the primary job separately. If false, then the final exit status reported will be zero if the primary job and all spawned jobs exit normally, or the first non-zero status returned by either primary or child jobs.
`PMIX_DISPLAY_MAP_DETAILED` `pmix.dispmapdet` | `(bool)` | display a highly detailed placement map upon spawn
`PMIX_DISPLAY_ALLOCATION` `pmix.dispalloc` | `(bool)` | display the resource allocation
`PMIX_DISPLAY_TOPOLOGY` `pmix.disptopo` | `(char*)` | comma-delimited list of hosts whose topology is to be displayed
`PMIX_DISPLAY_PROCESSORS` `pmix.dispcpus` | `(char*)` | comma-delimited list of hosts whose available CPUs are to be displayed
`PMIX_DISPLAY_PARSEABLE_OUTPUT` `pmix.dispparse` | `(bool)` | display requested info in a format more amenable to machine parsing
`PMIX_SORTED_PROC_ARRAY` `pmix.sorted.parr` | `(bool)` | Proc array being passed has been sorted
`PMIX_QUERY_PROVISIONAL_ABI_VERSION` `pmix.qry.prabiver` | `(char*)` | The PMIx Standard Provisional ABI version(s) supported, returned in the form of a comma separated list of “MAJOR.MINOR” pairs
`PMIX_QUERY_STABLE_ABI_VERSION` `pmix.qry.stabiver` | `(char*)` | The PMIx Standard Stable ABI version(s) supported, returned in the form of a comma separated list of “MAJOR.MINOR” pairs

Note

The attribute `PMIX_DEBUG_STOP_IN_APP` has been modified to only support a `PMIX_BOOL` value instead of an optional array of ranks due to questions over the use-case calling for stopping a subset of a job’s processes while allowing others to run “free”.

## 4.30. Datatypes

- `pmix_value_cmp_t`: an enum indicating the relative value of two `pmix_value_t objects`. Values include:

  - `PMIX_EQUAL`

  - `PMIX_VALUE1_GREATER`

  - `PMIX_VALUE2_GREATER`

  - `PMIX_VALUE_TYPE_DIFFERENT`

  - `PMIX_VALUE_INCOMPATIBLE_OBJECTS`

  - `PMIX_VALUE_COMPARISON_NOT_AVAIL`

- `pmix_boolean_t`: an enum indicating boolean state of a `pmix_value_t` (possibly contained in a `pmix_info_t` object):

  - `PMIX_BOOL_TRUE`

  - `PMIX_BOOL_FALSE`

  - `PMIX_NON_BOOL`

- `pmix_disk_stats_t`: contains statistics on disk read/write operations

- `pmix_net_stats_t`: contains statistics on network activity

- `pmix_node_stats_t`: contains statistics on node resource usage

- `pmix_proc_stats_t`: contains statistics on process resource usage

### 4.30.1. Datatype static initializers

Static initializers were added for each complex data type (i.e., a data type defined as a struct). Most are contained in the Standard, but the following extensions have been provided:

- `PMIX_PROC_STATS_STATIC_INIT`

- `PMIX_DISK_STATS_STATIC_INIT`

- `PMIX_NET_STATS_STATIC_INIT`

- `PMIX_NODE_STATS_STATIC_INIT`

## 4.31. Macros

Although the convenience macros have been deprecated, several were added (in deprecated form) that previously were missing. These are added for symmetry to support those who continue to use the macros, and include:

- `PMIX_XFER_PROCID`: transfer a `pmix_proc_t` to another one (non-destructive copy)

- `PMIX_INFO_SET_END`: mark this `pmix_info_t` as being at the end of an array

- `PMIX_INFO_SET_PERSISTENT`: mark that the data in this `pmix_info_t` is not to be released by `PMIX_Info_destruct()` (or its macro form)

- `PMIX_INFO_SET_QUALIFIER`: mark this `pmix_info_t` as a qualifier to the primary key

- `PMIX_INFO_IS_PERSISTENT`: test if this `pmix_info_t` has been marked as persistent

- `PMIX_INFO_IS_QUALIFIER`: test if this `pmix_info_t` has been marked as a qualifier

- `PMIX_DATA_ARRAY_INIT`: initialize a `pmix_data_array_t`

- `PMIX_CHECK_TRUE`: check if a `pmix_value_t` is boolean `true` (supports string as well as traditional boolean values)

- `PMIX_CHECK_BOOL`: check if a `pmix_value_t` is a boolean value (supports string as well as traditional boolean values)

Macros supporting `pmix_disk_stats_t` objects:

- `PMIX_DISK_STATS_CONSTRUCT`

- `PMIX_DISK_STATS_CREATE`

- `PMIX_DISK_STATS_DESTRUCT`

- `PMIX_DISK_STATS_FREE`

- `PMIX_DISK_STATS_RELEASE`

Macros supporting `pmix_net_stats_t` objects:

- `PMIX_NET_STATS_CONSTRUCT`

- `PMIX_NET_STATS_CREATE`

- `PMIX_NET_STATS_DESTRUCT`

- `PMIX_NET_STATS_FREE`

- `PMIX_NET_STATS_RELEASE`

Macros supporting `pmix_node_stats_t` objects:

- `PMIX_NODE_STATS_CONSTRUCT`

- `PMIX_NODE_STATS_CREATE`

- `PMIX_NODE_STATS_DESTRUCT`

- `PMIX_NODE_STATS_RELEASE`

Macros supporting `pmix_proc_stats_t` objects:

- `PMIX_PROC_STATS_CONSTRUCT`

- `PMIX_PROC_STATS_CREATE`

- `PMIX_PROC_STATS_DESTRUCT`

- `PMIX_PROC_STATS_FREE`

- `PMIX_PROC_STATS_RELEASE`

## 4.32. Scheduler Integration

OpenPMIx has taken some initial steps towards supporting the integration of schedulers to runtime environments (RTEs) using PMIx as the middleware. Supporting definitions will continue to be added going forward. This section describes the current state of those definitions.

### 4.32.1. Session Control Function

Used by the scheduler to request a session control action by the RTE - e.g., setup a session (allocate the specified nodes to the new session, provision the nodes with the specified image, setup a user-level DVM across those nodes, and startup the given application under control of that DVM). In addition to setting up a new session, the function can be called to direct that a currently executing session be preempted or terminated. The sessionID identifies the session to which the specified control action is to be applied. A `UINT32_MAX` value can be used to indicate all sessions under the caller’s control.

Also used by the RTE to report a change in session state - e.g., that the session has completed The directives are provided as `pmix_info_t` structs in the directives array. The callback function provides a status to indicate whether or not the request was granted, and to provide some information as to the reason for any denial in the `pmix_info_cbfunc_t` array of `pmix_info_t` structures. If non-`NULL`, then the specified release_fn must be called when the callback function completes — this will be used to release any provided `pmix_info_t` array.

Passing `NULL` as the `cbfunc` to this call indicates that it shall be treated as a blocking operation, with the return status indicative of the overall operation’s completion.

>     pmix_status_t PMIx_Session_control(uint32_t sessionID,
>                                        const pmix_info_t directives[], size_t ndirs,
>                                        pmix_info_cbfunc_t cbfunc, void *cbdata);

### 4.32.2. Session Control Attributes

Schedulers calling to create a session are required to provide:

- the effective userID and groupID that the session should have when instantiated.

- description of the resources that are to be included in the session

- if applicable, the image that should be provisioned on nodes included in the session

- an array of applications (if any) that are to be started in the session once instantiated

Attributes supported by this API when called by the scheduler include:

Attribute | Type | Description
--- | --- | ---
`PMIX_SESSION_APP` `pmix.ssn.app` | `(pmix_data_array_t*)` | Array of `pmix_app_t` to be executed in the assigned session upon session instantiation
`PMIX_SESSION_PROVISION` `pmix.ssn.pvn` | `(pmix_data_array_t*)` | description of nodes to be provisioned with specified image
`PMIX_SESSION_PROVISION_NODES` `pmix.ssn.pvnnds` | `(char*)` | regex identifying nodes that are to be provisioned
`PMIX_SESSION_PROVISION_IMAGE` `pmix.ssn.pvnimg` | `(char*)` | name of the image that is to be provisioned
`PMIX_SESSION_PAUSE` `pmix.ssn.pause` | `(bool)` | pause all jobs in the specified session
`PMIX_SESSION_RESUME` `pmix.ssn.resume` | `(bool)` | “un-pause” all jobs in the specified session
`PMIX_SESSION_TERMINATE` `pmix.ssn.terminate` | `(bool)` | terminate all jobs in the specified session and recover all resources included in the session.
`PMIX_SESSION_PREEMPT` `pmix.ssn.preempt` | `(bool)` | preempt indicated jobs (given in accompanying `pmix_info_t` via the `PMIX_NSPACE` attribute) in the specified session and recover all their resources. If no `PMIX_NSPACE` is specified, then preempt all jobs in the session.
`PMIX_SESSION_RESTORE` `pmix.ssn.restore` | `(bool)` | restore indicated jobs (given in accompanying `pmix_info_t` via the `PMIX_NSPACE` attribute) in the specified session, including all their resources. If no `PMIX_NSPACE` is specified, then restore all jobs in the session.
`PMIX_SESSION_SIGNAL` `pmix.ssn.sig` | `(int)` | send given signal to all processes of every job in the session

Attributes supported by this API when called by the RTE include:

Attribute | Type | Description
--- | --- | ---
`PMIX_SESSION_COMPLETE` `pmix.ssn.complete` | `(bool)` | specified session has completed, all resources have been recovered and are available for scheduling. Must include `pmix_info_t` indicating ID and returned status of any jobs executing in the session.

### 4.32.3. Server module function pointers

The PMIx server module was extended to include the following interface that is used by the PMIx server to pass control requests received by the RTE from the scheduler. These include requests to establish a newly allocated session, preempt jobs, etc. A `UINT32_MAX` value for the sessionID indicates that the specified action shall be applied to all currently existing sessions.

- Provide a session control operation request

      typedef pmix_status_t (*pmix_server_session_control_fn_t)(
                                    const pmix_proc_t *requestor,
                                    uint32_t sessionID,
                                    const pmix_info_t directives[], size_t ndirs,
                                    pmix_info_cbfunc_t cbfunc, void *cbdata);

### 4.32.4. Server module attributes

A number of allocation related attributes have already been defined in the Standard. These can be used to describe the request (e.g., the resources to be included in the session). The following attribute has been added to that list:

Attribute | Type | Description
--- | --- | ---
`PMIX_SESSION_CTRL_ID` `pmix.ssnctrl.id` | `(char*)` | provide a string identifier for this request. This identifier shall be included in all subsequent interactions relating to the request.

### 4.32.5. Scheduler query attributes

The scheduler typically discovers its available resources by querying the RTE for a list of them. The following attributes augment those already in the Standard to support the query:

Attribute | Type | Description
--- | --- | ---
`PMIX_QUERY_ALLOCATION` `pmix.query.allc` | `(pmix_data_array_t*)` | returns an array of `pmix_info_t` describing the nodes known to the server. Each array element will consist of the `PMIX_NODE_INFO` key containing a `pmix_data_array_t` of `pmix_info_t`. The first element of the array must be the hostname of that node, with additional info on the node in subsequent entries. SUPPORTED_QUALIFIER: a `PMIX_ALLOC_ID` qualifier indicating the specific allocation of interest
`PMIX_TOPOLOGY_INDEX` `pmix.topo.index` | `(int)` | index of a topology in a storage array. Used when returning an allocation to avoid duplicate topology information - the RTE can return an array of topologies and then indicate the index to the topology as part of each node entry.

### 4.32.6. Allocation attributes

A number of allocation related attributes have already been defined in the Standard. These can be used to describe the request (e.g., the number and type of resources being requested). The following attribute has been added to that list:

Attribute | Type | Description
--- | --- | ---
`PMIX_ALLOC_PREEMPTIBLE` `pmix.alloc.preempt` | `(bool)` | by default, all jobs in the resulting allocation are to be considered preemptible (can be overridden at per-job level)

### 4.32.7. Allocation directive values

The `PMIx_Allocation_request` API includes a `directive` parameter to specify the operation being requested. These values were extended to include:

- `PMIX_ALLOC_REQ_CANCEL` (value: 5): Cancel the indicated allocation request