# Slurm MPI Plugin API

## Overview

This document describes Slurm MPI selection plugins and the API that defines them. It is intended as a resource to programmers wishing to write their own SLURM node selection plugins. This is version 0 of the API.

Slurm mpi selection plugins are Slurm plugins that implement the which version of mpi is used during execution of the new Slurm job. API described herein. They are intended to provide a mechanism for both selecting mpi versions for pending jobs and performing any mpi-specific tasks for job launch or termination. The plugins must conform to the Slurm Plugin API with the following specifications:

const char plugin_type[]
The major type must be "mpi." The minor type can be any recognizable abbreviation for the type of node selection algorithm. We recommend, for example:

- lam—For use with LAM MPI.
- mpich-gm—For use with MPICH-GM.
- mpich-mx—For use with MPICH-MX.
- mpich1_p4—For use with MPICH1 (P4 communications).
- mpich1_shmem—For use with MPICH1 (Shmem communications).
- mvapich—For use with MVAPICH.
- openmpi—For use with OpenMPI.
- pmi2—For use with MPI2 and MVAPICH2.
- none—For use with most other versions of MPI.

The plugin_name and plugin_version symbols required by the Slurm Plugin API require no specialization for node selection support. Note carefully, however, the versioning discussion below.

A simplified flow of logic follows:
srun is able to specify the correct mpi to use. with --mpi=MPITYPE
srun calls
mpi_p_thr_create((srun_job_t *)job);
which will set up the correct environment for the specified mpi.
slurmd daemon runs
mpi_p_init((stepd_step_rec_t *)job, (int)rank);
which will set configure the slurmd to use the correct mpi as well to interact with the srun.
slurmstepd process runs
p_mpi_hook_slurmstepd_prefork(const stepd_step_rec_t *job, char ***env);
which executes immediately before fork/exec of tasks.

## Data Objects

These functions are expected to read and/or modify data structures directly in the slurmd daemon's and srun memory. Slurmd is a multi-threaded program with independent read and write locks on each data structure type. Therefore the type of operations permitted on various data structures is identified for each function.

## API Functions

The following functions must appear. Functions which are not implemented should be stubbed.

int init (void)

Description:
Called when the plugin is loaded, before any other functions are called. Put global initialization here.

Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.

void fini (void)

Description:
Called when the plugin is removed. Clear any allocated storage here.

Returns: None.

Note: These init and fini functions are not the same as those described in the dlopen (3) system library. The C run-time system co-opts those symbols for its own initialization. The system _init() is called before the SLURM init(), and the SLURM fini() is called before the system's _fini().

int mpi_p_init (stepd_step_rec_t *job, int rank);

Description: Used by slurmd to configure the slurmd's environment to that of the correct mpi.

Arguments:
job (input) Pointer to the slurmd_job that is running. Cannot be NULL.
rank (input) Primarily there for MVAPICH. Used to send the rank fo the mpirun job. This can be 0 if no rank information is needed for the mpi type.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR.

int mpi_p_thr_create (srun_job_t *job);

Description: Used by srun to spawn the thread for the mpi processes. Most all the real processing happens here.

Arguments: job (input) Pointer to the srun_job that is running. Cannot be NULL.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return -1.

int mpi_p_single_task ();

Description: Tells the system whether or not multiple tasks can run at the same time

Arguments: none

Returns: false if multiple tasks can run and true if only a single task can run at one time.

int p_mpi_hook_slurmstepd_prefork(const stepd_step_rec_t *job, char ***env);

Description: Used by slurmstepd process immediately prior to fork and exec of user tasks.

Arguments:
job (input) Pointer to the slurmd structure for the job that is running.
env (input) Environment variables for tasks to be spawned.

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return -1.

int mpi_p_exit();

Description: Cleans up anything that needs cleaning up after execution.

Arguments: none

Returns: SLURM_SUCCESS if successful. On failure, the plugin should return SLURM_ERROR, causing slurmctld to exit.

## Versioning

This document describes version 0 of the Slurm node selection API. Future releases of Slurm may revise this API. A node selection plugin conveys its ability to implement a particular API version using the mechanism outlined for Slurm plugins. In addition, the credential is transmitted along with the version number of the plugin that transmitted it. It is at the discretion of the plugin author whether to maintain data format compatibility across different versions of the plugin.

Last modified 8 May 2014