# Quick Start Administrator Guide

## Overview

Please see the Quick Start User Guide for a general overview.

Also see Platforms for a list of supported computer platforms.

## Super Quick Start

1. Make sure the clocks, users and groups (UIDs and GIDs) are synchronized across the cluster.
2. Install MUNGE for authentication. Make sure that all nodes in your cluster have the same munge.key. Make sure the MUNGE daemon, munged is started before you start the Slurm daemons.
3. bunzip2 the distributed tar-ball and untar the files:
   tar --bzip -x -f slurm*tar.bz2
4. cd to the directory containing the Slurm source and type ./configure with appropriate options, typically --prefix= and --sysconfdir=
5. Type make to compile Slurm.
6. Type make install to install the programs, documentation, libraries, header files, etc.
7. Build a configuration file using your favorite web browser and doc/html/configurator.html.
   NOTE: The SlurmUser must exist prior to starting Slurm and must exist on all nodes of the cluster.
   NOTE: The parent directories for Slurm's log files, process ID files, state save directories, etc. are not created by Slurm. They must be created and made writable by SlurmUser as needed prior to starting Slurm daemons.
   NOTE: If any parent directories are created during the installation process (for the executable files, libraries, etc.), those directories will have access rights equal to read/write/execute for everyone minus the umask value (e.g. umask=0022 generates directories with permissions of "drwxr-r-x" and mask=0000 generates directories with permissions of "drwxrwrwx" which is a security problem).
8. Install the configuration file in <sysconfdir>/slurm.conf.
   NOTE: You will need to install this configuration file on all nodes of the cluster.
9. Start the slurmctld and slurmd daemons.

NOTE: Items 3 through 6 can be replaced with

1. rpmbuild -ta slurm*.tar.bz2
2. rpm --install <the rpm files>

FreeBSD administrators should see the FreeBSD section below.

## Building and Installing Slurm

Instructions to build and install Slurm manually are shown below. See the README and INSTALL files in the source distribution for more details.

1. bunzip2 the distributed tar-ball and untar the files: tar --bzip -x -f slurm*tar.bz2
2. cd to the directory containing the Slurm source and type ./configure with appropriate options (see below).
3. Type make to compile Slurm.
4. Type make install to install the programs, documentation, libraries, header files, etc.

A full list of configure options will be returned by the command configure --help. The most commonly used arguments to the configure command include:

--enable-debug
Enable additional debugging logic within Slurm.

--prefix=PREFIX
Install architecture-independent files in PREFIX; default value is /usr/local.

--sysconfdir=DIR
Specify location of Slurm configuration file. The default value is PREFIX/etc

If required libraries or header files are in non-standard locations, set CFLAGS and LDFLAGS environment variables accordingly. Optional Slurm plugins will be built automatically when the configure script detects that the required build requirements are present. Build dependencies for various plugins and commands are denoted below:

- Authd The auth/authd plugin will be built and installed if the libauth development library and its dependency libs are installed.
- BLCR The checkpoint/blcr plugin will be built if the blcr development library is present.
- cgroup Task Affinity The task/cgroup plugin will be built with task affinity support if the hwloc development library is present.
- HDF5 Job Profiling The acct_gather_profile/hdf5 job profiling plugin will be built if the hdf5 development library is present.
- HTML Man Pages HTML versions of the man pages will be generated if the man2html command is present.
- IPMI Engergy Consumption The acct_gather_energy/ipmi accouting plugin will be built if the freeimpi development library is present.
- InfiniBand Accounting The acct_gather_infiniband/ofed InfiniBand accounting plugin will be built if the libibmad and libibumad development libraries are present.
- Lua Support The lua API will be available in various plugins if the lua development library is present.
- MUNGE The auth/munge plugin will be built if the MUNGE authentication development library is installed. MUNGE is used as the default authentication mechanism.
- MySQL MySQL support for accounting will be built if the mysql development library is present.
- OpenSSL The crypto/openssl CryptoType plugin will be built if the openssl development library is present.
- PAM Support PAM support will be added if the PAM development library is installed.
- NUMA Affinity NUMA support in the task/affinity plugin will be available if the numa development library is installed.
- Readline Support Readline support in scontrol and sacctmgr's interactive modes will be available if the readline development library is present.
- RRD External Sensor Data Collection The ext_sensors/rrd plugin will be built if the rrdtool development library is present.
- smap The smap command will be built only if the ncurses development library is installed.
- sview The sview command will be built only if gtk+-2.0 is installed.

Please see the Download page for references to required software to build these plugins.

To build RPMs directly, copy the distributed tar-ball into a directory and execute a command of this sort (substitute the appropriate Slurm version number):
rpmbuild -ta slurm-14.03.9.tar.bz2

The rpm file will be installed under the $(HOME)/rpmbuild directory of the user building them.

You can control some aspects of the RPM built with a .rpmmacros file in your home directory. Special macro definitions will likely only be required if files are installed in unconventional locations. Some macro definitions that may be used in building Slurm include:

_enable_debug
  Specify if debugging logic within Slurm is to be enabled
_prefix
  Pathname of directory to contain the Slurm files
_sysconfdir (or _slurm_sysconfdir)
  Pathname of directory containing the slurm.conf configuration file
with_munge
  Specifies the MUNGE (authentication library) installation location
with_proctrack
  Specifies AIX process tracking kernel extension header file location
with_ssl
  Specifies SSL library installation location

To build Slurm on our AIX system, the following .rpmmacros file is used:

    # .rpmmacros
    # For AIX at LLNL
    # Override some RPM macros from /usr/lib/rpm/macros
    # Set Slurm-specific macros for unconventional file locations
    #
    %_enable_debug     "--with-debug"
    %_prefix           /admin/llnl
    %_sysconfdir       %{_prefix}/etc/slurm
    %_defaultdocdir    %{_prefix}/doc
    %with_munge        "--with-munge=/opt/freeware"
    %with_proctrack    "--with-proctrack=/admin/llnl/include"
    %with_ssl          "--with-ssl=/opt/freeware"

### RPMs Installed

The RPMs needed on the head node, compute nodes, and slurmdbd node can vary by configuration, but here is a suggested starting point:

- Head Node (where the slurmctld daemon runs),
  Compute and Login Nodes

- slurm
- slurm-devel
- slurm-munge
- slurm-perlapi
- slurm-plugins
- slurm-sjobexit
- slurm-sjstat
- slurm-torque

- Slurmdbd Node

- slurm
- slurm-devel
- slurm-munge
- slurm-plugins
- slurm-slurmdbd
- slurm-sql

## Daemons

slurmctld is sometimes called the "controller" daemon. It orchestrates Slurm activities, including queuing of jobs, monitoring node states, and allocating resources to jobs. There is an optional backup controller that automatically assumes control in the event the primary controller fails (see the High Availability section below). The primary controller resumes control whenever it is restored to service. The controller saves its state to disk whenever there is a change in state (see "StateSaveLocation" in Configuration section below). This state can be recovered by the controller at startup time. State changes are saved so that jobs and other state information can be preserved when the controller moves (to or from a backup controller) or is restarted.

We recommend that you create a Unix user slurm for use by slurmctld. This user name will also be specified using the SlurmUser in the slurm.conf configuration file. This user must exist on all nodes of the cluster for authentication of communications. Note that files and directories used by slurmctld will need to be readable or writable by the user SlurmUser (the Slurm configuration files must be readable; the log file directory and state save directory must be writable).

The slurmd daemon executes on every compute node. It resembles a remote shell daemon to export control to Slurm. Because slurmd initiates and manages user jobs, it must execute as the user root.

If you want to archive job accounting records to a database, the slurmdbd (Slurm DataBase Daemon) should be used. We recommend that you defer adding accounting support until after basic Slurm functionality is established on your system. An Accounting web page contains more information.

slurmctld and/or slurmd should be initiated at node startup time per the Slurm configuration. A file etc/init.d/slurm is provided for this purpose. This script accepts commands start, startclean (ignores all saved state), restart, and stop.

### High Availability

A backup controller can be configured (see "BackupController" in the Configuration section below) to take over for the primary slurmctld if it ever fails. The backup controller should be hosted on a node different from the node hosting the primary slurmctld. However, both hosts should mount a common file system containing the state information (see "StateSaveLocation" in the Configuration section below).

The backup controller detects when the primary fails and takes over for it. When the primary returns to service, it notifies the backup. The backup then saves state and returns to backup mode. The primary reads the saved state and resumes normal operation. Other than a brief period of non-responsiveness, the transition back and forth should go undetected.

Any time the slurmctld daemon or hardware fails before state information reaches disk can result in lost state. Slurmctld writes state frequently (every five seconds by default), but with large numbers of jobs, the formatting and writing of records can take seconds and recent changes might not be written to disk. Another example is if the state information is written to file, but that information is cached in memory rather than written to disk when the node fails. The interval between state saves being written to disk can be configured at build time by defining SAVE_MAX_WAIT to a different value than five.

## Infrastructure

### User and Group Identification

There must be a uniform user and group name space (including UIDs and GIDs) across the cluster. It is not necessary to permit user logins to the control hosts (ControlMachine or BackupController), but the users and groups must be configured on those hosts.

### Authentication of Slurm communications

All communications between Slurm components are authenticated. The authentication infrastructure is provided by a dynamically loaded plugin chosen at runtime via the AuthType keyword in the Slurm configuration file. Currently available authentication types include authd, munge, and none. The default authentication infrastructure is "munge", but this does require the installation of the MUNGE package. An authentication type of "none" requires no infrastructure, but permits any user to execute any job as another user with limited programming effort. This may be fine for testing purposes, but certainly not for production use. Configure some AuthType value other than "none" if you want any security. We recommend the use of MUNGE unless you are experienced with authd. If using MUNGE, all nodes in the cluster must be configured with the same munge.key file. The MUNGE daemon, munged, must also be started before Slurm daemons.

While Slurm itself does not rely upon synchronized clocks on all nodes of a cluster for proper operation, its underlying authentication mechanism does have this requirement.

### MPI support

Slurm supports many different MPI implementations. For more information, see MPI.

### Scheduler support

Slurm can be configured with rather simple or quite sophisticated scheduling algorithms depending upon your needs and willingness to manage the configuration (much of which requires a database). The first configuration parameter of interest is PriorityType with two options available: basic (first-in-first-out) and multifactor. The multifactor plugin will assign a priority to jobs based upon a multitude of configuration parameters (age, size, fair-share allocation, etc.) and its details are beyond the scope of this document. See the Multifactor Job Priority Plugin document for details.

The SchedType configuration parameter controls how queued jobs are scheduled and several options are available.

- builtin will initiate jobs strictly in their priority order, typically (first-in-first-out)
- backfill will initiate a lower-priority job if doing so does not delay the expected initiation time of higher priority jobs; essentially using smaller jobs to fill holes in the resource allocation plan. Effective backfill scheduling does require users to specify job time limits.
- gang time-slices jobs in the same partition/queue and can be used to preempt jobs from lower-priority queues in order to execute jobs in higher priority queues.
- wiki is an interface for use with The Maui Scheduler
- wiki2 is an interface for use with the Moab Cluster Suite

For more information about scheduling options see Gang Scheduling, Preemption, Resource Reservation Guide, Resource Limits and Sharing Consumable Resources.

### Resource selection

The resource selection mechanism used by Slurm is controlled by the SelectType configuration parameter. If you want to execute multiple jobs per node, but apportion the processors, memory and other resources, the cons_res (consumable resources) plugin is recommended. If you tend to dedicate entire nodes to jobs, the linear plugin is recommended. For more information, please see Consumable Resources in Slurm. For BlueGene systems, the bluegene plugin is required (it is topology aware and interacts with the BlueGene bridge API).

### Logging

Slurm uses the syslog function to record events. It uses a range of importance levels for these messages. Be certain that your system's syslog functionality is operational.

### Accounting

Slurm supports accounting records being written to a simple text file, directly to a database (MySQL or MariaDB), or to a daemon securely managing accounting data for multiple clusters. For more information see Accounting.

### Corefile format

Slurm is designed to support generating a variety of core file formats for application codes that fail (see the --core option of the srun command). As of now, Slurm only supports a locally developed lightweight corefile library which has not yet been released to the public. It is expected that this library will be available in the near future.

### Parallel debugger support

Slurm exports information for parallel debuggers using the specification detailed here. This is meant to be exploited by any parallel debugger (notably, TotalView), and support is unconditionally compiled into Slurm code.

The following lines should also be added to the global .tvdrc file for TotalView to operate with Slurm:

    dset TV::parallel_configs {
    	name: Slurm;
    	description: Slurm;
    	starter: srun %s %p %a;
    	style: manager_process;
    	tasks_option: -n;
    	nodes_option: -N;
    	env: ;
    	force_env: false;
    }

### Compute node access

Slurm does not by itself limit access to allocated compute nodes, but it does provide mechanisms to accomplish this. There is a Pluggable Authentication Module (PAM) for restricting access to compute nodes available for download. When installed, the Slurm PAM module will prevent users from logging into any node that has not be assigned to that user. On job termination, any processes initiated by the user outside of Slurm's control may be killed using an Epilog script configured in slurm.conf. An example of such a script is included as etc/slurm.epilog.clean. Without these mechanisms any user can login to any compute node, even those allocated to other users.

## Configuration

The Slurm configuration file includes a wide variety of parameters. This configuration file must be available on each node of the cluster and must have consistent contents. A full description of the parameters is included in the slurm.conf man page. Rather than duplicate that information, a minimal sample configuration file is shown below. Your slurm.conf file should define at least the configuration parameters defined in this sample and likely additional ones. Any text following a "#" is considered a comment. The keywords in the file are not case sensitive, although the argument typically is (e.g., "SlurmUser=slurm" might be specified as "slurmuser=slurm"). The control machine, like all other machine specifications, can include both the host name and the name used for communications. In this case, the host's name is "mcri" and the name "emcri" is used for communications. In this case "emcri" is the private management network interface for the host "mcri". Port numbers to be used for communications are specified as well as various timer values.

The SlurmUser must be created as needed prior to starting Slurm and must exist on all nodes in your cluster. The parent directories for Slurm's log files, process ID files, state save directories, etc. are not created by Slurm. They must be created and made writable by SlurmUser as needed prior to starting Slurm daemons.

A description of the nodes and their grouping into partitions is required. A simple node range expression may optionally be used to specify ranges of nodes to avoid building a configuration file with large numbers of entries. The node range expression can contain one pair of square brackets with a sequence of comma separated numbers and/or ranges of numbers separated by a "-" (e.g. "linux[0-64,128]", or "lx[15,18,32-33]"). On BlueGene systems only, the square brackets should contain pairs of three digit numbers separated by a "x". These numbers indicate the boundaries of a rectangular prism (e.g. "bgl[000x144,400x544]"). See our Blue Gene User and Administrator Guide for more details. Up to two numeric ranges can be included in the expression (e.g. "rack[0-63]_blade[0-41]"). If one or more numeric expressions are included, one of them must be at the end of the name (e.g. "unit[0-31]rack" is invalid), but arbitrary names can always be used in a comma separated list.

Node names can have up to three name specifications: NodeName is the name used by all Slurm tools when referring to the node, NodeAddr is the name or IP address Slurm uses to communicate with the node, and NodeHostname is the name returned by the command /bin/hostname -s. Only NodeName is required (the others default to the same name), although supporting all three parameters provides complete control over naming and addressing the nodes. See the slurm.conf man page for details on all configuration parameters.

Nodes can be in more than one partition and each partition can have different constraints (permitted users, time limits, job size limits, etc.). Each partition can thus be considered a separate queue. Partition and node specifications use node range expressions to identify nodes in a concise fashion. This configuration file defines a 1154-node cluster for Slurm, but it might be used for a much larger cluster by just changing a few node range expressions. Specify the minimum processor count (CPUs), real memory space (RealMemory, megabytes), and temporary disk space (TmpDisk, megabytes) that a node should have to be considered available for use. Any node lacking these minimum configuration values will be considered DOWN and not scheduled. Note that a more extensive sample configuration file is provided in etc/slurm.conf.example. We also have a web-based configuration tool which can be used to build a simple configuration file, which can then be manually edited for more complex configurations.

    #
    # Sample /etc/slurm.conf for mcr.llnl.gov
    #
    ControlMachine=mcri
    ControlAddr=emcri
    BackupController=mcrj
    BackupAddr=emcrj
    #
    AuthType=auth/munge
    Epilog=/usr/local/slurm/etc/epilog
    FastSchedule=1
    JobCompLoc=/var/tmp/jette/slurm.job.log
    JobCompType=jobcomp/filetxt
    JobCredentialPrivateKey=/usr/local/etc/slurm.key
    JobCredentialPublicCertificate=/usr/local/etc/slurm.cert
    PluginDir=/usr/local/slurm/lib/slurm
    Prolog=/usr/local/slurm/etc/prolog
    SchedulerType=sched/backfill
    SelectType=select/linear
    SlurmUser=slurm
    SlurmctldPort=7002
    SlurmctldTimeout=300
    SlurmdPort=7003
    SlurmdSpoolDir=/var/tmp/slurmd.spool
    SlurmdTimeout=300
    StateSaveLocation=/tmp/slurm.state
    SwitchType=switch/elan
    TreeWidth=50
    #
    # Node Configurations
    #
    NodeName=DEFAULT CPUs=2 RealMemory=2000 TmpDisk=64000 State=UNKNOWN
    NodeName=mcr[0-1151] NodeAddr=emcr[0-1151]
    #
    # Partition Configurations
    #
    PartitionName=DEFAULT State=UP
    PartitionName=pdebug Nodes=mcr[0-191] MaxTime=30 MaxNodes=32 Default=YES
    PartitionName=pbatch Nodes=mcr[192-1151]

## Security

Besides authentication of Slurm communications based upon the value of the AuthType, digital signatures are used in job step credentials. This signature is used by slurmctld to construct a job step credential, which is sent to srun and then forwarded to slurmd to initiate job steps. This design offers improved performance by removing much of the job step initiation overhead from the slurmctld daemon. The digital signature mechanism is specified by the CryptoType configuration parameter and the default mechanism is MUNGE.

### OpenSSL

If using OpenSSL digital signatures, unique job credential keys must be created for your site using the program openssl. You must use openssl and not ssh-genkey to construct these keys. An example of how to do this is shown below. Specify file names that match the values of JobCredentialPrivateKey and JobCredentialPublicCertificate in your configuration file. The JobCredentialPrivateKey file must be readable only by SlurmUser. The JobCredentialPublicCertificate file must be readable by all users. Note that you should build the key files on one node and then distribute them to all nodes in the cluster. This insures that all nodes have a consistent set of digital signature keys. These keys are used by slurmctld to construct a job step credential, which is sent to srun and then forwarded to slurmd to initiate job steps.

openssl genrsa -out <sysconfdir>/slurm.key 1024
openssl rsa -in <sysconfdir>/slurm.key -pubout -out <sysconfdir>/slurm.cert

### MUNGE

If using MUNGE digital signatures, no Slurm keys are required. This will be addressed in the installation and configuration of MUNGE.

### Authentication

Authentication of communications (identifying who generated a particular message) between Slurm components can use a different security mechanism that is configurable. You must specify one "auth" plugin for this purpose using the AuthType configuration parameter. Currently, only three authentication plugins are supported: auth/none, auth/authd, and auth/munge. The auth/none plugin is built by default, but either Brent Chun's authd, or LLNL's MUNGE should be installed in order to get properly authenticated communications. Unless you are experienced with authd, we recommend the use of MUNGE. The configure script in the top-level directory of this distribution will determine which authentication plugins may be built. The configuration file specifies which of the available plugins will be utilized.

### Pluggable Authentication Module (PAM) support

A PAM module (Pluggable Authentication Module) is available for Slurm that can prevent a user from accessing a node which he has not been allocated, if that mode of operation is desired.

## Starting the Daemons

For testing purposes you may want to start by just running slurmctld and slurmd on one node. By default, they execute in the background. Use the -D option for each daemon to execute them in the foreground and logging will be done to your terminal. The -v option will log events in more detail with more v's increasing the level of detail (e.g. -vvvvvv). You can use one window to execute "slurmctld -D -vvvvvv", a second window to execute "slurmd -D -vvvvv". You may see errors such as "Connection refused" or "Node X not responding" while one daemon is operative and the other is being started, but the daemons can be started in any order and proper communications will be established once both daemons complete initialization. You can use a third window to execute commands such as "srun -N1 /bin/hostname" to confirm functionality.

Another important option for the daemons is "-c" to clear previous state information. Without the "-c" option, the daemons will restore any previously saved state information: node state, job state, etc. With the "-c" option all previously running jobs will be purged and node state will be restored to the values specified in the configuration file. This means that a node configured down manually using the scontrol command will be returned to service unless noted as being down in the configuration file. In practice, Slurm consistently restarts with preservation.

A thorough battery of tests written in the "expect" language is also available.

## Administration Examples

scontrol can be used to print all system information and modify most of it. Only a few examples are shown below. Please see the scontrol man page for full details. The commands and options are all case insensitive.

Print detailed state of all jobs in the system.

    adev0: scontrol
    scontrol: show job
    JobId=475 UserId=bob(6885) Name=sleep JobState=COMPLETED
       Priority=4294901286 Partition=batch BatchFlag=0
       AllocNode:Sid=adevi:21432 TimeLimit=UNLIMITED
       StartTime=03/19-12:53:41 EndTime=03/19-12:53:59
       NodeList=adev8 NodeListIndecies=-1
       NumCPUs=0 MinNodes=0 Shared=0 Contiguous=0
       MinCPUs=0 MinMemory=0 Features=(null) MinTmpDisk=0
       ReqNodeList=(null) ReqNodeListIndecies=-1

    JobId=476 UserId=bob(6885) Name=sleep JobState=RUNNING
       Priority=4294901285 Partition=batch BatchFlag=0
       AllocNode:Sid=adevi:21432 TimeLimit=UNLIMITED
       StartTime=03/19-12:54:01 EndTime=NONE
       NodeList=adev8 NodeListIndecies=8,8,-1
       NumCPUs=0 MinNodes=0 Shared=0 Contiguous=0
       MinCPUs=0 MinMemory=0 Features=(null) MinTmpDisk=0
       ReqNodeList=(null) ReqNodeListIndecies=-1

Print the detailed state of job 477 and change its priority to zero. A priority of zero prevents a job from being initiated (it is held in "pending" state).

    adev0: scontrol
    scontrol: show job 477
    JobId=477 UserId=bob(6885) Name=sleep JobState=PENDING
       Priority=4294901286 Partition=batch BatchFlag=0
       more data removed....
    scontrol: update JobId=477 Priority=0

Print the state of node adev13 and drain it. To drain a node, specify a new state of DRAIN, DRAINED, or DRAINING. Slurm will automatically set it to the appropriate value of either DRAINING or DRAINED depending on whether the node is allocated or not. Return it to service later.

    adev0: scontrol
    scontrol: show node adev13
    NodeName=adev13 State=ALLOCATED CPUs=2 RealMemory=3448 TmpDisk=32000
       Weight=16 Partition=debug Features=(null)
    scontrol: update NodeName=adev13 State=DRAIN
    scontrol: show node adev13
    NodeName=adev13 State=DRAINING CPUs=2 RealMemory=3448 TmpDisk=32000
       Weight=16 Partition=debug Features=(null)
    scontrol: quit
    Later
    adev0: scontrol
    scontrol: show node adev13
    NodeName=adev13 State=DRAINED CPUs=2 RealMemory=3448 TmpDisk=32000
       Weight=16 Partition=debug Features=(null)
    scontrol: update NodeName=adev13 State=IDLE

Reconfigure all Slurm daemons on all nodes. This should be done after changing the Slurm configuration file.

    adev0: scontrol reconfig

Print the curren. Slurm configuration. This also reports if the primary and secondary controllers (slurmctld daemons) are responding. To just see the state of the controllers, use the command ping.

    adev0: scontrol show config
    Configuration data as of 03/19-13:04:12
    AuthType          = auth/munge
    BackupAddr        = eadevj
    BackupController  = adevj
    BOOT_TIME         = 01/10-09:19:21
    CacheGroups       = 0
    CheckpointType    = checkpoint/none
    ControlAddr       = eadevi
    ControlMachine    = adevi
    ...
    WaitTime          = 0

    Slurmctld(primary/backup) at adevi/adevj are UP/UP

Shutdown all Slurm daemons on all nodes.

    adev0: scontrol shutdown

## OS X, Darwin

Build using the following execute line:
sh configure && MACOSX_DEPLOYMENT_TARGET=10.5 make all

## Testing

An extensive test suite is available within the Slurm distribution in testsuite/expect. There are about 250 tests which will execute on the order of 2000 jobs and 5000 job steps. Depending upon your system configuration and performance, this test suite will take roughly 80 minutes to complete. The file testsuite/expect/globals contains default paths and procedures for all of the individual tests. You will need to edit this file to specify where Slurm and other tools are installed. Set your working directory to testsuite/expect before starting these tests. Tests may be executed individually by name (e.g. test1.1) or the full test suite may be executed with the single command regression. See testsuite/expect/README for more information.

## Upgrades

Background: The Slurm version numbers contain three digits, which represent the major, minor and micro release numbers in that order (e.g. 14.11.3 is major=14, minor=11, micro=3). The major release number indicates the last two digits in the year of release and the minor release number indicates the month of release. Thus version 14.11.x was initially released in November 2014. Changes in the RPCs (remote procedure calls) and state files will only be made if the major and/or minor release number changes, which typically happens about once every nine months or so. Slurm's MPI libraries may also change if the major and/or minor release number change, requiring applications be re-linked (behavior may vary depending upon the MPI implementation used and the specific Slurm changes between releases). Locally developed Slurm plugins may also require modification. Slurm daemons will support RPCs and state files from the two previous minor releases (e.g. a version 15.08.x SlurmDBD will support slurmctld daemons and commands with a version of 14.03.x or 14.11.x). This means that upgrading at least once each year is recommended. Otherwise, intermediate upgrades will be required to preserve state information. Changes in the micro release number generally represent only bug fixes, but may also include very minor enhancements.

If the SlurmDBD daemon is used, it must be at the same or higher minor release number as the Slurmctld daemons. In other words, when changing the version to a higher release number (e.g from 14.11.x to 15.08.x) always upgrade the SlurmDBD daemon first. Database table changes may be required for the upgrade, for example adding new fields to existing tables. If the database contains a large number of entries, the SlurmDBD daemon may require tens of minutes to update the database and be unresponsive during this time interval.

The slurmctld daemon must also be upgraded before or at the same time as the slurmd daemons on the compute nodes. Generally, upgrading Slurm on all of the login and compute nodes is recommended, although rolling upgrades are also possible (i.e. upgrading the head node(s) first then upgrading the compute and login nodes later at various times). Also see the note above about reverse compatibility.

Pretty much each new major and/or minor release of Slurm (e.g. 14.11.x to 15.08.x) involves changes to the state files with new data structures, new options, etc. Slurm permits upgrades of up to two major or minor updates (e.g. 14.03.x or 14.11.x to 15.08.x) without loss of jobs or other state information, but the state information from older state files versions will not be recognized and will be discarded, resulting in loss of all running and pending jobs. State files are not recognized when downgrading (e.g. from 15.08.x to 14.11.x) and will be discarded, resulting in loss of all running and pending jobs. Therefore when upgrading Slurm (more precisely, the slurmctld daemon), saving the StateSaveLocation (as defined in slurm.conf) directory contents with all state information is recommended. If you need to downgrade, restoring that directory's contents will let you recover the jobs. Jobs submitted under the new version will not be in those state files, but it can let you recover most jobs. An exception to this is that jobs may be lost when installing new pre-release versions (e.g. 15.08.0-pre1 to 15.08.0-pre2). We'll try to note these cases in the NEWS file. Contents of major releases are also described in the RELEASE_NOTES file.

The libslurm.so version is increased every major release. So things like MPI libraries with Slurm integration should be recompiled. Sometimes it works to just symlink the old .so name(s) to the new one, but this has no guarantee of working.

Be mindful of your configured SlurmdTimeout and SlurmctldTimeout values. If the Slurm daemon's are down for longer than the specified timeout during an upgrade, nodes may be marked DOWN and their jobs killed. You can either increase the timeout values during an upgrade or insure that the slurmd daemons on compute nodes are not down for longer than SlurmdTimeout. The recommended upgrade order is as follows:

1. Shutdown the slurmdbd daemon
2. Upgrade the slurmdbd daemon
3. Restart the slurmdbd daemon
4. Shutdown the slurmctld daemon(s)
5. Shutdown the slurmd daemons on the compute nodes
6. Upgrade the slurmctld and slurmd daemons
7. Restart the slurmd daemons on the compute nodes
8. Restart the slurmctld daemon(s)

Note: The slurmd daemons must be restarted before the slurmctld daemons or within the SlurmdTimeout value. It is also possible to update the slurmd daemons on a node-by-node basis after the slurmctld daemon(s) are upgraded, but do make sure their down time is below the SlurmdTimeout value.

If you have built your own version of Slurm plugins, they will likely need modification to support a new version of Slurm. It is common for plugins to add new functions and function arguments during major updates. See the RELEASE_NOTES file for details.

## FreeBSD

FreeBSD administrators can install the latest stable Slurm as a binary package using:

    pkg install slurm-hpc

Or, it can be built and installed from source using:

    cd /usr/ports/sysutils/slurm-hpc && make install

The binary package installs a minimal Slurm configuration suitable for typical compute nodes. Installing from source allows the user to enable options such as mysql and gui tools via a configuration menu.

Last modified 15 April 2015