.SH "DESCRIPTION" \fBsacctmgr\fR is used to view or modify Slurm account information. The account information is maintained within a database with the interface being provided by \fBslurmdbd\fR (Slurm Database daemon). This database can serve as a central storehouse of user and computer information for multiple computers at a single site. Slurm account information is recorded based upon four parameters that form what is referred to as an \fIassociation\fR. These parameters are \fIuser\fR, \fIcluster\fR, \fIpartition\fR, and \fIaccount\fR. \fIuser\fR is the login name. \fIcluster\fR is the name of a Slurm managed cluster as specified by the \fIClusterName\fR parameter in the \fIslurm.conf\fR configuration file. \fIpartition\fR is the name of a Slurm partition on that cluster. \fIaccount\fR is the bank account for a job. The intended mode of operation is to initiate the \fBsacctmgr\fR command, add, delete, modify, and/or list \fIassociation\fR records then commit the changes and exit. \fBNOTE\fR: The contents of Slurm's database are maintained in lower case. This may result in some \f3sacctmgr\fP output differing from that of other Slurm commands. .SH "OPTIONS" .TP \fB\-s\fR, \fB\-\-associations\fR Use with show or list to display associations with the entity. This is equivalent to the \fBassociations\fR command. .IP .TP \fB\-h\fR, \fB\-\-help\fR Print a help message describing the usage of \fBsacctmgr\fR. This is equivalent to the \fBhelp\fR command. .IP .TP \fB\-i\fR, \fB\-\-immediate\fR Commit changes immediately without asking for confirmation. .IP .TP \f3\-\-json\fP, \f3\-\-json\fP=\fIlist\fR, \f3\-\-json\fP=<\fIdata_parser\fR> Dump information as JSON using the default data_parser plugin or explicit data_parser with parameters. Sorting and formatting arguments will be ignored. .IP .TP \fB\-n\fR, \fB\-\-noheader\fR No header will be added to the beginning of the output. .IP This is equivalent to the \fBquiet\fR command. .IP .TP \fB\-r\fR, \fB\-\-readonly\fR Makes it so the running sacctmgr cannot modify accounting information. The \fBreadonly\fR option is for use within interactive mode. .IP .TP \f3\-\-yaml\fP, \f3\-\-yaml\fP=\fIlist\fR, \f3\-\-yaml\fP=<\fIdata_parser\fR> Dump information as YAML using the default data_parser plugin or explicit data_parser with parameters. Sorting and formatting arguments will be ignored. .IP .TP \fB\-v\fR, \fB\-\-verbose\fR Enable detailed logging. This is equivalent to the \fBverbose\fR command. .IP .TP \fB\-V\fR , \fB\-\-version\fR Display version number. This is equivalent to the \fBversion\fR command. .IP .SH "COMMANDS" .TP \fBadd\fR <\fIENTITY\fR> <\fISPECS\fR> Add an entity. Identical to the \fBcreate\fR command. .IP .TP \fBarchive\fR {dump|load} <\fISPECS\fR> Write database information to a flat file or load information that has previously been written to a file. .IP .TP \fBclear stats\fR Clear the server statistics. .IP .TP \fBcreate\fR <\fIENTITY\fR> <\fISPECS\fR> Add an entity. Identical to the \fBadd\fR command. .IP Display a description of sacctmgr options and commands. .IP .TP \fBlist\fR <\fIENTITY\fR> [<\fISPECS\fR>] Display information about the specified entity. By default, all entries are displayed, you can narrow results by specifying SPECS in your query. Identical to the \fBshow\fR command. .IP .TP \fBload\fR <\fIFILENAME\fR> Load cluster data from the specified file. This is a configuration file generated by running the sacctmgr dump command. This command does not load archive data, see the sacctmgr archive load option instead. .IP .TP \fBmodify\fR <\fIENTITY\fR> \fBwhere\fR <\fISPECS\fR> \fBset\fR <\fISPECS\fR> Modify an entity. .IP .TP \fBreconfigure\fR Reconfigures the SlurmDBD if running with one. .IP .TP \fBremove\fR <\fIENTITY\fR> \fBwhere\fR <\fISPECS\fR> Delete the specified entities. Identical to the \fBdelete\fR command. .IP .TP \fBshow\fR <\fIENTITY\fR> [<\fISPECS\fR>] Display information about the specified entity. By default, all entries are displayed, you can narrow results by specifying SPECS in your query. Identical to the \fBlist\fR command. .IP .TP \fBshutdown\fR Shutdown the server. .IP .TP \fBversion\fP Display the version number of sacctmgr. .IP Print no messages other than error messages. .IP .TP \fBquit\fP Terminate the execution of sacctmgr interactive mode. Identical to the \fBexit\fR command. .IP .TP \fBverbose\fP Enable detailed logging. This includes time\-stamps on data structures, record counts, etc. This is an independent command with no options meant for use in interactive mode. .IP .TP \fB!!\fP Repeat the last command. .IP .SH "ENTITIES" .TP \fBaccount\fR A bank account, typically specified at job submit time using the \fB\-\-account=\fR option. These may be arranged in a hierarchical fashion, for example accounts 'chemistry' and 'physics' may be children of the account 'science'. The hierarchy may have an arbitrary depth. .IP .TP \fBassociation\fR The entity used to group information consisting of four parameters: \fBaccount\fR, \fBcluster\fR, \fBpartition\fR (optional), and \fBuser\fR. Used only with the \fBlist\fR or \fBshow\fR command. Add, modify, and delete should be done to a user, account or cluster entity, which will in turn update the underlying associations. Modification of attributes like limits is allowed for an association but not a modification of the four core attributes of an association. You cannot change the partition setting (or set one if it has not been set) for an existing association. Instead, you will need to create a new association with the partition included. You can either keep the previous association with no partition defined, or delete it. Note that these newly added associations are unique entities and any existing usage information will not be carried over to the new association. .IP .TP \fBcluster\fR This should be a trusted person since they can change limits on account and user associations, as well as cancel, requeue or reassign accounts of jobs inside their realm. .IP .TP \fBevent\fR Events like downed or draining nodes on clusters. .IP .TP \fBfederation\fR A group of clusters that work together to schedule jobs. .IP .TP \fBjob\fR Used to modify specific fields of a job: Derived Exit Code, Comment, AdminComment, Extra, SystemComment, or WCKey. .IP .TP \fBproblem\fR Use with \fBshow\fR or \fBlist\fR to display entity problems. .IP .TP \fBqos\fR Quality of Service. .IP .TP \fBreservation\fR A collection of resources set apart for use by a particular account, user or group of users for a given period of time. .IP .TP \fBresource\fR Software resources for the system. Those are software licenses shared among clusters. .IP .TP \fBRunawayJobs\fR Used only with the \fBlist\fR or \fBshow\fR command to report current jobs that have been orphaned on the local cluster and are now runaway. If there are jobs in this state it will also give you an option to "fix" them. \fBNOTE\fR: You must have an \fBAdminLevel\fR of at least \fBOperator\fR to perform this. .IP \fBtres\fR Used with \fBlist\fR or \fBshow\fR command to view a list of Trackable RESources configured on the system. .IP .TP \fBuser\fR The login name. Usernames are case\-insensitive (forced to lowercase) unless the \fBPreserveCaseUser\fR option has been set in the SlurmDBD configuration file. .IP .TP \fBwckeys\fR Workload Characterization Key. An arbitrary string for grouping orthogonal accounts. .IP .SH "GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES" \fBNOTE\fR: The group limits (GrpJobs, GrpTRES, etc.) are tested when a job is being considered for being allocated resources. If starting a job would cause any of its group limit to be exceeded, that job will not be considered for scheduling even if that job might preempt other jobs which would release sufficient group resources for the pending job to be initiated. .TP \fBDefaultQOS\fR=<\fIdefault_qos\fR> The default QOS this association and its children should have. This is overridden if set directly on a user. To clear a previously set value use the modify command with a new value of \-1. .IP .IP \fBFairshare\fR={<\fIfairshare_number\fR>|parent} .PD 0 .IP \fBShare\fR={<\fIfairshare_number\fR>|parent} .PD Number used in conjunction with other accounts to determine job priority. Can also be the string \fIparent\fR, when used on a user this means that the parent association is used for fairshare. If Fairshare=parent is set on an account, that account's children will be effectively re\-parented for fairshare calculations to the first parent of their parent that is not Fairshare=parent. Limits remain the same, only its fairshare value is affected. To clear a previously set value use the modify command with a new value of \-1. .IP .TP \fBGrpJobs\fR=<\fImax_jobs\fR> Maximum number of running jobs in aggregate for this association and all associations which are children of this association. To clear a previously set value use the modify command with a new value of \-1. .IP this association. To clear a previously set value use the modify command with a new value of \-1. .IP .TP \fBGrpTRES\fR=TRES=<\fImax_TRES\fR>[,TRES=<\fImax_TRES\fR>,...] Maximum number of TRES running jobs are able to be allocated in aggregate for this association and all associations which are children of this association. To clear a previously set value use the modify command with a new value of \-1 for each TRES id. \fBTRES\fR can be one of the Slurm defaults (i.e. \fIcpu\fR, \fImem\fR, \fInode\fR, etc...), or any defined generic resource. You can see the list of available resources by running \fBsacctmgr show tres\fR. \fBNOTE\fR: This limit only applies fully when using the Select Consumable Resource plugin. .IP .TP \fBGrpTRESMins\fR=TRES=<\fIminutes\fR>[,TRES=<\fIminutes\fR>,...] The total number of TRES minutes that can possibly be used by past, present and future jobs running from this association and its children. To clear a previously set value use the modify command with a new value of \-1 for each TRES id. \fBTRES\fR can be one of the Slurm defaults (i.e. \fIcpu\fR, \fImem\fR, \fInode\fR, etc...), or any defined generic resource. You can see the list of available resources by running \fBsacctmgr show tres\fR. \fBNOTE\fR: This limit is not enforced if set on the root association of a cluster. So even though it may appear in sacctmgr output, it will not be enforced. \fBNOTE\fR: This limit only applies when using the Priority Multifactor plugin. The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. When this limit is reached all associated jobs running will be killed and all future jobs submitted with associations in the group will be delayed until they are able to run inside the limit. .IP .TP \fBGrpTRESRunMins\fR=TRES=<\fIminutes\fR>[,TRES=<\fIminutes\fR>,...] Used to limit the combined total number of TRES minutes used by all jobs running with this association and its children. This takes into consideration time limit of running jobs and consumes it, if the limit is reached no new jobs are started until other jobs finish to allow time to free up. To clear a previously set value use the modify command with a new value of \-1 for each TRES id. \fBTRES\fR can be one of the Slurm defaults (i.e. \fIcpu\fR, \fImem\fR, \fBNOTE\fR: This limit only applies when using the Priority Multifactor plugin. The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. When this limit is reached all associated jobs running will be killed and all future jobs submitted with associations in the group will be delayed until they are able to run inside the limit. .IP .TP \fBMaxJobs\fR=<\fImax_jobs\fR> Maximum number of jobs each user is allowed to run at one time in this association. This is overridden if set directly on a user. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of \-1. .IP .TP \fBMaxJobsAccrue\fR=<\fImax_jobs\fR> Maximum number of pending jobs able to accrue age priority at any given time for the given association. This is overridden if set directly on a user. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of \-1. .IP .IP \fBMaxSubmit\fR=<\fImax_jobs\fR> .PD 0 .IP \fBMaxSubmitJobs\fR=<\fImax_jobs\fR> .PD Maximum number of jobs which this association can have in a pending or running state at any time. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of \-1. .IP .IP \fBMaxTRESMins\fR=TRES=<\fIminutes\fR>[,TRES=<\fIminutes\fR>,...] .PD 0 .IP \fBMaxTRESMinsPerJob\fR=TRES=<\fIminutes\fR>[,TRES=<\fIminutes\fR>,...] .PD Maximum number of TRES minutes each job is able to use in this association. This is overridden if set directly on a user. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of \-1 for each TRES id. \fBTRES\fR can be one of the Slurm defaults (i.e. \fIcpu\fR, \fImem\fR, \fInode\fR, etc...), or any defined generic resource. You can see the list of available resources by running \fBsacctmgr show tres\fR. .IP .IP \fBMaxTRES\fR=TRES=<\fImax_TRES\fR>[,TRES=<\fImax_TRES\fR>,...] \fBcons_tres\fR select type plugin. .IP .IP \fBMaxWall\fR=<\fImax_wall\fR> .PD 0 .IP \fBMaxWallDurationPerJob\fR=<\fImax_wall\fR> .PD Maximum wall clock time each job is able to use in this association. This is overridden if set directly on a user. Default is the cluster's limit.fR(5) man page for a full list of flags. The environment variable takes precedence over the setting in the slurm.conf. .IP .SH "EXAMPLES" \fBNOTE\fR: There is an order to set up accounting associations. You must define clusters before you add accounts and you must add accounts $ sacctmgr dump cluster=tux file=tux_data_file $ sacctmgr load tux_data_file .fi A user's account can not be changed directly. A new association needs to be created for the user with the new account. Then the association with the old account can be deleted. When modifying an object placing the key words 'set' and the optional 'where' is critical to perform correctly below are examples to produce correct results. As a rule of thumb anything you put in front of the set will be used as a quantifier. If you want to put a quantifier after the key word 'set' you should use the key word 'where'. The following is wrong: .nf $ sacctmgr modify user name=adam set fairshare=10 cluster=tux .fi This will produce an error as the above line reads modify user adam set fairshare=10 and cluster=tux. Either of the following is correct: .nf $ sacctmgr modify user name=adam cluster=tux set fairshare=10 $ sacctmgr modify user name=adam set fairshare=10 where cluster=tux .fi When changing qos for something only use the '=' operator when wanting to explicitly set the qos to something. In most cases you will want to use the '+=' or '\-=' operator to either add to or remove from the existing qos already in place. If a user already has qos of normal,standby for a parent or it was explicitly set you should use qos+=expedite to add this to the list in this fashion. If you are looking to only add the qos expedite to only a certain account and or cluster you can do that by specifying them in the sacctmgr line. .nf $ sacctmgr modify user name=adam set qos+=expedite .fi or .nf $ sacctmgr modify user name=adam acct=this cluster=tux set qos+=expedite .fi Let's give an example how to add QOS to user accounts. List all available QOSs in the cluster. \-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- zebra root normal zebra root normal zebra g normal zebra g1 normal .fi Add the QOS expedite to account G1 and display the result. Using the operator += the QOS will be added together with the existing QOS to this account. .nf $ sacctmgr modify account name=g1 set qos+=expedite $ sacctmgr show assoc format=cluster,account,qos Cluster Account QOS \-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- zebra root normal zebra root normal zebra g normal zebra g1 expedite,normal .fi Now set the QOS expedite as the only QOS for the account G and display the result. Using the operator = that expedite is the only usable QOS by account G .nf $ sacctmgr modify account name=G set qos=expedite $ sacctmgr show assoc format=cluster,account,qos Cluster Account QOS \-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- zebra root normal zebra root normal zebra g expedite zebra g1 expedite,normal .fi If a new account is added under the account G it will inherit the QOS expedite and it will not have access to QOS normal. .nf $ sacctmgr add account banana parent=G $ sacctmgr show assoc format=cluster,account,qos Cluster Account QOS \-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\- \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- zebra root normal zebra root normal zebra g expedite zebra banana expedite zebra g1 expedite,normal .fi .fi .SH "COPYING" Copyright (C) 2008\-2010 Lawrence Livermore National Security. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER). .br Copyright (C) 2010\-2022 SchedMD LLC. .LP This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>. .LP Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. .LP Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. .SH "SEE ALSO" \fBslurm.conf\fR(5), \fBslurmdbd\fR(8)