• Departments & Offices
• IT
• Services
• Research Computing
• Using
• Running HPC jobs with SLURM

Guide to running jobs on main-campus / VIMS HPC cluster

Migrating from Torque

Slurm is the batch system used to submit jobs on all main-campus and VIMS HPC clusters.   For those that are familiar with Torque, the following table may be helpful:

Table 1:  Torque vs. Slurm commands

Important differences between running jobs under Torque vs. Slurm:

1. NEW - now you can ssh into any node you are using within a job from the cluster front-end/login machine.
2. NEW - To do mpi over ssh (which some codes require), you will need to create/modify your .ssh config file and add an ssh-key.   See our page on:  Using Gaussian 16 on HPC for more information.
3. If you use a shell type other than your usual startup shell (i.e. run a batch script with bash but you use tcsh as your default shell), you will need to add "source /usr/local/etc/sciclone.bashrc" or "source /usr/local/etc/sciclone.cshrc" if you are using main-campus compute clusters, or "source /usr/local/etc/chesapeake.cshrc" or "source /usr/local/etc/chesapeake.bashrc".   For example, say your default shell is tcsh (the majority of users use tcsh, however you can tell for sure by entering "echo $0" at a prompt).   To use bash in a batch script, you would add the command "source /usr/local/etc/sciclone.bashrc" after your #SBATCH lines in your batch script.
4. All jobs start their life in the submission directory (where you run sbatch or salloc).   This is different than Torque which always started in your home directory.
5. sbatch jobs do NOT source your bashrc.XXX and will inherit the environment that exists when you run sbatch.
6. salloc DOES source your startup environment and will load your startup modules.
7. Most clusters (excluding main-campus 'bora/hima' and VIMS campus potomac and pamunkey) do not require a specific type of node (-C, --constraint) to be specified.
8. Main-campus clusters require submission from their respective front-end.   i.e. run sbatch, salloc from femto for femto jobs, kuro, for kuro jobs.   
9. The main-campus 'hima' cluster jobs must be submitting from bora (its front-end / login machine).  Use -C hi to specify hima, use -C bo to specify bora.  
10. VIMS cluster jobs can be submitted to any VIMS cluster from either chesapeake or james using -C or --contstraint (see examples below).

Running jobs with SLURM

Like most batch systems, SLURM allows one to request compute resources (nodes, memory, gpus, etc.) and then use these resources to run executables in the compute environment.   Under SLURM, there are multiple ways of doing this:

salloc - interactive session on nodes ; same as qsub -I under Torque

sbatch - submit a batch script for execution ; same as qsub under Torque

srun - run MPI job or a single command on a set of resources 

The following table lists the common options for selecting compute resources:

Table 2: Controlling slurm resources:

salloc:  

salloc requests resources for an interactive session.   Here are some examples:

sbatch:

sbatch submits a script to the batch system that will run once resources are available.    All batch scripts consists of two sections, the SLURM directives and the commands the user wants to run.  Here are some examples:

Table 4: sbatch examples:

#!/bin/tcsh
#SBATCH --job-name=serial 
#SBATCH -N 1 -n1
#SBATCH -t 0:30:00

#!/bin/tcsh
#SBATCH --job-name=kurotest
#SBATCH -N 5 --ntasks-per-node 64
#SBATCH -t 1-0

#!/bin/tcsh
#SBATCH --job-name=boratest
#SBATCH -N 10 --ntasks-per-node 20
#SBATCH -t 0:30:00

#!/bin/tcsh
#SBATCH --job-name=himatest
#SBATCH -N 1 -n  32 -C hi
#SBATCH -t 0:30:00

#!/bin/tcsh
#SBATCH --job-name=potomac_test
#SBATCH -N 1 -n 12 -C pt
#SBATCH -t 0:30:00

#!/bin/bash
#SBATCH --job-names=james_test
#SBATCH -N 5 --ntasks-per-node 20 -C jm
#SBATCH -t 2-0

 srun:

srun is the command for launching mpi jobs within SLURM.   It takes the place of mpirun, mvp2run, mpiexec for SLURM systems.   srun accepts the same arguments that sbatch and salloc take.  However, for the vast majority of cases, srun will not need arguments since the batch script or salloc command line has already selected the resources srun will use.

srun can also be used to run a command  (parallel or serial) on a set of resources from the front-end/login machine directly.  This second use is mainly for testing purposes    

e.g.   srun -N1 -n8 hostname  # will run 'hostname' from the front-end on one node and 8 cores.

Other useful slurm commands:

 array jobs:
Array jobs are a series of jobs that each get a unique array id that can be used within your job script.  This is useful for parameters studies where you have a number of different input files to work through.   For instance, imagine a scenario in which you have ten input files named INPUT.1 through INPUT.10.   You want to submit 10 jobs, each of which runs a different input file.   To do this you can submit the following script:

This will run 10 jobs, each using 1 core on 1 node for 30 minutes. Each array job will substitute its value of $SLURM_ARRAY_TASK_ID.

Standing reservations for debugging / short tests:
Currently, there are no resources set aside for debugging on main-campus or VIMS clusters. This will be changed in the near future.


Checking load on job
For jobs that use whole nodes, i.e. MPI/parallel jobs, it is useful to be able to check the load before running.  The script ckload can be run at the beginning of a job to report the high load on any nodes in the job and, optionally, kill the job so it can be submitted again.    

#!/bin/tcsh
#SBATCH --job-name=kurotest
#SBATCH -N 5 --ntasks-per-node 64
#SBATCH -t 1-0

In the above batch script example, the job is killed if any of the allocated five nodes have a 1-min avg load larger than 0.05.