Slurm Tips & Tricks

Here are a few advanced Slurm usage tricks to the experienced user

Slurm Job Arrays

Slurm job arrays allow you to submit multiple, similar jobs at once without creating separate scripts for each. Each job in the array is called a task and gets its own array ID. This is particularly useful for parameter sweeps, batch simulations, or processing multiple files in parallel.

Basic Syntax

sbatch --array=0-9 myscript.sh

  • --array=0-9 creates 10 tasks with IDs from 0 to 9.

    • you can also pass a comma-separated list of ids if you wish

  • Each task runs myscript.sh independently.

  • Slurm sets the environment variable $SLURM_ARRAY_TASK_ID to the task’s ID.

Accessing Task ID in Script

Inside myscript.sh, you can use the array ID to differentiate tasks. For example:

#!/bin/bash 
#SBATCH --job-name=array_example 
#SBATCH --output=output_%A_%a.txt 
#SBATCH --error=error_%A_%a.txt 
#SBATCH --time=00:10:00 

# $SLURM_ARRAY_TASK_ID is the task ID 
echo "Running task $SLURM_ARRAY_TASK_ID"  
# Example: process different input files INPUT_FILE="input_$SLURM_ARRAY_TASK_ID.txt" cat $INPUT_FILE`

  • %A → master job ID

  • %a → array task ID

This ensures each task writes to a separate output file.

Submitting a Job Array

sbatch --array=0-99 myscript.sh

  • Creates 100 tasks (IDs 0–99).

  • Slurm can limit concurrent tasks using %:

sbatch --array=0-99%10 myscript.sh

  • Only 10 tasks run simultaneously; the rest wait in the queue.

Dumping a Run-script

One of Slurm’s best features is accounting. A really handy feature is the ability to dump previous jobs’ run-scripts.

First, run sacct -X to get the job ID; you may need to add -S YEAR-MM-DD to include older jobs.

Then, dump the run-script.

sacct --batch-script --jobs <jobid>

This will print your subscript exactly as it was submitted to the scheduler, very helpful for when a particular run did well but you don’t remember which parameters it used.