Slurm Job Arrays¶

Job arrays let you submit many similar jobs at once. Each job can have a slight variation, such as a different input file or parameter, without writing separate job scripts for each one.

Example Task¶

Extract reads aligned to chromosome 21 from the BAM file of each sample in the SEA-3 data set.

Option 1: Submit Jobs in a Loop¶

while read BAM_PATH
do
  sbatch extract_chromosome_from_bam_file_ssubmit.sh ${BAM_PATH} chr21
done

This approach is familiar and simple, but submitting hundreds of jobs requesting multiple cores can consume most available resources and disrupt other users.

Option 2: Use a Job Array¶

Submit the array script with a file of file names, also called a FOFN, and a target chromosome.

sbatch extract_chromosome_from_bam_file_ssubmitArray.sh FOFN TARGET_CHR

Example array script:

#!/bin/bash
#SBATCH --job-name=extract_chrom
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=4G
#SBATCH --output=logs/%x-%A_%a.out
#SBATCH --error=logs/%x-%A_%a.err
#SBATCH --array=1-433

TARGET_CHR=${2}
INPUT_BAM_FILE=$(sed -n "${SLURM_ARRAY_TASK_ID}"p ${1})
OUTPUT_DIR="~/rcc_demo/outputs"

OUTPUT_FILE_PREFIX=$(basename ${INPUT_BAM_FILE} .bam)
OUTPUT_BAM_FILE="${OUTPUT_DIR}/${OUTPUT_FILE_PREFIX}.${TARGET_CHR}.bam"

samtools view --threads ${SLURM_CPUS_PER_TASK} \
              ${INPUT_BAM_FILE} \
              -o ${OUTPUT_BAM_FILE} \
              ${TARGET_CHR}

Job arrays give you more control and flexibility, especially when one or a few tasks fail.

How Array Task IDs Work¶

When you submit a job array, Slurm creates multiple jobs. Each one is called a task and has a unique ID called SLURM_ARRAY_TASK_ID.

For example:

sbatch --array=1-5 my_script.sh

Slurm runs my_script.sh five times, with SLURM_ARRAY_TASK_ID set to 1, 2, 3, 4, and 5.

If the submission receives job ID 2157, the task IDs are:

2157_1
2157_2
2157_3
2157_4
2157_5

Reading the Matching FOFN Line¶

The array script uses this line to select the input BAM file:

INPUT_BAM_FILE=$(sed -n "${SLURM_ARRAY_TASK_ID}"p ${1})

Breakdown:

sed is a stream editor used to parse and transform text
-n suppresses automatic printing of pattern space
"${SLURM_ARRAY_TASK_ID}p" tells sed to print only the line matching the array task ID
${1} is the first positional argument passed to the script, expected to be the FOFN

Array Log File Names¶

For regular jobs, log files often use %x-%j, where %x is the job name and %j is the job ID.

For arrays, use %x-%A_%a:

%x is the job name
%A is the array job ID
%a is the array task ID

Useful Array Submission Patterns¶

Submit a single task:

sbatch --array=21 extract_chromosome_from_bam_file_ssubmitArray.sh FOFN TARGET_CHR

Submit a range or non-consecutive list:

sbatch --array=21-23,97 extract_chromosome_from_bam_file_ssubmitArray.sh FOFN TARGET_CHR

Limit the number of tasks that run simultaneously:

sbatch --array=1-433%10 extract_chromosome_from_bam_file_ssubmitArray.sh FOFN TARGET_CHR

Submit indices between 1 and 7 with a step size of 2:

sbatch --array=1-7:2 extract_chromosome_from_bam_file_ssubmitArray.sh FOFN TARGET_CHR

Canceling Arrays or Tasks¶

Cancel all tasks in a job array:

scancel 24668

Cancel task IDs 7 through 10 from job array 24668:

scancel 24668_[7-10]

Cancel task IDs 7 and 10 from job array 24668:

scancel 24668_7 24668_10