Running fmriprep or mriqc on a cluster#
Connect to the cluster#
Open your terminal and type:
ssh <username>@graham.computecanada.ca # Graham login node
Add your ssh key#
SSH key pairs are very useful to avoid typing passwords
Modules#
There are pre-installed modules that you’ll need to load in order to use. To see
all modules available: module avail
To load module (you can put this in your .bashrc if you need the module all the
time): module load <module_name>
Example: Check if git is available and load it
module avail git
module load apps/git/2.13.0
Transfer files#
Use scp to copy individual files and directories#
scp <filename> <username>@graham.computecanada.ca:<PATH/TO/FILE>
scp <username>@graham.computecanada.ca:<PATH/TO/FILE> <LocalPath>
Use rsync to sync files or directories#
rsync <LocalPath/filename> <username>@graham.computecanada.ca:<PATH/TO/FILE>
rsync <username>@graham.computecanada.ca:<PATH/TO/FILE> <LocalPath>
Use datalad#
Install Datalad on your cluster :
module load git-annex python/3
virtualenv ~/venv_datalad
source ~/venv_datalad/bin/activate
pip install datalad
Running jobs#
Create job script#
Here is an example of a simple bash script:
#!/bin/bash
#SBATCH --time=00:05:00
#SBATCH --account=def-flepore
echo 'Hello, world!'
sleep 20
Submit job#
In the cluster terminal
sbatch <name of the file>
Example:
sbatch simple.sh
Submitted batch job 65869853
Check job status#
Use squeue or sq to list jobs
sq
JOBID USER ACCOUNT NAME ST TIME_LEFT NODES CPUS TRES_PER_N MIN_MEM NODELIST (REASON)
65869853 mmaclean def-flepore_cpu simple.sh PD 5:00 1 1 N/A 256M (Priority)
Use email notification to learn when your job starts and ends by adding the following at the top of your script:
#SBATCH --mail-user=michele.maclean@umontreal.ca
#SBATCH --mail-type=BEGIN
#SBATCH --mail-type=END
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=REQUEUE
#SBATCH --mail-type=ALL
Cancel job#
scancel <jobid>
scancel 65869853
Where does the output go#
By default the output is placed in a file named "slurm-", suffixed with the job ID number and ".out", e.g. slurm-65869853.out, in the directory from which the job was submitted. Having the job ID as part of the file name is convenient for troubleshooting Files will be output according to where you specified in your bash script
Tip#
- Use
$SCRATCH
disk to run your scripts, because$SCRATCH
is much faster than $HOME`. - Keep a working directory, if things crash you don't have to start from the beginning
- check usage/space left on cluster:
diskusage_report
- See if you need to delete files from scratch once jobs are complete- fmriprep output is quite heavy
Running singularity on a cluster#
Download the containers for fmriprep and mriqc here:
Repro nim containers: https://github.com/ReproNim/containers
- create directory
mkdir parallel_analysis
- install containers from repronim
cd parallel_analysis
datalad install https://github.com/ReproNim/containers.git
- retrieve the container you want, e.g., fmriprep
datalad get containers/images/bids/bids-fmriprep--21.0.1.sing
you might need to unlock the container to be able to use it
datalad unlock containers/images/bids/bids-fmriprep--21.0.1.sing
Run fmriprep on cluster#
Have your freesurfer license ready.
Here is an example script
#!/bin/bash
# fail whenever something is fishy
# -e exit immediately
# -x to get verbose logfiles
# -u to fail when using undefined variables
# https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html
set -e -x -u -o pipefail
#-------------------------------------------
#SBATCH -J fmriprep
#SBATCH --account=def-flepore
#SBATCH --time=15:00:00
#SBATCH -n 1
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=8G
#SBATCH --mail-user=michele.maclean@umontreal.ca
#SBATCH --mail-type=BEGIN
#SBATCH --mail-type=END
#SBATCH --mail-type=FAIL
#SBATCH --mail-type=REQUEUE
#SBATCH --mail-type=ALL
# ------------------------------------------
source ~/venv_datalad/bin/activate
module load git-annex/8.20200810
module load freesurfer/5.3.0
module load singularity/3.8
cd $HOME || exit
singularity run \
--cleanenv \
-B $HOME/scratch:/scratch \
-B $HOME/projects/def-flepore/mmaclean:/mmaclean \
$HOME/projects/def-flepore/mmaclean/parallel_analysis/containers/images/bids/bids-fmriprep--21.0.1.sing \
/mmaclean/raw \
/mmaclean/fmriprep-output \
participant \
--participant-label CTL01 \
--work-dir /scratch/work-fmriprep \
--fs-license-file /mmaclean/license/freesurfer.txt \
--output-spaces MNI152NLin2009cAsym T1w \
--skip_bids_validation \
--notrack \
--stop-on-first-crash
Extra reference: