NeuroPoly Internal
Search…
Compute Canada

Create Account

Links to Compute Canada:
Useful resources:

Getting Started

Transfer Files

1
scp <username>@guillimin.clumeq.ca:<PATH/TO/FILE>
Copied!

Environment Variables

You have a .bash_profile and .bashrc on the server. Use .bahsrc rather than .bash_profile.
1
PATH=${PATH}:/gs/project/<RAPI>/bin/ants/bin
2
export ANTSPATH=/gs/project/<RAPI>/bin/ants/bin/
Copied!

Modules

There are pre-installed modules that you'll need to load in order to use (e.g. cmake). To see all modules available:
1
module avail
Copied!
To load module (you can put this in your .bashrc if you need the module all the time):
1
module load <module_name>
Copied!
Example: Check if git is available and load it
1
module avail git
2
module load apps/git/2.13.0
Copied!
You have to build everything from source because you don't have root permission to install anything yourself. You can send an email to guillimin "at" calculquebec "dot" ca if you need them to install something on your session. They are quite responsive.

Disk Space

You have:
    1.
    home folder /home/<username> β†’ 10GB
    2.
    project space /gs/project/<id> id is the one shared by all the people in the group (login to calculquebec website and you’ll find it) β†’ 1TB
To check how much space you have left:
1
serveur-info
Copied!
The folder common to the lab (where you need to work) is:
1
/gs/project/<RAPI>
Copied!

Create job script

Example
1
#!/bin/bash
2
#PBS -l nodes=1:ppn=16,pmem=31700m,walltime=48:00:00
3
#PBS -A <RAPI>
4
#PBS -o output.txt
5
#PBS -e error.txt
6
#PBS -V
7
#PBS -N build_template
8
cd /gs/project/<RAPI>/final_data_T2
9
bash buildtemplateparallel.sh -d 3 -o AVT -n 0 -c 2 -j 16 *.nii.gz
Copied!

Submit job

1
qsub my_job.sh
2
-p Defines the priority of the job. The priority argument must be a integer between -1024 and +1023 inclusive. The default is no priority which is equivalent to a priority of zero.
3
-m be sends email when job begins and terminates
4
-z Directs that the qsub command is not to write the job identifier assigned to the job to the command’s standard output.
Copied!

Check jobs

1
qstat -u $USER
2
S: Q (queue), R (running)
3
checkjob JobID (only the number!) [-v] [-v]
Copied!
1
# Once running:
2
showq -r -u $USER
Copied!

Kill job

1
qdel JobID
Copied!

Tips and Tricks

Use $SCRATCH disk to run your scripts, because $SCRATCH is much faster than $HOME.

Python

1
# activate python
2
module load python/3.6
3
# create virtual environment
4
virtualenv <VENV_NAME>
5
# activate it
6
source <VENV_NAME>/bin/activate
7
# deactivate
8
deactivate
Copied!

List of Servers

Cedar (CC)

Server: cedar.computecanada.ca Documentation: Compute Canada: Cedar​
Example: Run SCT on Cedar
256 CPUS 1GB of memory per process default lab account (def-jcohen) slurm queueing system
1
#!/bin/bash
2
#SBATCH --account=def-jcohen
3
#SBATCH --ntasks=256 # number of MPI processes (1 for the main process and 255 for the workers)
4
#SBATCH --mem-per-cpu=1024M # memory; default unit is megabytes
5
#SBATCH --time=0-03:00 # time (DD-HH:MM)
6
7
# cd to a scratch space where outputs will be saved
8
cd /scratch/$USER/workdir
9
10
DATA_PATH=</PATH/TO/SUBJECTS/DATA/>
11
12
</PATH/TO/SCT_PIPELINE> --nb-cup 255 -f sct_propseg -d </PATH/TO/BIG_DATASET> -p \"-i t2s/t2s.nii.gz -c t2s\" # run propseg on 255 workers
Copied!
You can then submit the job with sbatch:
1
chmod 750 my_sct_script.sh # make sure the script is executable
2
sbatch my_sct_script.sh # submit the job
3
squeue -u $UID # check that the job is in the slurm queue
Copied!
One can also do interactive testing with MPI. We recommend using this method to run SCT on a smaller scale. For example, here we run on 16 cores:
1
salloc --time=0-03:00 --ntasks=16 --mem-per-cpu=1024M --account=def-jcohen
2
# [wait for cedar to give you a session, bigger are ntasks and mem-per-cpu, longer is that time]
Copied!
Then run sct_pipeline. The 16 cores will be accessible via the MPI interface:
1
<PATH/TO/SCT>/bin/sct_pipeline -cpu-nb 16 -f sct_propseg -d /home/poq/small -p \" -i t2s/t2s.nii.gz -c t2s \"
Copied!

Colosse (CQ/Laval)

Server: colosse.calculquebec.ca Documentation: https://wiki.calculquebec.ca/w/Colosse​

Guillimin (CQ/McGill)

Server: guillimin.clumeq.ca Documentation:

Briaree (CQ/UdeM)

Server: briaree.calculquebec.ca

Graham (CC)

Server: graham.computecanada.ca Documentation: Compute Canada: Graham Server​
To run the SCT on Graham, follow the same procedure as describe in the Cedar section.

Mammoth (CQ/Sherbrooke)

Server: jcohen-mp2.ccs.usherbrooke.ca

HELIOS (Quebec, GPU)

Make sure that the version you need has already been compiled:
1
ls /software/gpu/apps/python/wheelhouse/tensorflow-0.*
Copied!
Load the required modules. For example:
1
module load compilers/gcc/4.8.5 cuda/7.5 libs/cuDNN/5
Copied!
Load the correct Python module:
1
module load apps/python/3.6.0
Copied!
Create a virtual environment and activate it:
1
virtualenv <ENV_NAME>
2
source <ENV_NAME>/bin/activate
Copied!
Install tensorflow. If you need a version other than the latest version, you can specify the version number.
1
pip install tensorflow==<VERSION_NUMBER>+helios
Copied!
The addition of +helios after the version number isn't necessary, but it ensures that pip doesn't download a version from the internet, and instead uses the version you have compiled for Helios.
Submitting a script (use MOAB):
1
([email protected]) $ msub [options] script.pbs
Copied!

Cloud Account

There is the possibility to host processes on the cloud for public access. To do so, each PI needs to open a specific account to allocate cloud resource at: Compute Canada: Cloud.
Last modified 4mo ago