These training videos are aimed at new users of high–performance computing (HPC). They cover how to connect to an HPC facility and transfer files, basic Linux/Unix commands and the basics of how to use an HPC.
For further Linux training, we recommend this external resource: https://ctcms-uq.github.io/
Note: Click on the red links below to access the videos.
Set 1: Connecting to HPC
Users need to use a program that uses secure shell to connect to a HPC facility. Linux/Unix and Mac users will know how to use the command ssh to connect to HPC. The below linked videos give an overview of two such programs for users using a Windows desktop to connect. Putty is a well known and widely used program to connect to HPC facilities. MobaXterm is more recent but provides a complete solution for users who need to have access to a graphical user interface (GUI) when connected to HPC.
- The very basics on how to establish a connection to HPC
- How to add more specific options for the connection to HPC
- How to save the details and options for a connection for future use
MobaXterm coming soon.
Set 2: Transferring files and data
Users need a secure file transfer protocol (sftp) to transfer files from their desktop to HPC. FileZilla can be installed under Linux, Mac and Windows. It allows users to connect to HPC and then drag and drop files to and from the HPC to their desktop. MobaXterm offers sftp as well.
Filezilla-saved-session and MobaXterm-file-transfer coming soon.
Set 3: Basic Linux/Unix commands
These videos follow the content of Software Carpentry – The Unix Shell (https://swcarpentry.github.io/shell-novice/). Sessions 1, 2 and 3 provide instructions on the Linux/Unix commands needed (at the bare minimum level) for new users of HPC. The videos give an overview of the basic command. Further details and exercises with solutions can be found on the Software Carpentry – The Unix Shell page.
pwd – print name of current/working directory
ls – list directory contents
ls [OPTION] [PATH-TO-FILE or PATH-TO-DIRECTORY]
man – an interface to the system reference manuals
man [COMMAND]
2) Overview of how to move around the file system, change into other directories (cd).
cd – change the current directory
cd [PATH-TO-DIRECTORY]
cd on its own will change into the user’s home directory
cd .. will go one directory up from the current one
3) Explains the difference between an absolute and relative path.
A relative path specifies a location starting from the current location.
An absolute path specifies a location from the root of the file system.
4) Overview of how to create new directories and files. New files are created with the editor nano.
mkdir – creates a new directory.
mkdir [PATH-TO-DIRECTORY]
nano – Nano’s ANOther editor
nano [PATH-TO-FILE]
5) Overview of how to use the vi editor to edit files. Further commands within vi can be found here: http://www.atmos.albany.edu/daes/atmclasses/atm350/vi_cheat_sheet.pdf (PDF, 61.2 KB).
vi – powerful text editor
vi [PATH-TO-FILE]
i – insert mode
Esc – leave insert mode
:w – write/save file
:q – exit file
:wq – save and then exit file
:q! – override any changes and exit file without saving
6) Overview of file viewers available.
cat – prints the whole file content to screen.
cat [PATH-TO-FILE]
more – prints a file to screen page by page, press space bar for next page.
more [PATH-TO-FILE]
less – prints a file to screen page by page. Scrollable, use page-down and page-up or arrow keys to navigate through the file. Press q to quit.
less – [PATH-TO-FILE]
7) Overview of how to copy and move files and directories.
cp – copies a file or directory
cp [PATH-TO-OLD-FILE] [PATH-TO-NEW-FILE]
cp –r [PATH-TO-OLD-DIRECTOR] [PATH-TO-NEW-DIRECTORY]
mv – moves (renames) a file or directory
mv [PATH-TO-OLD-FILE] [PATH-TO-NEW-FILE]
mv –r [PATH-TO-OLD-DIRECTOR] [PATH-TO-NEW-DIRECTORY]
8) Overview of how to remove files and directories.
rm – removes (deletes) a file or directory. This is not reversible. Once deleted it is gone and cannot be recovered.
rm [PATH-TO-FILE]
rm –r [PATH-TO-DIRECTORY]
Set 4: HPC file systems and what users need to consider for appropriate and efficient usage
- General overview of different file systems that might be available on HPC. The video goes through shared file systems such as /home and /scratch, local compute node file systems (local scratch or $TMPDIR) and storage file system. It outlines what users need to consider if they wish to use any of these in their workflows.
- Overview of the different directories that might be present on HPC. These could include /home, /scratch, /opt, /lib and lib64, /sw and others.
- Overview of the Message-of-the-day file and message that is displayed to users every time they log in. This displays info about general help and often current problems or upcoming outages.
Set 5: Queuing system and commands
uptime — gives a one-line display of the following information: The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
2) Overview of how to monitor jobs.
qstat — display status of PBS batch jobs, queues, or servers
qstat -u [USERNAME] — shows all jobs owned by USERNAME
qstat -aw -u [USERNAME] — shows jobs in a more informative and wider format
qstat -aT -u [USERNAME] — shows estimated start time of queued jobs
qstat -a -1n -u [USERNAME] — shows the compute nodes used for running jobs
qstat -f [JobID] — shows full output for running jobs including the resources it is using.
qstat –xf [JobID] — shows full output for already finished jobs.
3) Overview of how to submit jobs.
The submission of an interactive job is shown first and then is translated to a batch job submission.
qsub —submit PBS job
qsub -I -A [ACCOUNT-STRING] -l select=1:ncpus=1:mem=5GB -l walltime=1:00:00
In the above line:
- qsub -I (qsub minus capital i) indicates that this is an interactive job
- -l (minus little L) indicates a resource request (in front of select and walltime)
- select=1 requests resources on a single compute node
- :ncpus=1 requests a single core on the compute node
- :mem=5GB requests 5GB of memory (ram) for the requested cores on the single node
- walltime=1:00:00 requests 1 hour, 00 minutes and 00 seconds of runtime.
To terminate the interactive job type exit on the command line.
To submit a batch job open a file and enter these lines:
#!/bin/bash
#PBS -A [ACCOUNT-STRING]
#PBS -N [JOBNAME]
#PBS -l select=1:ncpus=1:mem=5GB
#PBS -l walltime=1:00:00
[module commands go here]
[commands to run calculation go here]
If you have named the file test-file then you submit via qsub test-file.
4) Overview of how to delete a job from the queue.
qdel — deletes PBS jobs
qdel [JobID]
Set 6: Software — how to use modules
Overview of using software modules to use HPC installed software.
module – access to software modules
module avail – shows all available modules
module avail [SOFTWARE-NAME] — shows all modules for SOFTWARE-NAME
module load [SOFTWARE-NAME/VERSION] — loads a specific software version
module unload [SOFTWARE-NAME/VERSION] — unloads a specific software version
module list – lists all currently loaded software modules
module purge – unloads ALL currently loaded software modules
Set 7: The University of Queensland Research Data Manager (RDM) storage
RDM website: https://rdm.uq.edu.au
User Guides:
- https://rdm.uq.edu.au/resources
- https://guides.library.uq.edu.au/for-researchers/uq-research-data-manager
- How to access via HPC
- Instructions for institutes (access from in-house clusters and in-house desktops and laptops)
How-to videos:
1) General overview of “Q” RDM
This video shows how it is essential that users are aware of from where they are accessing their “Q” RDM. For access from the HPC clusters contact rcc-support@uq.edu.au. For access from desktops/laptops please check the instructions for individual institutes (see link above) if applicable or contact RDM-support via the https://rdm.uq.edu.au portal.
Note: “A” and “I” RDM are not covered. Please contact RDM support for help with “A” and “I” RDM. This can be done via https://rdm.uq.edu.au on the left-hand side.
2) General overview of how to access “Q” RDM from HPCs Tinaroo, FlashLite & Bunya
In this video, the problems and bottlenecks in incorporating a “Q” RDM in the data workflow of a HPC job are outlined for the user to consider carefully.