eResearch Tips

Installing software locally on a HPC system

By Dr Minh Dinh, QCIF eResearch Analyst at UQ

Software, both commercial and open source, are often prebuilt and installed system-wide on HPC systems. They can be accessed using the module load command, which updates the system setting to expose the path to the executables and to include any library dependency. However, users often find themselves needing to build and install software, from source code, for their work.

This is where things could get very messy because one needs to:

  1. understand the configuration, compilation, and installation process
  2. use the right compiler and compilation flags
  3. make sure dependencies are correctly installed (likely to be local installation again), and correctly expose them for the compilation and linking process.

This document highlights several important steps to build and install software, from source code, locally on a HPC system.


Setting your environment

Environment variables are named string variables. For example, ‘HOME=/home/uqmdinh1’. One can get the value of an environment variable using ‘$’. For example, type echo $HOME, or echo $SHELL.
 
Important environment variables for the software installation process are:

  • $PATH: contains a colon-separated set of directories, which specifies a search for executable files in response to commands issued by a user.
  • $LD_LIBRARY_PATH: contains a colon-separated set of directories, which specifies where to search for the dynamic shared libraries and if an application was linked against it.

Once a dependent software or shared library is installed by the user, it might be necessary to update these environment variables as part of compiling and installing the target software. DO NOT make changes to these variables in $HOME/.bashrc or $HOME/.bash_profile scripts.


Configure your build

The target software often comes with a configure script. This text-based executable specifies the process to check for compiler availability and usage, software dependencies, and to generate a compilation script (A.K.A. the Makefile). The configure script can take a few parameters and flags. The installation directory is ‘/usr/local’ (by default). Therefore, to install the software locally, you NEED TO change the installation path to a writable directory by using the --prefix option.
 
./configure  --prefix=/home/<user-id>/<path-to-install-dir>
 
To see more options, for example to specify the location of a locally-installed shared library or to enable or disable a certain component of the target software, type:
 
./configure --help

Also, look into the log file (config.log). You can also save, configure and make an output using the tee command:
 
./configure  --prefix=/home/<user-id>/<path-to-install-dir> | tee <outputfile>


Compiling source code

To compile the source code, you need to know which compiler to use and what compilation flags to include. Details can often be found in the README or INSTALL text files that come with the software.

HPC compilers

Vendor C/C++  Fortran Potentially located at
GNU (Open source) gcc/g++ gfortran /usr/bin
Intel icc/icpc ifort /opt/intel/<version>/bin
PGI Pgcc/pgc++ pgfortran /opt/pgi/<version>/bin
OpenMPI mpicc/mpicxx/mpic++ mpifort

/opt/openmpi/intel/<version>/bin

Compilation

Makefile gets created at the source directory once the configuration gets through. Now, you can issue make command. To install the binaries and libraries at the location prefixed by configure, issue the make install command.
 
Software can be compiled at different optimisation levels (using –O flag), for debugging purposes (using –g flag), or if it needs to be linked to particular shared libraries (using –L flag). These flags are often pre-defined and generated by running the configure script. However, you might need to update some flags according to your needs.
 
A typical compile command for C code (as below) can be found in a Makefile file.
 
cc ${CCFLAGS} src.c ${CPPFLAGS} ${LDFLAGS} -o binary_name

Flag name What is it? Example
CCFLAGS Flags to pass the C compiler CCFLAGS='-g -O1'
CPPFLAGS Where the C pre-processor can find include (.h) files CPPFLAGS=-I/usr/lib/ /hdf5/include
LDFLAGS Which libraries (.so, .a files) to use and where the linker can find them

LDFLAGS=-L/usr/lib/hdf5/lib


Common issues

Type of problem What does it mean How to fix it
  • Permission denied

  • Read-only file system

You don’t have access permission to the installation directory.


Install to a location where you have ‘write permission’

./configure --prefix=path/to/dir

  • Library (.so or .a) file not found

  • Undefined reference to...

  • Cannot find shared object file

Your program uses shared libraries that the installer can’t find.

Set your library search path

LDFLAGS=-L/path/to/lib./configure ...

  • Header (.h) files not found

  • Undefined reference to...

  • Checking for <name>.h... no

  • <name>.h: No such file or directory 

Your program uses headers (.h files) that the installer can’t find.

Set your “include” path

CPPFLAGS= -I/path/to/include./configure ...