Spack usage guide#

In order to use the compute nodes for training your models, start interactive sessions or use the remote debugging on pycharm, you may want to use some system libraries specific to your application. On RHPC, this is managed by a package manager called spack. This article will guide you to properly load the system packages you want to use while using the compute nodes.

For illustration, we will consider three user-specific system packages pyvips, openslide and pixman, and CUDA drivers which are generally necessary for everyone.

An important environment variable#

On any system, the environment variable LD_LIBRARY_PATH is important. This is so that you know the exact paths to each of the system packages you need the computer to use while running your jobs. Usually, we set this variable in the ~/.bashrc file with the paths to relevant packages installed within the system. With spack, we let the package manager automatically take care of this for us.

Before we start loading packages using spack, let us unset the LD_LIBRARY_PATH just to be sure. Although, this is not a necessary step.

unset LD_LIBRARY_PATH

Now, let us look at how to load some important system packages.

Loading the right CUDA drivers#

RHPC has different GPU nodes with different architectures. It is important to know what version of CUDA is necessary to use a particular node. This can be found in the GPU nodes table.

To illustrate an example, let us consider an interactive session on aristarchus. Before loading this system library, we must first perform a search of all the CUDA libraries installed. Use the following in your bash terminal:

spack find cuda

Aristarchus has eight A6000 cards compatible with CUDA versions higher than 11. So, we can choose any one of the libraries in principle but be mindful of other python libraries you may be using that is dependent on the CUDA drivers (for example, pytorch) and choose the right one.

To load your preferred CUDA driver, use:

spack load cuda@version

Now, let’s look at the LD_LIBRARY_PATH environment variable to confirm if these changes have been properly made.

echo $LD_LIBRARY_PATH

Along with this, you can also load cudnn in a similar manner.

spack find cudnn
spack load cudnn@version

Loading user-specific system packages#

If you want to use dlup, you need specific versions of openslide, pixman and pyvips. Without these, your implementations won’t work.

We repeat the steps described above to find the available packages and load them:

spack load openslide@3.4.1
spack load pixman@0.40.0
spack load libvips@8.13.0

Note, however, that if you wish to use the aiforoncology fork of openslide so be able to read .mrxs files from the 3dhistech scanner, you need to load pixman and libvips first, and the openslide-aifo@3.4.1-nki package last. It seems that otherwise libvips overwrites the openslide aifo fork with the normal installation.

# Install system dependencies
spack load pixman@0.40.0
spack load libvips@8.13.0
spack load openslide-aifo@3.4.1-nki

Resetting the python interpreter#

spack offers a default python interpreter which may not be useful for you in all scenarios. If you have a particular python interpreter in a separate conda environment set up with special python libraries, you should first select it before continuing with your job submissions. For this, first do the following:

spack unload py-pip py-wheel py-setuptools python

then, activate your conda environment normally and verify the interpreter path.

conda activate <env_name>
which python

The path to the python interpreter inside your environment should be printed on the console.

Setting the library paths in the pycharm remote debugger#

If you’re someone like me, then you probably use a remote debugger for all the dirty work with your code. To debug remotely with the right system packages, load all the necessary packages described above and copy the contents in the LD_LIBRARY_PATH variable.

Paste this into the environment variables field in the pycharm debugger settings.

If you have done this, then congratulations! you have configured everything correctly! Good luck with training your models and debugging your code!

Note: Sometimes it helps to do the spack loading and unsetting, but not having the environment variable in pycharm