Available Software

Contents

1. The module command
   1. The module command
   2. Library packages
   3. Common errors
2. Software packages available
   1. Unclassified by field
   2. Classified by field, etc
      1. Packages related to Aeronautics
      2. Packages related to CFD
      3. Packages related to ComputerVision
      4. Packages related to DataCompression
      5. Packages related to DataFormat
      6. Packages related to Genomics
      7. Packages related to MachineLearning
      8. Packages related to Meteorology
      9. Packages related to NumericalMethods
      10. Packages related to SoftwareDevelopment
      11. Packages related to Utilities
      12. Packages related to Visualization
   3. Available by cluster
      1. All packages available on the Deepthought2 cluster (RHEL8)
      2. All packages available on the Juggernaut cluster
3. Packages for scripting languages
   1. Packages/modules for Python
   2. Packages/modules for R
   3. Packages/modules for Perl

The module command

A large number of software packages are available on each of the clusters, and for some of them, multiple versions of the same package. Not all of them like to co-exist with each other, so many of the packages are not available by default. You need to tell the system you want to use them with the module command.

For example, matlab is available on the HPC clusters, but only after you module load it. E.g.

login-1: matlab
matlab: Command not found.
login-1: module load matlab
login-1: module help matlab

----------- Module Specific Help for 'matlab/2014b' ---------------


    This is Matlab Release 2014b

    Releases after 2009 require Red Hat 5 or later.
    Releases after 2013 require Red Hat 6 or later.
    Matlab 2009b is the last release supported under Solaris.

    Run command 'matlab' to start up the program,
    or 'matlab -h' to see various command-line options.

    ***  To see all releases, issue the command
	module avail matlab

login-1: matlab
                              < M A T L A B >
                  Copyright 1984-2014 The MathWorks, Inc.
                   R2014b (8.4.0.150421) 64-bit (glnxa64)
                             September 15, 2014


  To get started, type one of these: helpwin, helpdesk, or demo.
  For product information, visit www.mathworks.com.

>> 

You can use the module help and module whatis commands before (or without) loading the package.

The module command

The module command associates a tag with the various packages. It has a hierarchical nature, so we have tags like matlab/2013b and matlab/2011a in addition to plain old matlab. The former specify a specific version of matlab, whereas the latter will enable the default version of matlab.

With the roleout of a new software library with the RHEL8 upgrade of the Deepthought2 HPC cluster, we are no longer keeping the "default" versions of software packages fixed over long periods of time. Instead, if you do not provide a version in your "module load" command, you will almost always get the latest installed version compatible with other modules you have loaded. (There will be rare and brief exceptions if we find problems in the latest version of a package). This is a change from the behavior of many packages on the RHEL6 version of Deepthought2. For reproducibility over time, it is strongly recommended that in job submission batch scripts you provide the full version specification for any packages for which you care about the version number.

For many packages, the tag is basically just the base name of the package, followed by a slash (/) and a version number. E.g., the examples matlab/2013b and matlab/2011a refer to versions 2013b and 2011a, respectively, of matlab. You can also just use matlab as the tag when e.g. loading a module, in which case you will get one of the versions (usually the latest, sometimes we pin to an earlier version if the newest version is still being tested).

Many packages compiled from source will have different builds depending on what CPU microarchitecture they were compiled for. Running a code compiled for a more modern CPU than you are currently using will likely result in the code crashing; running a code compiled for a less modern CPU will generally work, but you might lose some of the optimizations available on the newer chip. Generally, these packages will have two equivalent module tags, one of the form "foo/VERSION/ARCHITECTURE" and one of the form "foo/ARCHITECTURE/VERSION"; both forms load foo version VERSION built for architecture ARCHITECTURE. The module load command will attempt to load the build for the most advanced architecture compatible with the CPU you are running on. If you omit the version, it will look for the latest version built for the most advanced CPU architecture compatible with the CPU you are running on.

Some packages are fussy about the compiler and/or MPI libraries they are used with. This is particularly true of libraries which have fortran90 and/or C++ components; some examples are openmpi, hdf5, and netcdf. In these cases, the package will again have multiple module tags referring to the same build of the software. The first starts like the simple case, with the base name of the package followed by the version of the package, but to this is appended the other dependencies (like compiler family and version, and perhaps MPI library family and version, etc.). For example, one tag for version 1.8.13 of hdf5, built with version 4.8.1 of the GNU compiler suite and version 1.8.1 of , with shared linkage would be:

hdf5/1.8.13/gnu/4.8.1/openmpi/1.8.1/shared  

There is also an alternative form giving the compiler, MPI library, and linkage portions of the tag immediately after the base package name, and ending with the version of the package, e.g.

hdf5/gnu/4.8.1/openmpi/1.8.1/shared/1.8.13

Both of these tags do exactly the same thing, and set up exactly the same build of hdf5, when loaded. You can use either.

The advantage of the module command is that you generally do NOT need to give these long tag names, but can leave off components on the right and it will try to do the right thing. In particular, if you try loading just the base package name, e.g. module load hdf5, the system will determine if you have previously loaded a compiler and/or MPI library. If so, then the system will see if there are any builds of any version of the package matching the compiler and MPI library versions you previously loaded, and if one is found load the build for the most recent version that matches. If no matching build is found, and error will be displayed and no module loaded.

The following except gives an example of this:

login-2:> module purge 
login-2:> module list 
No Modulefiles Currently Loaded.
login-2:> module load gcc/4.8.1
login-2:> module load openmpi/1.8.1
login-2:> module load hdf5
login-2:> module list
Currently Loaded Modulefiles:
  1) gcc/4.8.1(default)
  2) openmpi/1.8.1/gnu/4.8.1
  3) hdf5/gnu/4.8.1/openmpi/1.8.1/shared/1.8.13
login-2:> 
login-2:> module purge
login-2:> module list
No Modulefiles Currently Loaded.
login-2:> module load intel/2016
login-2:> module load openmpi/1.10.2
login-2:> module load hdf5
login-2:> module list
Currently Loaded Modulefiles:
  1) ofed/3.12(default)
  2) intel/2016.3.210(default)
  3) openmpi/1.10.2/intel/2016.3.210
  4) hdf5/intel/2016.3.210/openmpi/1.10.2/shared/1.10.0
login-2:> 
login-2:> module purge
login-2:> module list
No Modulefiles Currently Loaded.
login-2:> module load hdf5/1.8.15p1/sunstudio/12.4/openmpi/1.8.6/shared
login-2:> module list
Currently Loaded Modulefiles:
  1) sunstudio/12.4(default)
  2) openmpi/1.8.6/sunstudio/12.4
  3) hdf5/1.8.15p1/sunstudio/12.4/openmpi/1.8.6/shared

We start by issuing a module purge command to delete any previously loaded modules (in general, you should use this with caution. If you do use it, you should immediately load the modules hpcc/deepthought or hpcc/deepthought2, as appropriate for the cluster you are on, and slurm. These are the default modules on the Deepthought HPC clusters and things might not work properly without them). We then do a module list to confirm that there are no modules loaded. We then load a specific version of the gcc compiler (4.8.1) and the openmpi libraries (1.8.1) and then ask the system to load the best hdf5 module. We then list what was loaded, and note that the system loaded an openmpi build (version 1.8.1) for the compiler we loaded, and a hdf5 version (1.8.13) for the previously loaded compiler and MPI library.

In the second example above, we load the 2016 version of the intel compiler suite, and the 1.10.2 version of openmpi, and when we issue the simple module load hdf5, this time it loads hdf5 version 1.10.0 built for that compiler and openmpi library. (Note also that the openmpi library loaded was also built for the correct compiler.)

In the final example above, we do not explicitly load any compiler or MPI library, but instead give a full path the the hdf5 module, e.g. hdf5/1.8.15p1/sunstudio/12.4/openmpi/1.8.6/shared. In this case, we note that not only was the requested hdf5 module loaded, but we also loaded the corresponding compiler (sunstudio/12.4) and openmpi (openmpi/1.8.6) modules as well.

In summary, in almost all cases, you need only specify a portion of the tag, and the system will try to default it properly. Generally, you should load the compiler you wish to use first (or do not load anything if you want to use the default gcc 4.6.1 compiler) followed by the MPI library if needed. This will allow the system to pick the correct build of libraries which depend on such. Alternatively, if you specify the full path to a package which depends on a specific compiler and MPI library, these will be loaded for you if not already loaded. In general, if the package has dependencies, they will be loaded automatically for you if needed.

However, if you want reproducable results, it is generally recommended that you fully specify version numbers when loading modules in your job scripts. I.e., the latest version of the foo package might be 1.5 when you run your job today, but if you run it again a month from today, the latest version might by 1.7 at that time, and that could change the results unexpectedly if your script merely did a module load foo. If you wish to ensure the same version is always used, specifying the version number in the module load command is advised, e.g. module load foo/1.5; this way you will get the same version of the foo package both today and a month from now. But if you always prefer to use the latest package available, then in general you should leave the version numbers off.

NOTE: If you are submitting a job using a bourne style shell (e.g. sh or bash), your dot files will not be read automatically if your default shell is something else (e.g. csh or tcsh). So you will need to include a . ~/.profile line near the top of your script, after the #SBATCH lines but before you try to use tap or module commands.

The module command allows you to issue commands of the form module SUBCOMMAND OPTIONAL_ARGUMENT. Recognized subcommands include:

• list: This will list the packages you currently have loaded. Note that if a tap entry was migrated to modules, it will show up here if you loaded it using tap.
• avail OPTIONAL_TAG: this will list all the packages available. If a tag is given, it will list all packages which match the tag. Note the matching is purely lexical, so module avail netcdf will pick up all netcdf modules, as well as netcdf-fortran and netcdf-cxx modules. To restrict to just netcdf modules, use something like module avail netcdf/ (note the final /)
• load TAG: this is the most common usage, and it sets up the environment for the application associated with TAG. Again, if you do not specify the full tag, the system will attempt to default it, and usually it can do that with some success. Most of our application definitions will pull in any dependencies as well. Note that unlike tap, this is a silent operation when it succeeds.
• help TAG: This displays a brief intro message about the package. Generally the same message you see when you tap a package. It is advised that you read this once for each package you use.
• unload TAG: This unloads the package; i.e. all the environment that was set up for the package is returned to what it was before you loaded it. This is not 100% successful all the time, but for the most part works. You generally do not need this unless you which to change the version of an application you are using.
• purge: This unloads ALL currently loaded modules. This should be used sparingly. If you do use it, please follow it with the loading of either hpcc/deepthought or hpcc/deepthought2 (as appropriate for the cluster you are on) and slurm. These modules are loaded for you automatically when you log into the clusters, and if you remove them (e.g. with the purge command), things might not work as expected.

Please note that the HPC compute nodes have abridged copies of the software directories available on other Glue/TerpConnect systems. The HPC login nodes have the full software directories. Please look at software lists below, and in particular click on the software package name for the full breakdown by version number to see which, if any, HPC clusters a specific version of the package is on.

Library packages

Some packages are not so much applications as libraries; i.e. they provide libraries which your code can use, and might or might not have any binaries that you can run directly. To make use of these libraries in your code takes a little bit of work. Please see the section on compiling codes for more information on how to do this.

Common errors from module/tap commands

As stated early, not all packages co-exist nicely. This is particularly true of different versions of the same package, or related libraries (e.g. the various MPI libraries). The module command will generate an error if you try to load a package which does not play well with packages you have previously loaded; if this occurs you will need to unload the previous package and reload the new package. E.g.

login-1: module load matlab/2012b
login-1:~: module load matlab/2013b
matlab/2013b(8):ERROR:150: Module 'matlab/2013b' conflicts with the currently loaded module(s) 'matlab/2012b'
matlab/2013b(8):ERROR:102: Tcl command execution failed: source $moduledir/common
login-1: module unload matlab
login-1:~: module load matlab/2013b

Although not as user friendly as one would like, you can see that the first time I tried to load matlab/2013b it complained because matlab/2012b was already loaded. Also note that I did not need to specify the full tag to unload matlab/2012b, as there was only one matlab package loaded, so it could default it.

You can also run into similar incompatibillity problems if you mix packages (especially library packages) built for one compiler/MPI library/etc with a different compiler/MPI library/etc. For example

login-1: module avail fftw/3.3.3

------------------- /cell_root/system/common/modulefiles/sys -------------------
fftw/3.3.3/gnu/4.6.1/openmpi/1.6/shared
fftw/3.3.3/gnu/4.6.1/openmpi/1.6/static
fftw/3.3.3/gnu/4.6.1/openmpi/1.6.5/shared
fftw/3.3.3/gnu/4.6.1/openmpi/1.6.5/static
fftw/3.3.3/intel/2013.1.039/openmpi/1.6.5/shared
fftw/3.3.3/intel/2013.1.039/openmpi/1.6.5/static

login-1: module load intel/2013.1.039

login-1: module load fftw/3.3.3/gnu

***** Compiler mismatch

You have loaded compiler intel/2013.1.039
This package wants compiler gnu/4.6.1
login-1: 

In the above, we show that there are various builds of fftw/3.3.3 available. We then load the intel compiler module, and try to load a GNU compiler build of fftw/3.3.3, and get an error. That is because using a GNU build of fftw/3.3.3 with the intel compiler suite is likely not to work and cause you headaches. In general, you should not be specifying more than fftw/3.3.3 anyway, and the module command will figure out which build to load based on which compilers you already have loaded. Alternatively, you can give the full path (e.g. module load fftw/3.3.3/intel/2013.1.039/openmpi/1.6.5) and the module command will auto-load the appropriate intel compiler and openmpi module for if, if it can. If you have a conflicting module already loaded, you will still get the error.

For the most part, tap is just a wrapper around the module command for backwards compatibility, and so you will get the same errors with tap if you try to load conflicting modules. The tap command is limited, so you generally do NOT get a whole lot of choice as to which builds of packages get loaded, so it is less likely to be a problem, but it can still arise. Some packages have not been migrated from tap to module command, and these will NOT generate such errors when you tap incompatible modules. But things will probably NOT work as you expected; the incompatibility is REAL, not an artifact of the module command.

We are currently in the process of integrating the software documentation for RHEL6 and RHEL8. This page primarily shows information for packages in the RHEL8 software library; information on the RHEL6 software library can be found at the old RHEL6 software documentation pages. We will gradually be adding the RHEL6 content under here, and in particular will be migrating (and updating as needed) the package specific documentation here. But for now, you will likely want to look at the old usage documentation (and module help PACKAGENAME).

List of available software packages

In this section we list the various software packages available on the UMD HPC clusters. We start with a list of all packages, unclassified by field of research, and follow with some shorter lists for various fields of research. If there is a mistake in the classification of software (e.g. a package is listed in the wrong category, or is missing from a category where it belongs), please contact systems staff so that we may correct the situation. Similarly, contact us if there is a category for software which is missing and you believe would be useful to have.

For each list, you can click on the name of the package to bring up a page with more detailed information on what versions are available on which clusters, and some basic (and sometimes not so basic) usage and help.

1. All packages (unsorted by field)
2. Software lists classified by field of research, etc
   1. Packages related to Aeronautics
   2. Packages related to CFD
   3. Packages related to ComputerVision
   4. Packages related to DataCompression
   5. Packages related to DataFormat
   6. Packages related to Genomics
   7. Packages related to MachineLearning
   8. Packages related to Meteorology
   9. Packages related to NumericalMethods
   10. Packages related to SoftwareDevelopment
   11. Packages related to Utilities
   12. Packages related to Visualization
3. Software lists by cluster
   1. All packages available on the Deepthought2 cluster (RHEL8)
   2. All packages available on the Juggernaut cluster

All packages (unsorted by field)

This section list all installed research software, unsorted by field of research/category/etc.

Software packages related to Aeronautics

This section lists various software packages related to Aeronautics, Aviation, Aerospace Sciences and related fields (Astronautics, Hypersonics, Aerothermodynamics, etc.)

Software packages related to CFD

This section lists various software packages related to Computational Fluid Dynamics (CFD).

Software packages related to ComputerVision

This section lists various software packages related to Computer Vision and related fields. You might also want to look at the section on packages related to Machine Learning.

Software packages related to DataCompression

This section lists various software packages related to Data Compression. Basically libraries and binaries for compressing and uncompressing data to/from various compressed formats and/or compression algorithms.

Software packages related to DataFormat

This section lists various software packages related to various scientific or other data formats, including libraries and binaries for reading, writing, and manipulating data in specific formats.

Software packages related to Genomics

This section lists various software packages related to Genomics, Genetics, and related fields.

Software packages related to MachineLearning

This section lists various software packages related to Machine Learning, Deep Learning, Neural Nets and related fields. You might also wish to look at the section on Computer Vision

Software packages related to Meteorology

This section lists various software packages related to Meteorology, Weather, Atmospheric and Oceanographic Sciences and related fields.

Software packages related to NumericalMethods

This section lists various software packages related to Numerical Methods and Computational Mathematics. It includes both underlying numerical libraries as well as advanced computational (and often GUI) environments. These packages are generally useful across a wide range of disciplines.

Software packages related to SoftwareDevelopment

This section lists various software packages related to the development and optimization, etc. of software and codes. It includes compilers, debuggers, profiling tools, as well as relatively low level libraries. It also includes various scripting languages.

Software packages related to Utilities

This section lists various utility software packages, etc. This is sort of a catch-all category for packages which are useful across a wide range of disciplines but do not really fall into any discipline. These are often just newer versions of utilities supplied by the OS.

Software packages related to Visualization

This section lists various software packages related to Visualization; basically packages to assist in visualizing data generated from other packages/codes. It includes both basic graphics libraries and high end (GUI) visualization platforms.

All packages available on the Deepthought2 cluster (RHEL8)

This section lists all software packages available on the Deepthought2 cluster (RHEL8)

All packages available on the Juggernaut cluster

This section lists all software packages available on the Juggernaut cluster

Packages installed for scripting languages

We are currently in the process of integrating the software documentation for RHEL6 and RHEL8. This page primarily shows information for packages in the RHEL8 software library; information on the RHEL6 software library can be found at the old RHEL6 software documentation pages. We will gradually be adding the RHEL6 content under here, and in particular will be migrating (and updating as needed) the package specific documentation here. But for now, you will likely want to look at the old usage documentation (and module help PACKAGENAME).

The Division of Information Technology maintains several versions of several scripting languages on the clusters. These scripting languages can be enhanced by the addition of packages/modules/etc, and the Division of IT maintains a collection of packages for these. In most cases, the packages/modules are fairly lightweight, and we simply install them into the standard system directory location for the particular scripting language, upgrading them as needed. This means that they will just be found when you import/use/etc them; no additional work is needed on your part.

For a few cases, they are major packages in their own right, and we give them their own module. Those packages are included in the lists of software earlier in this document, and you will need to "module load" the appropriate package in order to use them.

This section is for the packages of the first type, and for the convenience of our users we provide a list of the packages/modules available for each install of a scripting language, by cluster and version. Note: that these lists do not include the standard library for the scripting language, i.e. those packages/modules that get included automatically in the installation of the scripting language, only those that need to be installed separately.

These sections are still under development. We hope to have it more accurate in the near future.

1. Modules/packages for Python
2. Modules/packages for R
3. Modules/packages for Perl

Packages/modules for Python by cluster, version

Extensions for Python version 3.7.7 (built for gcc@8.4.0 on Deepthought2

Extensions for Python version 3.7.7 (built for gcc@7.4.0 on Deepthought2

Extensions for Python version 2.7.16 (built for gcc@8.4.0 on Deepthought2

Packages/modules for R by cluster, version

Packages/modules for Perl by cluster, version