python: Python scripting language

*** DEPRECATED *** NOTE: The authors of Python have stopped maintenance of all Python2 versions as of 1 Jan 2020. While the UMD Division of Information Technology continues to provide access to the existing Python2 installs for now, all Python2 installations are ***DEPRECATED*** and will not be upgraded, have new extensions installed, etc. It is likely that Python2 will not be made available when the Zaratan cluster is stood up. All Python users are strongly encouraged to migrate to Python3.

Contents

1. Overview of package
   1. General usage
2. Availability of package by cluster
3. System modules
4. Installing modules
   1. Using setup.py
   2. Using virtual environments and pip
   3. Using conda
5. Assorted Tips and Tricks
   1. Matplotlib Tricks
6. Numba and GPU Support
7. Using python with MPI

Overview of package

General usage information

Python is a high-level scripting language.

This module will add the python and related commands to your path.

In case you need to link against this library in your code, the following environmental variables have been defined:

• \$PYTHON_ROOT has been set to the root of the python installation
• \$PYTHON_LIBDIR points to the directory containing the libraries
• \$PYTHON_INCDIR points to the directory containing the header files

You will probably wish to use these by adding the following flags to your compilation command (e.g. to CFLAGS in your Makefile):

• -I\$PYTHON_INCDIR

• -L\$PYTHON_LIBDIR -Wl,-rpath,\$PYTHON_LIBDIR

Available versions of the package python, by cluster

This section lists the available versions of the package pythonon the different clusters.

Available versions of python on the Zaratab cluster

*** DEPRECATED *** NOTE: The authors of Python have stopped maintenance of all Python2 versions as of 1 Jan 2020. While the UMD Division of Information Technology continues to provide access to the existing Python2 installs for now, all Python2 installations are ***DEPRECATED*** and will not be upgraded, have new extensions installed, etc. It is likely that Python2 will not be made available when the Zaratan cluster is stood up. All Python users are strongly encouraged to migrate to Python3.

When using in conjunction with your own code, you might wish to note the compiler and MPI libraries used when the python binaries and packages were built. MPI in particular can be fussy and generate strange errors if the different parts of the code are linked against different MPI libraries (even different versions of OpenMPI or the same version of OpenMPI built with a different compiler), or if the mpirun command used to start the code is from a different MPI version or was built with a different compiler. In general, it is best to ensure everything is built with the same compiler and, if used, the same MPI library.

System modules

The system installations of Python include a large number of Python modules to enhance the capability of Python. The table below list those modules and version numbers for the various installations of Python which are made available with the "module load python" or similar commands. (The ones which list a CUDA version are only available if the appropriate CUDA module was loaded before the python module was loaded.)

Packages/modules for Python

In addition, a number of other packages, C libraries, applications, etc. provide a Python API, etc. through a module. As these package are more than just Python modules, and indeed in many cases most users of the package are not accessing the Python modules they define, these packages have their own modulefiles. Generally, you can access these by loading their corresponding modulefiles before starting python (normally it should not matter whether you load the package module before or after loading python). These packages are:

Packages defining a Python Module

This section lists various software packages containing python modules which can be imported in your python code, after the corresponding environmental module is loaded.

Installing modules

Python's capabilities can be significantly enhanced through the addition of modules. Code can import a module to enable its functionality.

The supported python interpretters on the system have a selection of modules preinstalled. If a module you are interested in is not in that list, you can either install a personal copy of the module for yourself, or request that it be installed site wide. We will make reasonable efforts to accomodate such requests as staffing resources allow.

Installing modules yourself

1. Using setup.py
2. Using virtual environments and pip
3. Using conda

The mechanism for installing a module is of course dependent on the module being installed, but most modern python modules support the setup.py mechanism described below. But many packages will support installing via pip and virtual environments as well, and that is typically easier.

Installing python modules using setup.py

Note: Users might wish to look at Installing python modules using virtual environments first, as that is often easier.

The standard procedure for installing your own copy of a module is:

1. module load python/X.Y.Z to select the version of python you wish to use.
2. Create a directory to contain your python module, if not already done. Typically, you will want one directory to house all of the modules you are installing, so something like mkdir ~/.mypython will work. You should also create lib and lib/python directories beneath it, e.g. mkdir ~/.mypython/lib ~/.mypython/lib/python.
3. You will need to tell python where to look for your modules. Assuming you are putting your modules under ~/.mypython, something like setenv PYTHONPATH ~/.mypython/lib/python (bash/bourne shell users should do PYTHONPATH=~/.mypython/lib/python; export PYTHONPATH ). You probably want to add this to your .cshrc.mine or .bashrc.mine.
4. Download and unzip/untar/etc the module sources. Cd to the main module source directory (it should contain the file setup.py
5. Run python setup.py install --home ~/.mypython

If all goes well, the module should now be installed under ~/.mypython or wherever you specified. If there are executables associated with it, they should be in ~/.mypython/bin. You should be able to import the module in python now (this assumes that PYTHONPATH is set as indicated above).

Of course, not all modules install easily. Unfortunately, the install process can fail in far too many ways than can be reasonably enumerated. If you are comfortable with building modules, you might find reasonable guidance from error messages to assist you in getting the module to build, but it is probably easiest to just request the module be installed to the system libraries.

Installing modules yourself using virtual environments

Although the standard procedure described above works for most cases, there are cases where more separation is required. Python3 includes a venv module which allows you to create a fully independent virtual python environment, copying the python executables and standard and system libraries to your own directory, and allowing you to add/update/delete from there. This has the advantage that the virtualenv is almost completely isolated; so changes made in the system installation of python are unlikely to impact your virtualenv. This can be important if you have a code or application which requires e.g. version 1.6 of the foo package, but will break if it is upgraded to 1.7 (it appears that when using standard scheme above using PYTHONPATH, the system library directories are ALWAYS searched before PYTHONPATH, meaning that method can be used to add modules, but not to upgrade or downgrade modules).

However, the virtualenv takes up a significant amount of diskspace, and the isolation from the system python can be a negative as well as upgrades and/or new modules added to the system python will NOT be visible --- this is good when as in the example above it breaks something, but most of the time the upgrades are desirable.

To install a package with the virtualenv mechanism, you must first create a virtual python environment.

1. module load python/X.Y.Z to select the version of python you wish to use in this virtual environment.
2. You should select a directory where the virtual python environment should live. Each virtual environment you create will be a subdirectory of this directory. For the examples below, we will use the my-venv subdirectory of your home directory (e.g. ~/my-venv).
3. For python3, you can create a virtual env with either of the simple commands:
   1. python -mvenv --system-site-packages ~/my-venv: This variant will give the virtual environment access to system installed python packages, e.g. numpy, scipy and matplotlib. This is the easiest version, but as it is less isolated from the system python installation it can lead to problems if there are version compatibility issues.
   2. python -mvenv ~/my-venv: This variant will isolate the resulting environment from system packages. This is the safest approach, but may require you to install packages available on the system, and can be trickier in some cases.
4. For python2, you need to create the empty directory (mkdir ~/my-venv), and then unset the PYTHONHOME variable set by the module load command (i.e. unsetenv PYTHONHOME for the csh and tcsh shells, and unset PYTHONHOME for bash). Then issue either of the following commands:
   1. virtualenv --system-site-packages ~/my-venv
   2. virtualenv ~/my-venv
   The first version, with the --system-site-packages flag, behaves like the python3 version with the same flag --- the system packages are still available. The second version isolates your virutal environment from the system packages.
5. In order to use this virtual python (for either python version) environment, you must first activate it. This must be done in every process using the virtual environment. You do this by issue one of the following commands:
   1. source ~/venv/bin/activate.csh for csh or tcsh shells
   2. source ~/venv/bin/activate for bash shells
6. When you are finished with the environment, the command deactivate will deactivate the virtual environment.

Once the virtual environment is created and activated, installation is usually relatively simple using the pip command. You should just be able to do pip install NameOfPackage. Pip should take care of downloading the package and installing it for you.

Of course, not all modules install easily. Unfortunately, the install process can fail in far too many ways than can be reasonably enumerated. If you are comfortable with building modules, you might find reasonable guidance from error messages to assist you in getting the module to build, but it is probably easiest to just request the module be installed to the system libraries.

Installing modules yourself using conda

NOTE: We generally recommend that you try installing your own Python packages using one of the methods above, i.e. either using setup.py or using the venv module and virtual environments rather than using conda.

Conda is another option for installing python and other packages, but it does even more separation and isolation than the venv method. This can either be an advantage or a liability depending on the use case. This isolation means that you are not constrained but what system staff have installed on the cluster, but it also means it is very difficult if not impossible to take advantage of any packages system staff have installed on the cluster. Also, since you are relying on an entire environment which you installed, the ability of HPC staff to provide assistance should there be issues is limited. Therefore, we encourage users to try one of the other methods if such is feasible.

NOTE:The Anaconda distribution of Python and R is restrictively licensed. The Anaconda company states that the use of Anaconda's offerings at an organization of more than 200 employees requires a Business or Enterprise license. As the Univesity of Maryland does not have such a license, the use of the anaconda distribution or its repositories at UMD (outside of use in a formally scheduled curriculum based course) is not free and is not allowed, even though Anaconda is not currently blocking such accesses via technological means. I.e., the fact that your conda installation commands succeed does not mean that your use of the command is consistent with the Anaconda company's terms of use. See e.g. the Anaconda blog for more information regarding Anaconda licensing.

Due to the license restrictions above, we encourage users to use the miniforge/mambaforge installer which pulls packages from the conda-forge repository, which is not covered by the restrictive Anaconda license and is free to use. (You still may need to check for any This provides similar functionality to miniconda without the license restrictions.

Miniconda is a subset of Anaconda, without some extras like Anaconda Navigator, etc. By default, it pulls packages from the default Anaconda repositories, which requires licensing from Anaconda. The actual conda program which installs packages is free to use, but only if you are downloading packages from repositories which are free to use. If you insist on using miniconda, please be sure to remove the default channel (as well as any subchannels of the default channel) as these are restrictively licensed and require licesnses, and replace it with conda-forge channel, e.g.

conda config --show channels
conda config --remove channels defaults
# You might also need to remove subchannels like ananconda, main, r, pro, etc

conda config --add channels conda-forge

But we encourage you to use miniforge/mambaforge instead.

To install python packages using conda, you should usually follow a process like outlined below:

1. If you have not already done so, install a copy of mambaforge/miniforge or similar, as follows:
   1. Choose a location for your conda environments. Because conda environments potentially can get quite large, a subdirectory of your scratch directory is usually best. E.g. ~/scratch/condaroot. Create that directory if it does not already exist.
   2. Download Mambaforge from https://conda-forge.org/miniforge into your condaroot directory.
   3. Make a copy of your initialization dot files, e.g. cp ~/.bashrc ~/.bashrc.HOLD. You might wish to do similar for .cshrc and other init dot files if they exist.
   4. Install miniforge/mambaforge, cd ~/scratch/condaroot; bash Mambaforge-X.Y.Z-A-Linux-s86_64.sh -p mambaforge-X.Y.z This will install mambaforge to the mabaforge-X.Y.Z subdirectory of the directory you ran the script from (~/scratch/condaroot). The script will ask some questions, ansering yes to all should generally work.
   5. The script will have updated your ~/.bashrc and possibly some other dot files. Although you can leave things like that, in which case conda will be initialized in every shell whether it needs it or not (and potentially causing delays in shell startup), we recommend that you copy the modifications made to the initialization scripts to a conda specific script (e.g. cp ~/.bashrc ~/bashrc.conda; cp ~/.bashrc.HOLD ~/.bashrc and edit the new ~/bashrc.conda to remove the lines before the # >>> conda initialize >>> line. You can then source this file whenever you wish to use conda.
2. Initialize conda if needed. If you followed the instructions to remove the conda initialization from your initialization dot files, you should initialize conda with something like source ~/bashrc.conda.
3. If you do not already have a conda environment you are working in, create one. You should generally have a separate environment for each project and/or major software package you intend to use. To create an environment, do something like: cd ~/scratch/condaroot; conda create -n myenv This will create a conda environment myenv; we recommend you choose a more meaningful name.
4. Activate your conda environment. E.g. conda activate myenv, replacing myenv with the name of the environment to activate.
5. If this is a new miniforge environment, you will need to install pip. Do conda install pip.
6. If this is a previously created package, or a miniforge environment, you should update pip. Do pip install --upgrade pip
7. Install your python packages. The exact procedure may vary a bit depending on the package, but usually it is something like pip install PACKAGE_NAMNE.

Assorted Tips and Tricks

Matplotlib Tricks

• Using matplotlib in batch jobs/without an X server: By default, the matplotlib package in Python expects to work with a graphical user interface (GUI), which on Unix-like systems means an X server running. This can be problematic if one wishes to use matplotlib in batch jobs (e.g. on an HPC cluster) because typically a display will not be available. The easiest way to do this is to specify a non-interactive backend. There are several ways to do this, but since you probably want to continue using an interactive backend when using python interactively, the best approach is to have your batch code select a non-interactive backend. A common choice for such is Agg (for Anti-Grain Geometry engine) which can produce PNG files, Cairo and Gdk are other options. Use would be something like:

          import matplotlib
          # This needs to be done *before* importing pyplot or pylab
          matplotlib.use('Agg')
          import matplotlib.pyplot as plt
  
          #Do your plotting, e.g.
          fig = plt.figure()
          ax = fig.add_subplot(111)
          ax.plot(range(10))
          fig.savefig('test.png')

  For more information, see: Matplotlib Documentation on running without a GUI

from numba import jit
@jit
def sum1d(my_array):
        total = 0.0
        for i in range(my_array.shape[0])
                total += my_array[i]
        return total

#!/bin/bash
#Assume will be finished in no more than 8 hours
#SBATCH -t 8:00:00
#Launch on 40 cores distributed over as many nodes as needed
#SBATCH -n 40
#Assume need 6 GB/core (6144 MB/core)
#SBATCH --mem-per-core=6144

#Make sure module cmd gets defined
. ~/.profile

#Load required modules
module load python/3.5.1
#Load correct gcc (4.9.3) and mpi (openmpi/1.8.6) for python/3.5.1
module load gcc/4.9.3
module load openmpi/1.8.6

#Normally do not need to give -n 40, as openmpi will determine from Slurm
#environment variables
mpirun mp-mpi4py-script.py