Using the HPC clusters: The Basics

This page covers some preliminary information about using the environment on the HPC clusters. I.e., the stuff you need to know how to do before we can even start talking about submitting jobs to the system using the command line.

Note: Users who are uncomfortable or unfamiliar with the Unix/Linux command line should consider using the OnDemand Web portal for accessing the cluster. While the command line is the more powerful interface to the cluster, the OnDemand portal has a gentler learning curve. The OnDemand portal is only currently available for the Deepthought2 and Juggernaut clusters.

  1. Logging into the system
    1. Common ssh warning/error messages
    2. Ssh Fingerprints
    3. Setting up ssh public-key authentication
      1. Setting up public-key authentication for passwordless ssh among the nodes of the HPC cluster.
      2. Setting up public-key authentication for passwordless ssh into the HPC cluster from your workstation, etc.
  2. Graphics
  3. Basic unix commands
    1. Changing your default shell
  4. Setting up your environment
    1. Preventing output from your dot files
  5. Files and storage basics
  6. Transferring files to/from the HPCCs and other systems
    1. Using scp/sftp
    2. Using Globus
    3. To/from Cloud Storage Providers
  7. Compiling codes
    1. Optimization of Code
    2. Compiling with OpenMP

Logging into the system

Each cluster has at least two nodes available for users to log into. From from these login nodes, you can submit and monitor your jobs, compile codes, look at the results of jobs, etc. On the two Deepthought clusters, these nodes can also be used for transferring files and data to/from the clusters and other systems.. For the MARCC/Bluecrab cluster, there are data transfer nodes (listed below which should be used instead of the login nodes for data transfers).

DO NOT RUN computationally intensive processes on the login nodes!!!. These are in violation of policy, interfere with other users of the clusters, and will be killed without warning. Repeated offenses can lead to suspension of your privilege to use the clusters.

For most tasks you will wish to accomplish, you will start by logging into one of the login nodes for the appropriate cluster. To log into the cluster, you need to use the Secure Shell protocol (SSH). This is usually standardly installed as ssh on Unix systems, and clients are available for Windows, Mac, and even Android, however on non-Unix systems you typically must install an SSH client .

Once you have an ssh client is installed on your system, you just tell it you wish to connect to the one of the login nodes for the cluster desired. Assuming your official UMD email address is,

Cluster Login Node Username
Deepthought johndoe
Deepthought2 johndoe
Login nodes
Data transfer nodes

Note that for MARCC/Bluecrab, you must append to your username (the is actually part of your username on that cluster).

Note that the MARCC/Bluecrab cluster has both login nodes and data transfer nodes. The data transfer nodes should only be used for transferring large amounts of data to or from the cluster.

NOTE: In or about Jan 2016, the MARCC staff will start requiring multi-factor authentication to access the Bluecrab cluster. You will need to download an app to your smart-phone (the phone you registered with MARCC staff when requesting an account on their web page) to generate an one-time passcode to be used in addition to your normal password. For more information, visit the MARCC web site for more information.

On an TerpConnect/Glue system, you would just issue the command ssh to connect to Deepthought, or similarly ssh to connect to Deepthought2. The unix ssh command by default assumes your login name on the remote system is the same as on the local system, which is true for the Deepthought clusters and for TerpConnect/Glue systems. From other Unix systems, you might need to specify your Deepthought/Deepthought2 username, e.g. ssh or ssh -l USERNAME , where USERNAME is your Deepthought/Deepthought2 username.

For MARCC/Bluecrab, because of the in your username, you will always need to specify the user name and should either use the -l option to ssh or quote your username, e.g. ssh -l johndoe or ssh ""

  • Do NOT try to log into or These are the master nodes for the clusters, and you do NOT have access to it.
  • Your username and password are your standard campus directory id (the part of your or email address to the left of the @)

The first time you connect to the system via ssh, you might receive a warning message like (color added to assist in discussion below):

The authenticity of host ' (' can't be established.
# RSA key fingerprint is 5f:5d:98:d9:19:0a:d2:d4:59:b8:48:a5:c6:b4:0b:9a
# Are you sure you want to continue connecting (yes/no)?

The message sounds scary, but it is normal. SSH tries very hard to protect you from all sorts of hacking, and one such mechanism is to try to ensure you are talking to the system you think you are talking to. For every server you connect to, it remembers a secret (RSA fingerprint) to prove the identity of the server, and verifies that (for brevity, this is a gross oversimplification; for more information). But it cannot verify it the very first time you connect (unless, as is the case on some campus systems, systems staff have pre-populated SSH with information about the system you are connecting to). This message is just to inform you about that.

The IP address and the hostname (in green above) may vary, although the hostname should match the name of the system you want to connect to. The parts in red (the key type and the fingerprint) will depend on the system you are trying to communicate with. To be secure, you should verify that it matches one of the fingerprints listed below:

  • For or (RHEL8)
    • RSA key MD5 fingerprint 5f:5d:98:d9:19:0a:d2:d4:59:b8:48:a5:c6:b4:0b:9a
    • RSA key SHA256 fingerprint 4CcTjtOk4tR0dPNg2v/QIN6cRyt/3o+rYFGqGUGsCBU
    • DSA key MD5 fingerprint eb:62:e0:fe:a6:14:35:fe:00:ca:a6:9d:66:f0:0e:0c
    • DSA key SHA256 fingerprint /VVN4U2J9FbyrGcVToZNNdNf28d17RmMEyGWDhn8i2c
    • ECDSA key MD5 fingerprint 84:e6:63:42:e6:11:e0:df:af:32:e6:f2:0a:a7:87:21
    • ECDSA key SHA256 fingerprint y3mfY9BzFTQgln1SLlxKg5lFlGOWTzTW9k+QxrvlS7M
    • ED25519 key MD5 fingerprint a8:81:2c:85:c6:b4:59:23:f8:5a:34:bf:c7:25:61:da
    • ED25519 key SHA256 fingerprint UHJJv88sw6oXS2r1klbEjGqn/0UWehqF+GqDziBMESc
  • For (RHEL6)
    • RSA key MD5 fingerprint f1:ca:a9:c2:af:44:51:de:8b:34:e7:88:0a:a8:76:c1
    • RSA key SHA256 fingerprint YmasH7fEWPSXPpdJUy6Do2FHkhXOjGFCTkglBv9m0M0
    • DSA key MD5 fingerprint 90:02:49:d9:e1:76:8c:4c:59:20:8e:ca:3f:ad:31:dd
    • DSA key SHA256 fingerprint f0pBFbc4B+VwfCbhRBC5jKp0h+kbcJUdND+9/FyOaVE
  • For
    • RSA key MD5 fingerprint 72:c9:83:a7:c4:09:25:62:9d:bc:52:ec:f1:b2:11:48
    • RSA key SHA256 fingerprint 6vz247GeFAcZk6tFIWt3MvtgRdjWGaBmNvhiHEE2o0M
    • DSA key MD5 fingerprint 49:bc:ce:2d:e6:63:43:88:d8:87:db:82:ae:f0:98:88
    • DSA key SHA256 fingerprint LBGxzQG0f/4nvTvW7TFab00exMKvbnCOxRVYS3FNPU8
    • ECDSA key MD5 fingerprint 52:4b:d6:a0:fc:84:96:e0:af:e2:c7:bf:95:3c:ba:74
    • ECDSA key SHA256 fingerprint 69EJPSFNW+BrSsYgNaWFyl8XZta0lg83MpuumJUsFzA
    • ED25519 key MD5 fingerprint 3c:cb:69:14:32:d5:1f:1a:6c:e7:d5:87:4e:89:a2:9f
    • ED25519 key SHA256 fingerprint ZiswYeg/FjNeFiCdeN9TzUX+66TZ5Las2PVSnN2tEpU
  • For
    • RSA key MD5 fingerprint df:b7:48:55:fa:1c:80:41:22:64:d5:f7:04:7a:10:e1
    • RSA key SHA256 fingerprint guk52KukC0EgCBvFM1zLRXPbWFz7QQv/e+0htklsyyA
    • DSA key MD5 fingerprint 5f:33:be:43:97:b5:5c:26:f5:f8:a4:77:61:3c:54:e6
    • DSA key SHA256 fingerprint 6z3yzKV+7y+Y4Ynee1jPYhYCLWhn9NwXPcn7CRDPcLM
    • ECDSA key MD5 fingerprint f2:08:8f:a8:62:51:3a:fd:d2:c1:90:9f:92:a6:f6:f9
    • ECDSA key SHA256 fingerprint 5KNszbuzqNvDS+R9IcbosU1+rCYuAHtKmHzFXZ0iUz8
    • ED25519 key MD5 fingerprint 73:e7:3d:35:4f:1b:53:9b:0c:18:a8:78:e7:9d:bb:b5
    • ED25519 key SHA256 fingerprint addS+emJ00NCDUgyavXTXMNP/txTmSHmWgBxKf1iTp0
The MARCC/Bluecrab cluster is maintained by MARCC staff, not the Division of Information Technology at the University of Maryland. Therefore, the MARCC website should always be considered the definitive source of information about MARCC and the Bluecrab cluster. Any information provided on this UMD website is provided as a convenience to our users, but we cannot guarantee its accuracy or currency. In case of discrepency, you should assume the MARCC website is the correct one.

If the fingerprint does NOT match, you should NOT enter your password and contact system staff ASAP. If you enter your password in such a situation, as it is possible that someone is performing a man-in-the-middle attack and can obtain your password when you enter it.

If you see a message like

Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.

you should always refrain from logging in and contact system staff as soon as possible. This means that the server you are connecting to did not know the secret remembered by SSH for that system, as described above, which means either system staff changed the keys, or someone is hacking you. As system staff do not change the keys often, and will send email to everyone well in advance of changing the keys warning you of this if we were to, this likely means someone is attacking you unless you received an email from systems staff. Do NOT enter your password and contact system staff.

Setting up ssh public-key authentication

This section discusses how to setup and use SSH with public-key based authentication to allow for access to the cluster without typing your password for every ssh session. It can also be used to allow passwordless ssh between the various login and compute nodes; this is useful if you are using ssh to spawn processes on the different nodes allocated to your job.

The procedures listed in this section are NOT required for you to access the system, and you can use normal password based authentication instead. It also is NOT for most multinode jobs (using MPI or srun). It is only for users who wish to set up public-key authentication, either to allow passwordless access to the cluster from your workstation, or to allow passwordless ssh between the nodes of the cluster.

If you are new to Unix/Linux, it is recommended that you skip this section and just use password based authentication.

Public-key authentication uses asymetric encryption for its security. In asymmetric cryptographic systems, there exist distinct private and public keys. Data can be encrypted and/or digitally signed with the private key and can only be decrypted/signature verified with the public key. The private key is kept secure on your workstation, and the public key is copied to the HPC cluster. When you log in with public key encrpytion from your workstation, the ssh client on your workstation will digitally sign a standard message and send this to the sshd server process on the cluster along with the appropriate public key. The remote sshd process verifies that the public key is authorized, and if so verifies if the digital signature is valid. If the signature is valid, that means the ssh client is in possession of the private key corresponding to the public key, and because the public key is authorized, grants it access to the system.

NOTE: the public-key authentication process grants login access to anyone/any system/etc. that knows your private key. Thus you need to ensure the security of your private key. Protect your private key at least as strongly as you would your password.

Setting up public-key authentication with ssh

Instructions for setting up public key authentication is discussed below. We break it down into two cases:

  1. for passwordless ssh among the nodes of the cluster
  2. for passwordless ssh from your workstation to the login nodes of the cluster

Although the process is essentially the same in each case, because of the shared home directory among the nodes of the cluster, the first case is a bit simpler and will be treated separately. The second case is slightly more complicated because there are steps which need to be done on both your workstation and on the HPC cluster.

Depending on your needs, you can do none of the steps, either one of the step, or both of them.

NOTE: Because MARCC/Bluecrab requires multi-factor authentication, you cannot use the second option to enable passwordless ssh from your workstation into the Bluecrab cluster.

Setting up public-key authentication for passwordless ssh among the nodes of the HPC cluster

In certain cases, it might be necessary to enable passwordless ssh between the nodes of the cluster. The most typical such case is if you must use the ssh command to launch processes on multiple nodes as part of your job. (Most multinode jobs use MPI and/or the srun command, and so do not require this, but some may.) Because you cannot feasibly enter a password with a batch script, you need to enable passwordless ssh in such cases.

Because your home directory is shared among all nodes in the HPC cluster, everything for this process can be done on one of the HPC login nodes. Just log into a login node on the cluster, and then:

  1. Generate your host key, if needed.
    1. Run the command ls -l ~/.ssh/id_rsa.
    2. If the ~/.ssh/id_rsa already exists, you should already have the public host keys and should not need to do anything else and can skip the running the ssh-keygen command, although you should still verify the ownership and permission of the file.
    3. If you are having problems with passwordless among the cluster nodes, you can safely regenerate the public host key with the next step. If you arranged for passwordless ssh from the cluster to other systems, you will likely need to fix the authorized_keys file on the other systems if you regenerate the HPC host key.
    4. To (re-)generate the host key, run the ssh-keygen command. Accept all of the default value for the name of the file in which to save the key; the public key will be stored in the same directory, with a .pub extension. Generally, you should not enter a passphrase (just hit return at the two passphrase prompts) --- the most common case is to enable passwordless ssh among nodes in the cluster for use in job scripts, and that will not work if the key has a passphrase.
    5. Repeat the ls -l ~/.ssh/id_rsa command from above. It should now show the existance of the file. Please verify that the file is owned by you, and no one can access it but you. (Permission flags should be -rw-------.).
  2. Add the public host key to the authorized_keys file.
    1. If ~/.ssh/authorized_keys already exists, you should append the contents of ~/.ssh/ to the file; you can do this with the command 'cat ~/.ssh/ ~/.ssh/authorized_keys'.
    2. If the authorized_keys file does not exist, yo can create it with the proper contents with the command: 'cp ~/.ssh/ ~/.ssh/authorized_keys'.
Please ensure that your id_rsa file is only readable by you. This host key is all that is needed to access any system which has the corresponding public key added to its authorized_keys file.

You should now be able to ssh between nodes in the cluster without entering a password. Note that you do not have access to ssh to compute nodes unless you have a job currently running on the node. Typically this passwordless ssh is only needed to ssh between nodes allocated to a job within your job script.

Setting up public-key authentication for passwordless ssh into the HPC cluster from your workstation, etc.

Atlhough the process is in this case uses basically the same steps as above, because your workstation and the HPC cluster do not share a common home directory, the different steps need to be performed on different systems, and there is a file transfer required.

Some of the steps below are dependent on the ssh application being used on your workstation/desktop/laptop/etc. We are assuming putty on Windows systems and the standard command line openssh client on Macs or Linux systems. If you are using something else, hopefully it has some documentation describing how to set up public key authentication and which together with the instructions belong can be used to figure out what you need to do.

  1. First, you need to generate a private-public key pair. On
    • Windows systems: you should use the PuTTYgen utility to generate the key pair. Open the PuTTYgen utility, and
      1. At the bottom of the PuTTY Key Generator window, in the Parameters section, select the type of key to generate. It is recommended to use "RSA" (perhaps called "SSH-2 RSA" in some older versions) using the default (2048) number of bits.
      2. Click Generate in Actions section. You will be prompted to use your mouse/etc. to generate entropy that will help make the private key secure. Move the cursor around until the utility has generated the key (it will display in the area under Key).
      3. If desired, you can enter a passphrase to be used in encrypting the private key in the Key passphrase and Confirm passphrase boxes. Encrypting the private key with a passphrase will increase security, and with the Pageant utility provided with PuTTY you only need to enter the password once per login session on your workstation.
      4. Save the generated public key by clicking Save public key under Actions (next to Save the generated key). Enter the name (e.g. putty_public_key) and folder, and click Save.
      5. Save the generated private key by clicking Save private key under Actions (next to Save the private key). If you did not opt to encrypt the private key, it will ask if you are sure that you wish to save an unencrypted private key. "Yes" will proceed to save; "No" will allow you to go back and specify a passphrase to use. Use the default format/Save as type (PuTTY Private Key Files (*.ppk)), enter the name (e.g. putty_private_key) and folder, and click Save.
    • Mac or Linux systems: you should use something like ssh-keygen -t rsa.
      1. This will prompt you for a filename to save the private key to: the default is id-rsa in the .ssh subdirectory of your home directory. It is recommended that you use this default otherwise you need to tell the ssh or ssh-agent command which identity file to use when logging in.
      2. The public key will be placed in the same directory as the private key, with the same name but with a .pub suffix added.
      3. You will also be prompted for a passphrase to encrypt the private key with. Encrypting your private key with a passphrase will increase security, and with the use of ssh-agent (described below) you only need to enter the passphrase once per login session on your workstation. If you opt not to use a passphrase, just hit return without entering anything to leave the private key unencrypted.
      4. You will be prompted a second time for the passphrase (to ensure it was entered correctly). Type the passphrase from the previous step, or just hit return again if you opted for no encryption of private key.
  2. Next, you need to authorize that key to log into the Deepthought2 cluster as you. This will require logging into the cluster using your password because the public key authentication is not set up yet.
    1. Use a scp or sftp client to transfer the public key file created in the previous step to a Deepthought2 login node. For linux or Mac users that kept the default name, the public key file will be ~/.ssh/ For Windows users, it will be the name used when you saved the public key (e.g. putty_public_key). DO NOT transfer the private key file --- that should remain in a protected spot on your workstation.
    2. Ssh to a Deepthought2 login node using password authentication.
    3. Use the command mkdir -p ~/.ssh to create the .ssh directory under your home directory if it does not already exist (the command will not harm anything if it does already exist).
    4. Use the command touch ~/.ssh/authorized_keys. If there is no authorized_keys file in the .ssh subdirectory of your home directory, this command will create an empty file. If there was one, it does not change the contents of the file.
    5. Use the command chmod 600 ~/.ssh/authorized_keys to ensure the proper permissions on the file. No one but you should be able to read or write to the file.
    6. Use the command cat PUBLIC_KEY_FILE >> ~/.ssh/authorized_keys to append the public key file to the authorized_keys file. Be sure to use TWO > characters without space between them in the above command (otherwise you might overwrite the file and lose previous contents). The PUBLIC_KEY_FILE in the above command should be replaced by the name of the public key file you just copied to the cluster; e.g. for Linux or Macs or putty_public_key or whatever you saved as on Windows.
    7. Delete the public key file you copied to the cluster using rm PUBLIC_KEY_FILE as it is no longer necessary.
NOTE: Your private key is all that someone needs to access the cluster as you. KEEP THE PRIVATE KEY FILE SECURE. It is strongly suggested that you encrypt the private key file with a passphrase, so that both the passphrase and the file are needed to access the cluster.

You should now be able to ssh to the Deepthought2 cluster from your workstation using public key authentication, as described below.

Using up public-key authentication with ssh

In this section we discuss how to use public-key authentication with ssh. Because it is good practice to encrypt your private key, typical usage involves starting an agent that runs in the background. You start the agent when you log into your workstation, and it loads your un-encrypted private key into memory at that time (it will need to ask you for the passphrase to decrypt it if it was encrypted). Then for the rest of the time you are logged into your workstation, it will provide the key to the ssh client so that you can connect to the cluster without providing a password. The agent differs according to what ssh client you use.

NOTE: If you login to Deepthought2 using public-key authentication, you will not have Kerberos tickets or AFS tokens, so certain commands will not work. For example, you will not be able to access the contents of your ~/glue_home directory. The HPC clusters are much less dependent on your having Kerberos tickets and AFS tokens than standard Glue/TerpConnect systems, so this is usually not a significant issue. If you do find yourself needing such, you can just issue the renew command (which will prompt you for your password).
Using public-key authentication on Windows systems

On windows systems, should be a Pageant SSH authentication agent installed with PuTTY.

  1. First, ensure that PuTTY is configured to use Pageant. This is the default, but just to be certain:
    1. Open PuTTY and in the left hand "Category" panel click on the small plus (+) next to SSH under Connection to expand the SSH options.
    2. Click on the Auth subtree that now appears under SSH
    3. Ensure that the box "Attempt authentication using Pageant" is clicked.
    4. Exit PuTTY
  2. Open the Pageant SSH authentication agent. It runs in the background, so when it is open you will just see a new icon (a computer wearing a hat) for it in the Windows notification tray. Double click on that icon to open up the Pageant window.
  3. The main window will list the keys Pageant is holding, which is probably none at this time. Press the Add Key to add the private key created in the previous section. This will bring up a file dialog, so find the private key you created (e.g. putty_private_key.ppk) and "open" it.
  4. If the key was encrypted using a passphrase, you will be prompted to enter the passphrase now. You will not need to enter it again as long as Pageant is running.

You can now use PuTTY to login into the Deepthought2 login nodes as before, and it will use public key authentication and not ask for your password on the cluster.

You might wish to have Pageant start up whenever you login to your Windows workstation. To do that:

  1. Right-click inside the Startup folder, and select New and Shortcut.
  2. In the Type the location of the item box, you should enter the path to pageant.exe followed by a space and the full path to your private key (e.g. putty_private_key.ppk). You should put both paths (the executable and the private key) in double quotes.
  3. Then click Next and enter a name for the shortcut (e.g. Pageant).


"C:\Program Files (x86)\PuTTy\pageant.exe" "C:\Users\user_profile\ssh_key\putty_private_key.ppk"
All of that should be on a single line.

Then, every time you log into the Windows workstation, it will start Pageant (prompting you for the encryption key for the private key if it is encrypted) and you can use PuTTY to ssh to the Deepthought2 login nodes without entering any additional passwords for the remainder of your workstation session.

Using public-key authentication on Linux or Mac systems

On Linux or Mac systems, there are ssh-agent and ssh-add commands that should be standard. You can start the background agent by issuing the command ssh-agent. You can then add private keys to the agent using the ssh-add command. If you use the default location for your private key (e.g. ~/.ssh/id_rsa) you can just issue the command ssh-add and it will add any private keys in the default locations. If you added the key in a non-standard path, just use ssh-add PATH_TO_PRIVATE_KEY where PATH_TO_PRIVATE_KEY is the path to the private key file to use. If the private key is encrypted, the ssh-add will prompt you for the passphrase to decrypt it.

At this point, you can just ssh (where USERNAME is your username on the Deepthought2 cluster) and you will be logged into a login node using public key authentication; i.e. no additional password needed.

You can combine the ssh-agent and ssh-add commands into a single shell script if you want, or add to your .cshrc or .bashrc initialization scripts if you want them to start automatically when you login to your workstation.


The login nodes are capable of running graphical applications with the graphics appearing on your desktop using the network features of the X11 Windowing System.

In order for this to work, however, you need to be running an X11 server on your desktop. The exact procedure depends on what your desktop, etc. is, and you might wish to contact the UMD helpdesk for assistance.

  • For Linux or Unix/Unix-like systems, X11 is usually available by default. You should check your DISPLAY environmental variable (e.g. echo $DISPLAY should return something, usually your hostname followed by a colon and a number). You might need to add a -X to your ssh command to allow ssh to forward X11 connections.
  • For Macs running OS X, see this or this knowledge base article .
  • Microsoft Windows does not ship with an X11 server, but there are third party servers available. Cygwin/X is an open-source/free implementation, and Xming is another popular free choice. There are also some proprietary implementations like Xwin32, MKS X/Server, and Exceed.
  • There is also at least one java implementation, WeirdX, that is able to run as an applet in a web browser.

Some software packages have their own protocols for remote visualization, including:

  • Matlab: The Matlab Distributed Computing Server (MDCS) allows one to run Matlab on one's desktop but have computation run on the HPC cluster. Among other benefits, you can do plotting on local machine using data generated on the cluster.
  • ParaView: Paraview supports a remote visualization mode wherein you can do calculations and even rendering on compute nodes and have graphics sent back to your desktop. More information forthcoming.
  • ViSit: ViSit supports a client/server remote visualization mode enabling you to visualize data on the HPC cluster filesystems and/or do rendering on the HPC compute nodes.

Basic unix commands

This is a very short list of some very basic Unix commands to help you get started in the HPC environment.

  • To list files and directories, use the ls command. With no arguments, it lists all files in the directory. You can give it a -l option to show more detail, and/or a file or directory name to show that file or directory.
  • For help on a command, use the man command. E.g., you can use man ls to find out all about the ls command and its options.
  • To make a directory, use the mkdir command. Give it the name of the new directory to make, e.g. mkdir foo.
  • To change directories, use the cd command. Give it the name of the directory to change to, e.g. cd foo. You can also use the pushd and popd to temporarily go to another directory and return. If I am in the directory foo and I issue pushd /export/lustre/newfoo, do some stuff, then popd, I will be back in the foo directory I started in.
  • To copy a file, use the cp command. Usage is cp FILE1 FILE2, which will copy an already existing file FILE1 to FILE2.
  • To move a file to a new directory, or to rename a file, use the mv command, e.g. mv FILE1 FILE2 to rename FILE1 to FILE2.
  • To move files back and forth between one of the HPC clusters and other systems (including the other HPC clusters), the best options are probably sftp and/or scp. If you are on one of the Deepthought/Deepthought2 login nodes or the Bluecrab data transfer node, and wish to transfer files to/from a system which can act as a server for the scp or sftp protocols, you do one of
    • scp FILE1 USERNAME@RHOST:FILE2 to copy FILE1 on a Deepthought/Deepthought2 login node, or a Bluecrab data transfer node, to the system RHOST, putting it with the name FILE2 on the remote system. You can leave out FILE2 to keep the same name (but be sure to keep the colon (:)). USERNAME is your username on the remote system; you can omit the USERNAME@ if it is the same as on the HPC system.
    • scp USERNAME@RHOST:FILE2 FILE1 does the reverse; it copies FILE2 from RHOST to the login node.
    • sftp RHOST will start an interactive SFTP session from one of the Deepthought clusters' login nodes or the Bluecrab data transfer node to the SFTP server on system RHOST. You can use the get command to copy files from RHOST to the local system, or put to copy files from the local system to RHOST. For more information, see the following tutorial on using sftp.

    MARCC/Bluecrab users should use the data transfer nodes for sftping or scping large amounts of data. The MARCC/Bluecrab cluster also provides a Globus end point, see the MARCC/Bluecrab FAQ for more information about that.

    If the system you wish to get the files from/copy the files to does NOT provide an sftp or scp server, you can still use sftp or scp to transfer files if you can run an sftp or scp client on the system. You might need to install such a client, e.g. if it is a Windows desktop. This helpdesk article willshow you how to install and use sftp/scp clients on various systems .

  • To view a file, use the less command. You can use something like less FILE1 to view the contents of the text file FILE1. You can use the space key to page forward, the "b" key to page back, up and down arrows to scroll a line up or down, and "q" to exit. You can also use it to read long output from another command, e.g. squeue | less to allow you to page through all the jobs running on the system.
  • There are several text editors available on the system. If you are not familiar with either vi or emacs, we recommend that you stick with pico.
  • Logging out: Just type logout

Hopefully, the above list will get you through most of the basic tasks you need to complete. A full tutorial on using Unix is beyond the scope of this document. However, there are many tutorials for beginning to use Unix on the web. A few are tutorials we recommend are:

Changing your default shell

The shell is the command line prompt that you interact with. It also is what processes shell scripts. Although there are a number of shells available on TerpConnect/Glue/HPCC systems, the most common are tcsh and bash. Your default shell is the shell you get when you log into a system; by default on the Deepthought HPC clusters it is set to /bin/tcsh as this is (debatably) considered to be the easiest shell for interactive use. The MARCC administrators are of the other opinion, and set the default shell to /bin/bash, presumably for the same reason.

You can change your default shell on the two Deepthought clusters using the chsh command. Your default shell is common across all Glue/TerpConnect/Deepthought* systems at the University of Maryland.

  • You must have valid Kerberos and AFS credentials to use the chsh command. To ensure that, you should run renew first, and enter your password if requested. If you do NOT have valid credentials, the chsh command will fail with an error message about being unable to create a file and ask if you are authenticated. Run renew and try again.
  • The chsh command also allows you to change your room and phone numbers as listed in the Unix password file (i.e. the output of the finger command). You can just hit return to keep the values unchanged.
  • If you use the chsh command, it can take up to an hour for the changes you request to take effect. And for changing the default shell, this will only impact future logins.
  • If changing your default shell, you MUST give full paths to the shell executable. I.e., give either /bin/tcsh or /bin/bash.
  • Your default shell does NOT affect what shell your job scripts run under. This is controlled by the shebang at the top of the script; for more information about contact the MARCC/Bluecrab systems staff.

    Setting Up Your Environment

    In order to provide an environment that can support a wide variety of users, many software packages are available for your use. However, in order to simplify each individual user's environment, a large number of those packages are not included in your default PATH.

    Your account as provided gives you access to the basic tools needed to submit and monitor jobs, access basic Gnu compilers, etc. It is HIGHLY suggested that you DO NOT remove or modify the dot files (.cshrc, .profile, etc) that are provided for you on the Deepthought HPC clusters. Instead, add any customizations you need to the alternate set of files described here , e.g. .cshrc.mine and/or .bashrc.mine. If you choose to modify the system default files, you run the risk of losing any systemwide changes that are necessary to keep your account running smoothly.

    DO NOT modify your .cshrc or .bashrc files on the Deepthought clusters. Doing so could cause you to either lose your changes or lose systemwide updates that are needed to keep things running smoothly. Instead, we provide some alternative files, e.g. .cshrc.mine and .bashrc.mine which will be sourced from the main .cshrc and .bashrc, respectively. So modify the dot files ending in .mine. See here formore information .
    Bash users on the Deepthought clusters should NOT create a .bash_profile file in their home directory. This will prevent the execution of the system initialization scripts beginning with .profile. Again, .bashrc.mine is available for user customization.
    Tcsh users on the Deepthought clusters should NOT create a .tcshrc file in their home directory. This will prevent the execution of the system initialization scripts beginning with .cshrc. Again, .cshrc.mine is available for user customization.

    The MARCC/Bluecrab cluster does not use the .cshrc.mine or .bashrc.mine files. On the MARCC/Bluecrab cluster, you should simply edit you .cshrc (if you changed your default shell to tcsh or csh) or .bashrc file directly. It is HIGHLY advised that you DO NOT remove the part near the beginning that sources the global definitions, e.g. the part like

    # Source global definitions
    if [ -f /etc/bashrc ]; then
            . /etc/bashrc
    as doing so will lose any site wide set up.

    For packages that are not included in your default environment, the module command is provided. When run, this command will modify your current session by adding the appropriate entries to your PATH and whatever other variables are necessary to ensure the proper functioning of the package in question. Note that these changes are temporary and only exist until you log out. If you want to have code run for you automatically, add the command module load PACKAGE. If you run the module avail command, it will provide a list of all available packages. module avail STRING will return a list of all available packages starting with the string STRING. The commands module whatis PACKAGE and module help PACKAGE will return a short one-line description and a longer, multiline help text, respectively, for the named package.

    Note that many of the packages shown with module avail are NOT accessible on the compute nodes of cluster by default. This is due to the fact that the compute nodes have their own copy of the globally shared Glue software directory, and to conserve space only those packages considered useful for HPC environments have been copied. If you see a package listed under module avail which is not available on the compute nodes, let us know and if possible, we'll make it available. The software page contains listing of the various packages available. If you click on the package name, you will get detailed information as to which versions are available, and whether they are available on the compute nodes or not of a particular cluster.

    NOTE: The tap, which performed a similar role to the module command but was unique to UMD and lacks some of the features of the latter, is no longer available on either Deepthought cluster. It is deprecated on the rest of Glue already, and the migration to the module command is slowly occuring there as well, but due to the need to have separate builds depending on compiler and MPI library for many HPC packages, the process has been expedited on the HPC clusters.
    If you use the tap in your .cshrc.mine or .bashrc.mine scripts, be sure to give the -q argument before the package name, e.g. tap -q PACKAGE. This suppresses the normal help output, which can confuse some shells and break file transfer protocols.

    For example, if you want to run Matlab, you'll want to do the following. Notice that Matlab is not available until after the module load command has been run.

    f20-l1:~: matlab
    matlab: Command not found.
    f20-l1:~: module whatis matlab
    matlab               : Matlab 2014b
    f20-l1:~: module load matlab
    f20-l1:~: matlab
                                  < M A T L A B >
    		  Copyright 1984-2014 The MathWorks, Inc.
                       R2014b ( 64-bit (glnxa64)
                                 September 15, 2014
      To get started, type one of these: helpwin, helpdesk, or demo.
      For product information, visit
    If you are running a bourne or bash script under sbatch but your default shell is csh or tcsh, remember that you must include a . ~/.profile (on the Deepthought2 cluster) or a . ~/.bash_profile (on the Juggernaut cluster) in your script to enable the module load and/or tap commands.

    For more information, see the section on the module command.

    Preventing output from your dot files

    It is strongly recommended that your dot files (e.g. .cshrc.mine and/or .bashrc.mine) do not produce any terminal output, or at least, only do so when run in an interactive shell. Output in non-interactive shells can cause problems, most notably with some file transfer protocols like scp and sftp --- in most cases, the stray output will confuse the file transfer protocol usually causing it to abort.

    If commands in your .cshrc.mine, .bashrc.mine, or other dot files might produce output to the terminal, you should take measures to ensure such output is only produced in interactive shells. Otherwise, you might break file transfer protocols like sftp or scp.

    If you have commands in your dot files which might produce output, you should consider running them only in interactive shells so as not to confuse file transfer protocols. The method varies by the type of shell; for csh and/or tcsh style shells, something like:

    if ( $?prompt ) then
    	#Only execute the following in interactive shells
    	echo "Hello, today is "
    #Here we redirect output (if any) to /dev/null
    some_command >& /dev/null

    For Bourne-like shells (e.g. sh and/or bash) something like:

    if [ ! "x$PS1" = "x" ]; then
    	#Only execute the following in interactive shells
    	echo "Hello, today is "
    #Here we redirect output (if any) to /dev/null
    some_command > /dev/null 2> /dev/null

    You can also redirect output to /dev/null, as is done in the case of some_command in the above examples. Be sure to remember to redirect both stdout and stderr. In some cases, the command has options to silent it, which can also be used (e.g. the old tap has a -q option), but this generally will still get output from errors.

    Files and Storage Basics

    On each cluster, you have several options available to you regarding where files are stored. A whole section is devoted to this on another page, but it is important and basic enough that a short discussion is merited here.

    The file structure is somewhat different between the two Deepthought clusters and the MARCC/Bluecrab cluster.

    Files and Storage Basics: the Deepthought clusters

    With access to the cluster, you are granted a home directory, which is the directory you see when you first log in. It is distinct from your standard TerpConnect/Glue home directory. Also, if you have access to more than one HPC cluster, your home directory on each is distinct from the other(s). On the login nodes of the Deepthought clusters, your standard TerpConnect/Glue home directory is located as the subdirectory glue_home underneath your Deepthought/Deepthought2 home directory; this directory is NOT accessible from the compute nodes.

    Your home directory is, by default, private to you, and should be used as little as possible for data storage. In particular, you should not run jobs out of your home directory --- run your jobs out of a /data volume or from the lustre filesystem; these (especially lustre) are optimized to provide better read and write performance to improve the speed of your job. After the job is finished, you might wish to copy the more critical results files back to your home directory, which gets backed up nightly. (The /data and lustre filesystems are NOT backed up.)

    You should run jobs out of a /data volume or out of lustre. So one of your first actions on the cluster should be to create a directory for your jobs in lustre. To do this on Deepthought, mkdir /export/lustre_1/USERNAME. On Deepthought2, the directory /lustre/USERNAME has already been created for you. USERNAME is your Deepthought/Deepthought2 username. Then cd to that directory to run your jobs.

    Files and Storage Basics: the Bluecrab cluster

    With access to the cluster, you are granted a home directory, which is the directory you see when you first log in. It is distinct from your standard TerpConnect/Glue home directory. Also, if you have access to more than one HPC cluster, your home directory on each is distinct from the other(s).

    Your home directory is, by default, private to you, and has a 20 GB quota. This is the only directory of yours which is backed up automatically. In your home directory are the symlinks scratch and work. Both of these are links into the lustre filesystem, and are NOT BACKED UP. The scratch directory is specific to you, and by default is private to you, and has a 512 GB quota. It is for files needed for active jobs on the cluster. The work directory is shared by all users in your project/allocation; it has a 5 TB limit, and is readable and writable by all members of the allocation.

    Your home directory should be used as little as possible for data storage. In particular, you should not run jobs out of your home directory --- run your jobs out of the work or scratch directories.

    Transferring files to/from the HPC clusters and other systems

    Before long, you will need to transfer files to the HPC clusters from your computer, or from one of the HPC clusters to your computer, or otherwise move data to/from one of the HPC clusters to another location. There are several options for this.

    Using scp or sftp protocols

    The "standard" unix-like utilities for transferring data between nodes are the scp and sftp programs, which implement the Secure Copy (SCP) and Secure File Transfer Protocol (SFTP), respectively. These are typically preinstalled on Unix-like systems. Windows or Mac users might need to install a scp/sftp client on their machine.

    If you are transferring between clusters, generally both sides have the server processes running, and you can initiate the transfer from either side. When dealing with your workstation, it most likely does NOT have the server running, so you regardless of which way you wish to transfer data, you will likely want to initiate the transfer from your workstation.

    Assuming you are on your workstation and your have the client installed, just open up the client and point it to the Deepthought/Deepthought2 login nodes as described in the section on logging into the clusters.. You can also use sftp or scp to transfer data to/from the MARCC/Bluecrab cluster, but in this case you should direct traffic at the data transfer node rather than the login node. Also, because your username on Bluecrab includes the @ symbol, which is what scp and sftp normally use to specify the username in file string, you will need to use the -o User=USERNAME option. I. e.,

    unix-prompt> scp -o myfile 
    myfile                               100%  867     0.9KB/s   00:00
    unix-prompt> sftp -o
    Connecting to
    sftp> put morefiles* 
    Uploading morefiles1 to /home-3/
    morefiles1                              100%  132     0.1KB/s   00:00    
    Uploading morefiles2 to /home-3/
    morefiles2                              100%  308     0.3KB/s   00:00    
    sftp> quit
    The above example shows how to transfer files using scp and sftp for the user; you will obviously need to replace that username with your own.

    This will by default allow you to move files to and from your home directory. For larger data sizes (more than a few GB) you almost certainly wish to place them in lustre or in one of the data directories..

    Some more detailed information regarding the use of the scp command can be found in the section on basic Unix commands

    Using globus

    For large amounts of data, you might wish to use Globus for transferring files. Globus can automatically user multiple streams (speeding up the transfers) and can automatically restart failed transfers (very useful when dealing with many GBs of data), and is supported by most HPC clusters.

    You can also install the free Globus Connect Personal; see instructions here.

    To use globus, go to the login page

    More detailed instructions re using Globus

    Transferring files to/from Cloud Storage Providers

    The use of Cloud Storage providers for storing data is becoming increasingly popular. The University provides free access to

    These Cloud Storage providers can be particularly useful for storing archival data. Please see the service catalog entries above for more information on the services, and in particular for a discussion of what data is permissible to store there.

    Each Cloud storage provider backend has slightly different procedures for uploading and downloading data. However, the rclone utility is a nice utility that can access many different Cloud service providers (including both Google Drive and Box) with a common command line interface.

    More information about using rclone can be found here.

    Compiling codes

    Compiling codes in an HPC environment can be tricky, as they often involve MPI and other external libraries. Furthermore, the Deepthought and Bluecrab clusters offer multiple compilers and versions, and multiple versions of many libraries, which can make this task even more complicated.

    First off, you must choose your compiler family. The Deepthought clusters offer the "standard" GNU compiler suite, the Intel compiler suite, and the PGI compiler suite; Bluecrab offers GNU and Intel compiler suites. The GNU compiler suite is probably the most widely used for open source packages, most of which will most easily compile on this suite. Intel makes frequent updates to their compilers, although we usually only update annually. The Intel compilers have "bleeding edge" optimization options, but have also been known to have bugs. The PGI compiler suite has some enhancements which might make it preferable for CUDA programming.

    In general, you should use the most recent version of the compiler available on the system unless you have a specific need for an older compiler. Bear in mind that the MPI libraries and a number of other external libraries (NetCDF, HDF, FFTW, etc) have dependencies on the compiler version --- you are strongly advised to ensure all dependent libraries and all parts of your code were built with the same compiler version and the same versions of any common dependent libraries. For system provided external libraries made accessible with the module command, the module avail subcommand will show you the different builds of the package available --- if module avail is showing tags with compilers, MPI library, etc. versions, the library is sensitive about such versions and you are advised to ensure that things match. The module load subcommand will complain if you try to load a version which does not match a previously loaded compiler, etc., but that is not perfect. If module avail does not show build tags with compiler, MPI library versions, etc., then the package should be usable with any compiler. See the section on the module command for more information.

    Once you decide which compiler you want to use, you should module load it. The only exception to this is if you want to use the default compiler, which is a somewhat old version of the GNU compiler suite (gcc version 4.6.1 at the time of this writing). It is recommended that one first loads the desired compiler, then the desired MPI library (if any), and then any other library packages needed. For the MPI and other libraries, it is recommended that one just gives the package name possibly with a version number (the module command will look for an appropriate build based on the previously loaded compiler and/or MPI libraries).

    The Deepthought clusters support various versions of the GNU Compiler Suite, the Intel Compiler Suite, and the PGI Compiler Suite; follow the links to see the different versions supported. With the exception of the default gcc compiler (version 4.6.1 at /usr/local/bin/gcc) you will need to issue a module load command to make the desired compilers available, e.g. module load gcc/4.9.3 or module load intel/2015 or module load pgi.

    If you plan to use an MPI library, you must module load the appropriate MPI library. (Exception: On the Deepthought clusters if you are using Intel compilers and wish to use the Intel MPI libraries, the module load for the intel compilers gets you both. For the MARCC/Bluecrab, you would need to load both the compiler (module load intel-ics) and the MPI library (module load intel-mpi)). The MPI library chosen must match your compiler choice, or bad things might happen. Use of the MPICH and LAM MPI libraries is strongly deprecated. OpenMPI is the recommended choice. You are advised to use the latest; normally you only need to specify openmpi/VERSION in the module load command, and we will figure out which compiler you previously loaded, and configure for that compiler. If you chose an incompatible pair (e.g intel/2011 and openmpi/1.6.5) you will get an error when you load the openmpi module.

    The following compiler commands are available on the Deepthought HPC clusters:

    Compiler FamilyMPI library C compilerC++ compilerFortran77 compiler Fortran90 compiler
    GNUnonegccg++ gfortrangfortran
    GNUOpenMPImpiccmpic++ mpifort (for OpenMPI>1.7)
    mpifort (for OpenMPI>1.7)
    GNUIntel MPI*** Illegal combination ***
    Intelnoneiccicc ifortifort
    IntelIntel MPImpiiccmpiicc mpiifort (NOTE the doubled i) mpiifort (NOTE the doubled i)
    IntelOpenMPImpiccmpic++ mpifort (for OpenMPI>1.7)
    mpifort (for OpenMPI>1.7)
    PGInonepgccpgc++ pgf77 pgf90
    PGIOpenMPImpiccmpic++ mpifortmpifort
    WARNING: If you are using the Intel MPI libraries, there are commands mpicc, mpif77, and mpif90 defined, but these will NOT WORK. These attempt to compile the code using GNU compilers and Intel MPI, a combination which is NOT supported on these clusters.
    WARNING: When using recent OpenMPI libraries and the Intel Fortran Compiler, the preferred (by OpenMPI authors) command to use is mpifort (note: the i is NOT doubled). This is unfortunately very similar to mpiifort (with the i doubled); be careful to use the correct one. The latter (mpiifort, with the doubled i) also exists, and will compile with the Intel Fortran Compiler and the Intel MPI libraries, which is not what you want.

    Note that if you are planning to run the code on a different machine than it was compiled on, there can be issues with the instruction sets recognized by the compilers on the two systems. This is not much of a problem on a mostly homogeneous cluster like Deepthought2, but can be problematic on a heterogeneous cluster like the original Deepthought. The PGI compilers in particular have a tendency to use instructions specific to the processor being compiled on by default, and since the DT1 login nodes are newer than some of the compute nodes, this can cause problems. If you are getting Illegal instruction errors at runtime, you may need to specify a more generic target processor. See the PGI compiler suite page for more information on dealing with that with the PGI compilers.

    With the release of OpenMPI 1.7, the commands mpif77 and mpif90 have been deprecated in favor of mpifort (NOTE: the i is NOT doubled). The mpif77 and mpif90 commands still exist as of OpenMPI 1.8.6 (and simply are aliases for mpifort), but might be dropped in future versions. The choice of name is regrettably similar to the command for compiling with the Intel Fortran Compiler and MPI libraries (namely, mpiifort, with the doubled i); users of Intel Fortran compilers intending to compile with OpenMPI libraries are encouraged to use care; as in this case both mpifort and mpiifort are defined, the the former using OpenMPI libraries and the latter using Intel MPI libraries.

    Also note that not all MPI libraries/versions are not built for all compiler versions. The Intel MPI libraries are currently only supported for the Intel compiler suites (within the same version of the compiler suite). For the OpenMPI libraries, the command module avail openmpi/ will list the compatible combinations, both as openmpi/OPENMPI_VERSION/COMPILER_FAMILY/COMPILER_VERSION and as openmpi/COMPILER_FAMILY/COMPILER_VERSION/OPENMPI_VERSION. For your convenvience, the following table also attempts to list the compatible versions. The boxes marked with an "X" indicate a compatible pair:

    Compiler MPI library
    GNU 4.6.1 X        
    4.8.1 X X X    
    4.9.3     X    
    6.1.0       X  
    Intel 2013 X X X   X
    2015     X   X
    2016       X X
    PGI 17.3       X  

    For the MARCC/Bluecrab cluster, use the command module avail to see what compiler and MPI library combinations are available. The compiler names should be the same as in the table for the Deepthought clusters above.

    If you have any external libraries you need to use, you need to module load or tap these as well. Some libraries have specific versions compiled with and for a specific compiler/MPI library combination; in such cases you need to pick a version which matches what you are using. Not all combinations exist; if yours does not you can submit a help ticket requesting that combination. We generally try to avoid doing this for old versions of compilers/packages/etc. unless there are extraordinary reasons, so you are generally advised to try to use the latest versions available on the system. Fortran90 codes are particularly sensitive to this, and the *.mod files between different versions of the same compiler might not be compatible (and are definitely not compatible across compilers).

    For packages which are libraries used by other packages (e.g. LAPACK, NETCDF, etc), the module load command is not enough. The package being compiled needs to know where to find the libraries, and the way to inform the package of that depends on the build system used by the package.

    Doing a module load or tap for an external library generally only defines some environmental variables for you, you still need to instruct the compiler where to find any needed include files and where to find the libraries. Generally, "module help MODULENAME" or "tap TAPNAME" will give a brief usage summary giving the names of these variables. Typically there is a variable with a name like "*INC" or "*INCDIR", and another one with a name like "*LIB" or "*LIBDIR". E.g. the netcdf package defines NETCDF_INCDIR and NETCDF_LIBDIR. The package fftw/3 on the Deepthought clusters defines FFTWINC and FFTWLIB.

    The "INC" variable provides the location of the include directory for the package. You will generally want to either add arguemnts with these variables preceded by the -I flag to your compilation command. The -I flag takes a single path, so you should repeat it if you have multiple library packages that you are using.

    login-1:~: gcc -c -I$NETCDF_INCDIR -I$FFTWINC my_netcdf_code.c
    or to the CFLAGS and FFLAGS variables in your Makefile, e.g.

    The "LIB" variables work similarly, except these are needed when the compiler/linker creates the executable. For small codes, the compilation and linking usually occur in a single step; larger codes, especially those with makefiles, usually break this into separate steps. Also, the dynamic linker needs to know where the libraries files are when you run the code, so it is usually easiest to set the "rpath" during compilation. (Otherwise you will need to set the environmental variable LD_LIBRARY_PATH each time before you run the code). To tell the compiler where to find the library for linking, you provide the "LIB" variables as arguments to the "-L" flag. To set the rpath, you provide them as arguments to the "-Wl,-rpath," flag. Both the -L and -Wl,-rpath, take a single path, so you should repeat these flags for each library package.

    For the simple case, you would do something like

    login-1:~: ifort -o mycode -I$NETCDF_INCDIR -I$FFTWINC 
    	-Wl,-rpath,$NETCDF_LIBDIR  -Wl,-rpath,$FFTWLIB my_netcdf_code.f90

    Since this compiles and links my_netcdf_code.f90 in a single step, we need to provide both the compile stage (-I) flags and the link stage (-L and -Wl,-rpath) flags.

    More complicated cases typically use makefiles, and here you typically will just do something like:

    	LDFLAGS+= -L$(FFTWLIB) -Wl,-rpath,$(FFTWLIB)

    Here we have the CFLAGS and FFLAGS definitions from above (which will be used in the compilation stage), and we put the -L and -Wl,-rpath flags in the LDFLAGS variable (we did this in two steps to make it more readable).

    If you opt not to go the "rpath" route, and instead compile the code with something like


    (note that the -Wl,-rpath arguments are missing in the above), then before running the resulting binary (which we will call myprog) you will need to set LD_LIBRARY_PATH appropriately. This (and the module loads which precede it) will need to be done in every interactive or batch session which plans to run the code. E.g.

    login-1: echo $LD_LIBRARY_PATH
    LD_LIBRARY_PATH: Undefined variable.
    login-1: ./myprog
    ./myprog:  error while loading shared libraries: cannot open shared object file: No such file or directory
    login-1: module load fftw/2.1.5
    login-1: module load netcdf
    login-1: setenv LD_LIBRARY_PATH "${FFTWLIB}:${NETCDF_LIBDIR}"
    login-1: ./myprog
    Program runs successfully

    If you do NOT use the "rpath" arguments shown earlier, every time you run the program the variable LD_LIBRARY_PATH must be properly defined to point to the library locations, or you will get an error like shown above. In general, you will need to load the modules and issue the setenv command once in every interactive login session in which you will use it, and once in every batch script. And you MUST set the directories correctly; if you, e.g., give the $FFTWLIB path for a different version of FFTW than the one the code was compiled with, the binary might run, but it might crash with a difficult to debug error at some seemingly arbitrary place. Or perhaps even worse, it might run to a seemingly successful conclusion but produce erroneous output.

    We strongly recommend that for your own code, or for code that you are compiling, that you use the rpath arguments shown earlier. The LD_LIBRARY_PATH option is, in our opinion, best used when you do not have the option of compiling the code with the rpath settings.

    Optimization of Code

    Since you are using a High Performance Computing cluster, you most likely have long, compute intensive jobs. These can generally benefit greatly from various optimization techniques. The general topic of code optimization is quite broad, far too large to give more than a cursory discussion here. You are encouraged to look at the myriad of resources on the topic that exist on the web. Here we just discuss some specific details related to the UMD clusters.

    A major performance feature of modern processors is the ability to vectorize certain operations. In this mode, an instruction is decoded once and operates on a stream of data. This allows for significant performance boosts. There are various vectorization command sets available on Intel processors, known by names like SSE and AVX (with numerical suffixes to specify the version of the command sets). Selecting the correct optimization level can be tricky, as different processors support different vectorization command sets.

    Things are relatively simple on the Deepthought2 and Bluecrab clusters, since these clusters are fairly homogeneous. All nodes on the Deepthought2 cluster support SSE3, SSSE3, SSE4.1, SSE4.2, and AVX. This includes the login nodes. All nodes on Bluecrab support all of those as well as AVX2 since they are a bit newer than the DT2 nodes.

    Things are trickier on the original Deepthought cluster, as it grew over time and is quite heterogeneous. All of the original Deepthought cluster nodes support SSE3 and SSSE3, but some support one or more of SSE4.1, SSE4.2 and AVX. In particular, the login nodes have newer processors (and therefore support more vectorization command sets) than some of the compute nodes, which can cause issues if the Makefiles, etc. try to optimize aggressively based on the capabilities of the compilation node --- the resulting code might not work on all of the DT1 compute nodes. For best results, you might wish to throttle optimization to the least capable processors on the cluster (e.g. SSE3 and SSSE3 only).

    The following table shows the breakdown by cluster:

    ClusterProcessor TypesNode count SSE3 ?SSSE3 ? SSE4.1 ?SSE4.2 ? AVX ?AVX2 ?
    Bluecrab E5-2680v3 (Haswell-EP) ~700 YYYYYY
    E5-2690v4 (Broadwell) ~125 YYYYYY
    E7-8857v2 (IvyBridge-EP) ~50 YYYYYY
    Deepthought (DT1) E5130 (Woodcrest) ~40 YYNNNN
    E5150 (Woodcrest) ~89 YYNNNN
    E5345 (Clovertown) ~39 YYNNNN
    E5410 (Harpertown) ~81 YYYNNN
    E5520 (Gainestown) ~16 YYYYNN
    E5650 (Westmere-EP) ~4 YYYYNN
    E5670 (Westmere-EP) ~78 YYYYNN
    E5-2660 (Sandybridge-EP) ~22 YYYYYN
    E5-2670 (Sandybridge-EP) ~8 YYYYYN
    Deepthought2 (DT2) E5-2680v2 (Ivybridge-EP) ~490 YYYYYN
    E5-4640v2 (Ivybridge-EP) ~5 YYYYYN

    As previously stated, it is sometimes useful to specify the processor type and/or vectorization command sets to compiler/optimize for. When specifying processor type, we recommend that one uses an option which generates code which will run on any processor but is optimized for the expected processor type (instead of code which is optimized for a specific processor but might not run on other processors). I.e., use -mtune instead of -march.

    Some useful optimization options:

    Optimization TypeGCC FlagsIntel Flags
    SSE3 -msse3 -mSSE3
    SSE4.1 -msse4.1 -mSSE4.1
    SSE4.2 -msse4.2 -mSSE4.2
    AVX -mavx -mAVX
    AVX2 -mavx2 -mAVX2
    Optimize for sandybridge
    (but run anywhere)
    Optimize for ivybridge
    (but run anywhere)
    Optimize for broadwell
    (but run anywhere)

    Compiling with OpenMP

    OpenMP is an API for shared memory parallelization, i.e. for spliting a code into multiple threads running on the same node. Because the parallelization is limited to a single node, it is less powerful than other APIs (e.g. MPI) which can span multiple nodes, but is also much easier to code for.

    Indeed, OpenMP is implemented by the compiler, and is generally invoked through various compiler directives and/or pragmas. E.g., if you have a C code with a for loop, you can add a pragma just before the start of the for loop which will tell the compiler to try to split the loop into multiple threads each running in parallel on a different processor core.

    If you have a code with OpenMP directives, you need to tell the compiler to compile it with the OpenMP extensions enabled. The exact mechanism is compiler dependent:

    Compiler FamilyFlag to use OpenMPDefault number of threads
    (if OMP_NUM_THREADS not set, etc)
    GNU compiler suite-fopenmpnumber of available cores
    Intel Compiler Suite-openmpnumber of available cores
    PGI Compiler Suite-mp1 thread

    NOTE: If you are using the Intel compiler suite and the Math Kernel Libraries (MKL), some of the MKL routines might use OpenMP even if you did not request OpenMP in the main compilation. You can set the environmental variable OMP_NUM_THREADS to 1 to effectively disable that if really desired.

    By default, OpenMP will attempt to use as many threads as there are cores on the system (except with PGI cmopilers, which default to one thread). This can be problematic in some cases. At runtime, you can set the environmental variable OMP_NUM_THREADS to an integer to control the maximum number of threads that OpenMP will use.

    NOTE: Be sure to use the -openmp or -fopenmp flag (as appropriate for the compiler using) on all; of the compilation AND link stages for your code.