Skip to main content

Using the HPC clusters: The Basics

This page covers some preliminary information about using the environment on the HPC clusters. I.e., the stuff you need to know how to do before we can even start talking about submitting jobs to the system using the command line.

Note: Users who are uncomfortable or unfamiliar with the Unix/Linux command line should consider using the OnDemand Web portal for accessing the cluster. While the command line is the more powerful interface to the cluster, the OnDemand portal has a gentler learning curve. The OnDemand portal is only currently available for the Zaratan and Juggernaut clusters.

  1. Logging into the system
    1. Multifactor Authentication
    2. Common ssh warning/error messages
    3. Ssh Fingerprints
    4. Setting up ssh public-key authentication
      1. Setting up public-key authentication for passwordless ssh among the nodes of the HPC cluster.
      2. Setting up public-key authentication for passwordless ssh into the HPC cluster from your workstation, etc.
  2. Graphics
  3. Basic unix commands
    1. Changing your default shell
  4. Setting up your environment
    1. Preventing output from your dot files
  5. Files and storage basics
  6. Transferring files to/from the HPCCs and other systems
    1. Using scp/sftp
    2. Using Globus
    3. To/from Cloud Storage Providers
  7. Compiling codes
    1. Optimization of Code
    2. Compiling with OpenMP

Logging into the system

Each cluster has at least two nodes available for users to log into. From these login nodes, you can submit and monitor your jobs, compile codes, look at the results of jobs, etc. These nodes can also be used for transferring files and data to/from the cluster and other systems.. However, for large data transfers, there are data transfer nodes (listed below which should be used instead of the login nodes).

WARNING
DO NOT RUN computationally intensive processes on the login nodes!!!. These are in violation of policy, interfere with other users of the clusters, and will be killed without warning. Repeated offenses can lead to suspension of your privilege to use the clusters.

For most tasks you will wish to accomplish, you will start by logging into one of the login nodes for the appropriate cluster. To log into the cluster, you need to use the Secure Shell protocol (SSH). This is usually standardly installed as ssh on Unix systems, and clients are available for Windows, Mac, and even Android, however on non-Unix systems you typically must install an SSH client .

Once you have an ssh client is installed on your system, you just tell it you wish to connect to the one of the login nodes for the cluster desired. Assuming your official UMD email address is johndoe@umd.edu,

Cluster Login Node Username
Zaratan login.zaratan.umd.edu johndoe
Juggernaut login.juggernaut.umd.edu johndoe
WARNING
NOTE: Zaratan requires multifactor authentication (MFA) using the standard campus DUO MFA system. Either you must be on the standard campus VPN (which requires MFA to authenticate), or when you ssh you will get prompted to enter your passcode or a single digit to send a "push" to a phone after entering your password,

On a TerpConnect/Glue system, you would just issue the command ssh login.zaratan.umd.edu to connect to Zaratan, or similarly ssh login.juggernaut.umd.edu to connect to Juggernaut. The unix ssh command by default assumes your login name on the remote system is the same as on the local system, which is true for the UMD HPC clusters and for TerpConnect/Glue systems. From other Unix systems, you might need to specify your cluster username, e.g. sshUSERNAME@login.zaratan.umd.edu or ssh -l USERNAME login.zaratan.umd.edu. , where USERNAME is your Zaratan username.

Multifactor Authentication

Starting with Zaratan we are requiring multifactor authentication to access the HPC clusters. We are using the standard campus DUO MFA system.

Since the standard campus VPN does multifactor authentication, if you are on the VPN, ssh connections to the login nodes will not prompt you for multifactor --- you just need to enter your password as before.

If you are not on campus VPN, when you ssh to one of the Zaratan login nodes, you will first be prompted for your password and then you will be prompted to enter a passcode or a single digit from a menu for a "push" or phone call for verification. E.g., you will see something like the session below, and at the passcode prompt you can enter a passcode from the Duo app on your mobile phone, or have Duo send a push or make a phone call to a previously registered device for the second authentication factor.

For more information, see the web page on the campus Duo Multifactor Authentication System.

my-workstation:~: ssh login.zaratan.umd.edu

                              * * * WARNING * * * 

   Unauthorized access to this computer is in violation of Md.
   Annotated Code, Criminal Law Article sections 8-606 and 7-302 and the 
   Computer Fraud and Abuse Act, 18 U.S.C. sections 1030 et seq. The University 
   may monitor use of its computing resources as permitted by state 
   and federal law, including the Electronic Communications Privacy Act, 
   18 U.S.C. sections 2510-2521 and the Md. Annotated Code, Courts and Judicial 
   Proceedings Article, Section 10, Subtitle 4.  Anyone using this system 
   acknowledges that all use is subject to University of Maryland Policy 
   on the Acceptable Use of Information Technology Resources available at 
   http://www.umd.edu/aup.

   By logging in I acknowledge and agree to all terms and conditions
   regarding my access and the information contained therein.

To report problems or request assistance call the Help Desk at 301-405-1500

Password: 
Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-1234
 2. Phone call to XXX-XXX-1234
 3. Phone call to XXX-XXX-4444

Passcode or option (1-3):

The first time you connect to the system via ssh, you might receive a warning message like (color added to assist in discussion below):

The authenticity of host 'login.zaratan.umd.edu' can't be established.
# RSA key fingerprint is e8:41:71:ac:fc:4c:08:c7:bc:0f:f0:33:95:5b:c4:e0
# Are you sure you want to continue connecting (yes/no)?

The message sounds scary, but it is normal. SSH tries very hard to protect you from all sorts of hacking, and one such mechanism is to try to ensure you are talking to the system you think you are talking to. For every server you connect to, it remembers a secret (RSA fingerprint) to prove the identity of the server, and verifies that (for brevity, this is a gross oversimplification; for more information). But it cannot verify it the very first time you connect (unless, as is the case on some campus systems, systems staff have pre-populated SSH with information about the system you are connecting to). This message is just to inform you about that.

The IP address and the hostname (in green above) may vary, although the hostname should match the name of the system you want to connect to. The parts in red (the key type and the fingerprint) will depend on the system you are trying to communicate with. To be secure, you should verify that it matches one of the fingerprints listed below:

If the fingerprint does NOT match, you should NOT enter your password and contact system staff ASAP. If you enter your password in such a situation, as it is possible that someone is performing a man-in-the-middle attack and can obtain your password when you enter it.

If you see a message like

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
5c:9b:16:56:a6:cd:11:10:3a:cd:1b:a2:91:cd:e5:1c.
Please contact your system administrator.

you should always refrain from logging in and contact system staff as soon as possible. This means that the server you are connecting to did not know the secret remembered by SSH for that system, as described above, which means either system staff changed the keys, or someone is hacking you. As system staff do not change the keys often, and will send email to everyone well in advance of changing the keys warning you of this if we were to, this likely means someone is attacking you unless you received an email from systems staff. Do NOT enter your password and contact system staff.

Setting up ssh public-key authentication

WARNING
Note that the SHELL medium tier storage on Zaratan uses the Auristor file system which uses Kerberos authentication protocol . If you use SSH public-key based authentication, you will not be given Kerberos tickets and Auristor tokens on login. Because of that, you will not be able to access the SHELL storage system until you obtain Kerberos tickets/Auristor tokens.

This section discusses how to setup and use SSH with public-key based authentication to allow for access to the cluster without typing your password for every ssh session. It can also be used to allow passwordless ssh between the various login and compute nodes; this is useful if you are using ssh to spawn processes on the different nodes allocated to your job.

The procedures listed in this section are NOT required for you to access the system, and you can use normal password based authentication instead. It also is NOT for most multinode jobs (using MPI or srun). It is only for users who wish to set up public-key authentication, either to allow passwordless access to the cluster from your workstation, or to allow passwordless ssh between the nodes of the cluster.

If you are new to Unix/Linux, it is recommended that you skip this section and just use password based authentication.

Public-key authentication uses asymetric encryption for its security. In asymmetric cryptographic systems, there exist distinct private and public keys. Data can be encrypted and/or digitally signed with the private key and can only be decrypted/signature verified with the public key. The private key is kept secure on your workstation, and the public key is copied to the HPC cluster. When you log in with public key encrpytion from your workstation, the ssh client on your workstation will digitally sign a standard message and send this to the sshd server process on the cluster along with the appropriate public key. The remote sshd process verifies that the public key is authorized, and if so verifies if the digital signature is valid. If the signature is valid, that means the ssh client is in possession of the private key corresponding to the public key, and because the public key is authorized, grants it access to the system.

WARNING
NOTE: the public-key authentication process grants login access to anyone/any system/etc. that knows your private key. Thus you need to ensure the security of your private key. Protect your private key at least as strongly as you would your password.

Setting up public-key authentication with ssh

Instructions for setting up public key authentication is discussed below. We break it down into two cases:

  1. for passwordless ssh among the nodes of the cluster
  2. for passwordless ssh from your workstation to the login nodes of the cluster

Although the process is essentially the same in each case, because of the shared home directory among the nodes of the cluster, the first case is a bit simpler and will be treated separately. The second case is slightly more complicated because there are steps which need to be done on both your workstation and on the HPC cluster.

Depending on your needs, you can do none of the steps, either one of the step, or both of them.

Setting up public-key authentication for passwordless ssh among the nodes of the HPC cluster

In certain cases, it might be necessary to enable passwordless ssh between the nodes of the cluster. The most typical such case is if you must use the ssh command to launch processes on multiple nodes as part of your job. (Most multinode jobs use MPI and/or the srun command, and so do not require this, but some may.) Because you cannot feasibly enter a password with a batch script, you need to enable passwordless ssh in such cases.

Because your home directory is shared among all nodes in the HPC cluster, everything for this process can be done on one of the HPC login nodes. Just log into a login node on the cluster, and then:

  1. Generate your host key, if needed.
    1. Run the command ls -l ~/.ssh/id_rsa.
    2. If the ~/.ssh/id_rsa already exists, you should already have the public host keys and should not need to do anything else and can skip the running the ssh-keygen command, although you should still verify the ownership and permission of the file.
    3. If you are having problems with passwordless among the cluster nodes, you can safely regenerate the public host key with the next step. If you arranged for passwordless ssh from the cluster to other systems, you will likely need to fix the authorized_keys file on the other systems if you regenerate the HPC host key.
    4. To (re-)generate the host key, run the ssh-keygen command. Accept all of the default value for the name of the file in which to save the key; the public key will be stored in the same directory, with a .pub extension. Generally, you should not enter a passphrase (just hit return at the two passphrase prompts) --- the most common case is to enable passwordless ssh among nodes in the cluster for use in job scripts, and that will not work if the key has a passphrase.
    5. Repeat the ls -l ~/.ssh/id_rsa command from above. It should now show the existance of the file. Please verify that the file is owned by you, and no one can access it but you. (Permission flags should be -rw-------.).
  2. Add the public host key to the authorized_keys file.
    1. If ~/.ssh/authorized_keys already exists, you should append the contents of ~/.ssh/id_rsa.pub to the file; you can do this with the command 'cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys'.
    2. If the authorized_keys file does not exist, yo can create it with the proper contents with the command: 'cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys'.
WARNING
Please ensure that your id_rsa file is only readable by you. This host key is all that is needed to access any system which has the corresponding public key added to its authorized_keys file.

You should now be able to ssh between nodes in the cluster without entering a password. Note that you do not have access to ssh to compute nodes unless you have a job currently running on the node. Typically this passwordless ssh is only needed to ssh between nodes allocated to a job within your job script.

Setting up public-key authentication for passwordless ssh into the HPC cluster from your workstation, etc.

Atlhough the process is in this case uses basically the same steps as above, because your workstation and the HPC cluster do not share a common home directory, the different steps need to be performed on different systems, and there is a file transfer required.

Some of the steps below are dependent on the ssh application being used on your workstation/desktop/laptop/etc. We are assuming putty on Windows systems and the standard command line openssh client on Macs or Linux systems. If you are using something else, hopefully it has some documentation describing how to set up public key authentication and which together with the instructions belong can be used to figure out what you need to do.

  1. First, you need to generate a private-public key pair. On
    • Windows systems: you should use the PuTTYgen utility to generate the key pair. Open the PuTTYgen utility, and
      1. At the bottom of the PuTTY Key Generator window, in the Parameters section, select the type of key to generate. It is recommended to use "RSA" (perhaps called "SSH-2 RSA" in some older versions) using the default (2048) number of bits.
      2. Click Generate in Actions section. You will be prompted to use your mouse/etc. to generate entropy that will help make the private key secure. Move the cursor around until the utility has generated the key (it will display in the area under Key).
      3. If desired, you can enter a passphrase to be used in encrypting the private key in the Key passphrase and Confirm passphrase boxes. Encrypting the private key with a passphrase will increase security, and with the Pageant utility provided with PuTTY you only need to enter the password once per login session on your workstation.
      4. Save the generated public key by clicking Save public key under Actions (next to Save the generated key). Enter the name (e.g. putty_public_key) and folder, and click Save.
      5. Save the generated private key by clicking Save private key under Actions (next to Save the private key). If you did not opt to encrypt the private key, it will ask if you are sure that you wish to save an unencrypted private key. "Yes" will proceed to save; "No" will allow you to go back and specify a passphrase to use. Use the default format/Save as type (PuTTY Private Key Files (*.ppk)), enter the name (e.g. putty_private_key) and folder, and click Save.
    • Mac or Linux systems: you should use something like ssh-keygen -t rsa.
      1. This will prompt you for a filename to save the private key to: the default is id-rsa in the .ssh subdirectory of your home directory. It is recommended that you use this default otherwise you need to tell the ssh or ssh-agent command which identity file to use when logging in.
      2. The public key will be placed in the same directory as the private key, with the same name but with a .pub suffix added.
      3. You will also be prompted for a passphrase to encrypt the private key with. Encrypting your private key with a passphrase will increase security, and with the use of ssh-agent (described below) you only need to enter the passphrase once per login session on your workstation. If you opt not to use a passphrase, just hit return without entering anything to leave the private key unencrypted.
      4. You will be prompted a second time for the passphrase (to ensure it was entered correctly). Type the passphrase from the previous step, or just hit return again if you opted for no encryption of private key.
  2. Next, you need to authorize that key to log into the HPC cluster (e.g. Zaratan) as your identity. This will require logging into the cluster using your password because the public key authentication is not set up yet.
    1. Use a scp or sftp client to transfer the public key file created in the previous step to a Deepthought2 login node. For linux or Mac users that kept the default name, the public key file will be ~/.ssh/id_rsa.pub. For Windows users, it will be the name used when you saved the public key (e.g. putty_public_key). DO NOT transfer the private key file --- that should remain in a protected spot on your workstation.
    2. Ssh to a Deepthought2 login node using password authentication.
    3. Use the command mkdir -p ~/.ssh to create the .ssh directory under your home directory if it does not already exist (the command will not harm anything if it does already exist).
    4. Use the command touch ~/.ssh/authorized_keys. If there is no authorized_keys file in the .ssh subdirectory of your home directory, this command will create an empty file. If there was one, it does not change the contents of the file.
    5. Use the command chmod 600 ~/.ssh/authorized_keys to ensure the proper permissions on the file. No one but you should be able to read or write to the file.
    6. Use the command cat PUBLIC_KEY_FILE >> ~/.ssh/authorized_keys to append the public key file to the authorized_keys file. Be sure to use TWO > characters without space between them in the above command (otherwise you might overwrite the file and lose previous contents). The PUBLIC_KEY_FILE in the above command should be replaced by the name of the public key file you just copied to the cluster; e.g. id_rsa.pub for Linux or Macs or putty_public_key or whatever you saved as on Windows.
    7. Delete the public key file you copied to the cluster using rm PUBLIC_KEY_FILE as it is no longer necessary.
WARNING
NOTE: Your private key is all that someone needs to access the cluster as you. KEEP THE PRIVATE KEY FILE SECURE. It is strongly suggested that you encrypt the private key file with a passphrase, so that both the passphrase and the file are needed to access the cluster.

You should now be able to ssh to the Zaratan cluster from your workstation using public key authentication, as described below.

Using public-key authentication with ssh

In this section we discuss how to use public-key authentication with ssh. Because it is good practice to encrypt your private key, typical usage involves starting an agent that runs in the background. You start the agent when you log into your workstation, and it loads your un-encrypted private key into memory at that time (it will need to ask you for the passphrase to decrypt it if it was encrypted). Then for the rest of the time you are logged into your workstation, it will provide the key to the ssh client so that you can connect to the cluster without providing a password. The agent differs according to what ssh client you use.

WARNING
NOTE: If you login to Zaratan using public-key authentication, you will not have Kerberos tickets or AFS tokens, so certain commands will not work. For example, you will not be able to access the contents of your SHELL directory. The HPC clusters are much less dependent on your having Kerberos tickets and AFS tokens than standard Glue/TerpConnect systems, so this is usually not a significant issue. If you do find yourself needing such, you can just issue the renew command (which will prompt you for your password).

Using public-key authentication on Windows systems

On windows systems, should be a Pageant SSH authentication agent installed with PuTTY.

  1. First, ensure that PuTTY is configured to use Pageant. This is the default, but just to be certain:
    1. Open PuTTY and in the left hand "Category" panel click on the small plus (+) next to SSH under Connection to expand the SSH options.
    2. Click on the Auth subtree that now appears under SSH
    3. Ensure that the box "Attempt authentication using Pageant" is clicked.
    4. Exit PuTTY
  2. Open the Pageant SSH authentication agent. It runs in the background, so when it is open you will just see a new icon (a computer wearing a hat) for it in the Windows notification tray. Double click on that icon to open up the Pageant window.
  3. The main window will list the keys Pageant is holding, which is probably none at this time. Press the Add Key to add the private key created in the previous section. This will bring up a file dialog, so find the private key you created (e.g. putty_private_key.ppk) and "open" it.
  4. If the key was encrypted using a passphrase, you will be prompted to enter the passphrase now. You will not need to enter it again as long as Pageant is running.

You can now use PuTTY to login into the Deepthought2 login nodes as before, and it will use public key authentication and not ask for your password on the cluster.

You might wish to have Pageant start up whenever you login to your Windows workstation. To do that:

  1. Right-click inside the Startup folder, and select New and Shortcut.
  2. In the Type the location of the item box, you should enter the path to pageant.exe followed by a space and the full path to your private key (e.g. putty_private_key.ppk). You should put both paths (the executable and the private key) in double quotes.
  3. Then click Next and enter a name for the shortcut (e.g. Pageant).

E.g.

"C:\Program Files (x86)\PuTTy\pageant.exe" "C:\Users\user_profile\ssh_key\putty_private_key.ppk"
All of that should be on a single line.

Then, every time you log into the Windows workstation, it will start Pageant (prompting you for the encryption key for the private key if it is encrypted) and you can use PuTTY to ssh to the Deepthought2 login nodes without entering any additional passwords for the remainder of your workstation session.

Using public-key authentication on Linux or Mac systems

On Linux or Mac systems, there are ssh-agent and ssh-add commands that should be standard. You can start the background agent by issuing the command ssh-agent. You can then add private keys to the agent using the ssh-add command. If you use the default location for your private key (e.g. ~/.ssh/id_rsa) you can just issue the command ssh-add and it will add any private keys in the default locations. If you added the key in a non-standard path, just use ssh-add PATH_TO_PRIVATE_KEY where PATH_TO_PRIVATE_KEY is the path to the private key file to use. If the private key is encrypted, the ssh-add will prompt you for the passphrase to decrypt it.

At this point, you can just ssh USERNAME@login.zaratan.umd.edu (where USERNAME is your username on the Zaratan cluster) and you will be logged into a login node using public key authentication; i.e. no additional password needed.

You can combine the ssh-agent and ssh-add commands into a single shell script if you want, or add to your .cshrc or .bashrc initialization scripts if you want them to start automatically when you login to your workstation.

Graphics

The login nodes are capable of running graphical applications with the graphics appearing on your desktop using the network features of the X11 Windowing System.

In order for this to work, however, you need to be running an X11 server on your desktop. The exact procedure depends on what your desktop, etc. is, and you might wish to contact the UMD helpdesk for assistance.

Some software packages have their own protocols for remote visualization, including:

Basic unix commands

This is a very short list of some very basic Unix commands to help you get started in the HPC environment.

Hopefully, the above list will get you through most of the basic tasks you need to complete. A full tutorial on using Unix is beyond the scope of this document. However, there are many tutorials for beginning to use Unix on the web. A few are tutorials we recommend are:

Changing your default shell

The shell is the command line prompt that you interact with. It also is what processes shell scripts. Although there are a number of shells available on DIT maintained Unix systems, the most common are tcsh and bash. Your default shell is the shell you get when you log into a system; by default on the Zaratan cluster it is set to /bin/bash. (NOTE: This is a change from the previous clusters, e.g. Deepthought2)

To change your default shell on Zaratan, currently you need to submit help ticket to HPC staff; be sure to include your username on the cluster and the new login shell you wish to use. At some point in the future, we hope to implement a process to allow you to change your shell without administrator intervention.