Lftp

Contents

• Summary and Version Information
• General usage
• Using with UMD Box

Summary and Version Information

Notes:
^*: Packages labelled as "available" on an HPC cluster means that it can be used on the compute nodes of that cluster. Even software not listed as available on an HPC cluster is generally available on the login nodes of the cluster (assuming it is available for the appropriate OS version; e.g. RedHat Linux 6 for the two Deepthought clusters). This is due to the fact that the compute nodes do not use AFS and so have copies of the AFS software tree, and so we only install packages as requested. Contact us if you need a version listed as not available on one of the clusters.

In general, you need to prepare your Unix environment to be able to use this software. To do this, either:

• tap TAPFOO

• module load MODFOO

where TAPFOO and MODFOO are one of the tags in the tap and module columns above, respectively. The tap command will print a short usage text (use -q to supress this, this is needed in startup dot files); you can get a similar text with module help MODFOO. For more information on the tap and module commands.

For packages which are libraries which other codes get built against, see the section on compiling codes for more help.

Tap/module commands listed with a version of current will set up for what we considered the most current stable and tested version of the package installed on the system. The exact version is subject to change with little if any notice, and might be platform dependent. Versions labelled new would represent a newer version of the package which is still being tested by users; if stability is not a primary concern you are encouraged to use it. Those with versions listed as old set up for an older version of the package; you should only use this if the newer versions are causing issues. Old versions may be dropped after a while. Again, the exact versions are subject to change with little if any notice.

In general, you can abbreviate the module tags. If no version is given, the default current version is used. For packages with compiler/MPI/etc dependencies, if a compiler module or MPI library was previously loaded, it will try to load the correct build of the package for those packages. If you specify the compiler/MPI dependency, it will attempt to load the compiler/MPI library for you if needed.

General Usage

Lftp operates rather similarly to the old command line ftp utility. You should do a module load lftp first, and then connect to a remote site using lftp -u REMOTE_USERNAME REMOTE_SITE. You will be prompted for a password if needed, and then you will enter the interactive lftp prompt. At this point you can enter any of the many commands supported. The help command shows a quick overview of all the commands, we will discuss a few of the most commonly used ones below: (in the following, local site means the system on which you invoked the lftp command, and remote site is the site you specified with with REMOTE_SITE argument to the lftp command):

• help: will print out a short list of all available commands. You can use help CMD for some more information on a specific command CMD
• ls: will list the files in the current directory on remote site, or specific files
• cd DIR: will change the current directory on the remote site to DIR
• lcd DIR: will change the current directory on the local site to DIR
• get FILE: will download the file FILE from the remote site to the local site (using current directories on each if not otherwise specified)
• mget FILE_PATTERN: like get, will will download all files matching FILE_PATTERN, which can contain wild cards (*, etc).
• put FILE: will upload the file FILE from the local site to the remote site.
• mput FILE_PATTERN: like put, but uploads all files matching the FILE_PATTERN.
• mirror REMOTE_DIR LOCAL_DIR: this will mirror (i.e. copy/delete/etc files to make the target directory look like the source directory). This subcommand has lots of options controlling various aspects, including whether the source/target directories are the remote/local directories or the local/remote directories. I suggest reading the man page before using it.
• quit: exits lftp

The following example shows some of the commands for accessing Box:

login-2:~> module load lftp
login-2:~> lftp -u USERNAME@umd.edu ftp.box.com
Password: Enter your Box external application password here
lftp USERNAME@umd.edu@ftp.box.com:~> help
    !                     (commands)
    alias [ []]             attach [PID]
    bookmark [SUBCMD]                    cache [SUBCMD]
    cat [-b]                      cd 
    chmod [OPTS] mode file...            close [-a]
    [re]cls [opts] [path/][pattern]      debug [OPTS] [|off]
    du [options]                   edit [OPTS] 
    exit [|bg]                     get [OPTS]  [-o ]
    glob [OPTS]               help []
    history -w file|-r file|-c|-l [cnt]  jobs [-v] []
    kill all|                    lcd 
    lftp [OPTS]                    ln [-s]  
    ls []                          mget [OPTS] 
    mirror [OPTS] [remote [local]]       mkdir [OPTS] 
    module name [args]                   more 
    mput [OPTS]                   mrm 
    mv                     mmv [OPTS]  
    [re]nlist []                   open [OPTS] 
    pget [OPTS]  [-o ]     put [OPTS]  [-o ]
    pwd [-p]                             queue [OPTS] []
    quote                           repeat [OPTS] [delay] [command]
    rm [-r] [-f]                  rmdir [-f] 
    scache []                set [OPT] [ []]
    site                       source 
lftp USERNAME@umd.edu@ftp.box.com:~> ls
drwx------   1 owner group            0 Apr  4 12:24 testDirectory
-rw-------   1 owner group           24 Apr  4 13:26 test.file
-rw-------   1 owner group           24 Apr  4 13:51 test.file2
-rw-------   1 owner group           23 Apr  4 13:51 test.file3
lftp USERNAME@umd.edu@ftp.box.com:~> mget test.file*
71 bytes transferred in 1 second (50 B/s)           
Total 3 files transferred
lftp USERNAME@umd.edu@ftp.box.com:~> mput newdata.2019-03-*.csv
12 bytes transferred in 2 seconds (5 B/s)                      
Total 2 files transferred
lftp USERNAME@umd.edu@ftp.box.com:~> quit
login-2:~> 

Using with UMD Box

One of the protocols supported by UMD Box is ftps, making lftp a good choice for connecting to box. We discuss a few aspects of using lftp with Box here. See the UMD Box service catalog entry for more information on the Box service, including restrictions on what data can be stored.

When connecting to UMD Box over the ftps protocol, the standard campus directory name/password single-sign-on cannot be used, so you will need to set up in Box a password for external applications. To do this:

1. From a web browser on your desktop/laptop/etc, go to umd.box.com and log in using the UMD single sign-on (your standard UMD campus directory ID and password).
2. In the upper right is an icon (usually either your image or initials) with a drop down menu; open the menu and select "Account Settings".
3. Under the "Account" tab, there is a section called "Authentication". There should be a link to the right entitled "Create Password" (if you already created your external password, the link will be called "Change Password". You can use this if you forgot what your external password is).
4. Click the "Create Password" link and enter your password for accessing Box. Use a different password than your main UMD and other passwords.
5. Click on "Save Changes" near the top right.

Do not re-use your main UMD directory single-sign-on or any other password for your UMD Box external application password. Re-use of passwords is a security risk, since if you reuse passwords and hackers compromise one site they can then leverage that to compromise additional sites.

After your external application Box password is set up, you can then access Box with a command like lftp -u USERNAME@umd.edu ftp.box.com.