Whatbox Logo
Login for certain variables to be updated with your slot's information

Wiki > lftp

Disclaimer: The process in this article may seem complicated and slow, but it is designed for you to learn the basics first. The final section of the article is dedicated to making things easier and faster for everyday use. So, please don't give up early.

lftp 4.4.7 has a bug that causes an endless loop when using the mirror command with sftp. Make sure your version of lftp is higher than 4.4.7.

Installing

Windows

Install lftp, OpenSSH, and OpenSSL from the Net category in Cygwin

Linux

Install lftp with your package manager, something like: sudo apt-get install lftp

Mac OS X

Using the Homebrew package manager you can: brew install lftp

lftp on your Whatbox slot

lftp is installed on your Whatbox slot by default. You can use it to transfer between your slot and other FTP/SFTP servers (e.g. another seedbox). Just connect to your slot using SSH and use it as normal.

Connecting to Whatbox

  1. Open your terminal (Cygwin Terminal for Windows users) and Start lftp and connect to Whatbox. Type lftp sftp://user@server.whatbox.ca. Enter your password at the prompt. lftp keeps your password hidden, so it's easiest to paste with the right-click menu.

    lftp sftp://user@server.whatbox.ca
 Password:

  2. Now type ls in the terminal to show a listing of your home directory. Since most users will want to start in their files directory, type cd files to navigate there for the next step.

    ls
 drwxrwx---   2  user    user        4096 Aug 29 01:35 files
 drwxr-x---   2  user    user           6 Jul 30  2011 watch
    cd files

    Note: If your 'ls' hangs at "Connecting ..." you can enable debug mode to determine the issue, but it is most likely that the SSH host key is not trusted.

    There are 2 ways to fix this:

    • nano ~/.lftprc and add the line set sftp:auto-confirm yes or
    • exit lftp via quit, manually connect to your whatbox instance one time via ssh user@server.whatbox.ca, and accept the SSH host key. Then try the lftp connection again.

  3. (optional) If you want your password to be saved when you create a bookmark, use the command set bmk:save-passwords true now.
    bookmark add Whatbox will save a bookmark to your slot named Whatbox. You can use bookmark list to make sure it saved properly. This allows you to use lftp Whatbox to open a connection to your slot instead of having to type out the address.

    set bmk:save-passwords true
    bookmark add Whatbox
    bookmark list
 Whatbox sftp://user:XXXX@server.whatbox.ca/home/user/files/

Local and Remote Navigation

Note: I have set up directories and files for demonstration. You can set things up like I have for practice, if you like, or replace them in the commands with your own.

  1. You've used cd to go down one level into a directory. You can use cd .. to go up one level. Type ls to see that you're back in your home directory.

    cd ../
    ls
 drwxrwx---   2  user    user        4096 Aug 29 01:35 files
 drwxr-x---   2  user    user           6 Jul 30  2011 watch

  2. Next, you'll use cd to navigate directly to a directory more than one level away. cd ~/files/Video brings me into the directory I set up.

    cd ~/files/Video
    ls
 drwxr-xr-x   2  user    user        4096 May 30 18:48 Folder
 -rw-rw-rw-   2  user    user   325558260 May 30 18:47 Video1.mp4
 -rw-rw-rw-   2  user    user   325590711 May 30 18:47 Video2.mp4

  3. Now to navigate to directories on your computer with the lcd(read: local cd) command. The directory structure will be different on Linux than with Cygwin. On Linux, simply type lcd /path/to/directory. With Cygwin, if you want C:\Download you would type lcd /cygdrive/c/Download. To list what's in the current local directory, add an exclamation point to the ls command like !ls.

    lcd /cygdrive/c/Download
 lcd ok, local cwd=/cygdrive/c/Download
    !ls
 New Text Document.txt

Transfers

  1. There are two main commands used for downloading: pget for files and mirror for directories.

    pget DiskImage1.iso
 10800307 bytes transferred in 8 seconds (1.25M/s)
    mirror Video/
 Total: 1 directory, 2 files, 0 symlinks
 New: 2 files, 0 symlinks
 21600614 bytes transferred in 17 seconds (1.18M/s)

  2. There are two main commands used for uploading: put for files and mirror -R for directories.

    put DiskImage1.iso
 10800307 bytes transferred in 133 sseconds (79.0K/s)
    mirror -R Video/
 Total: 1 directory, 2 files, 0 symlinks
 New: 2 files, 0 symlinks
 21600614 bytes transferred in 298 seconds (70.8K/s)

  3. To cancel a transfer, use Ctrl+c

  4. To resume partially downloaded files/directories, you will use the -c switch: mirror -c Directory pget -c Video1.mp4 put -c Video1.mp4

Segmented Downloading

Note: This guide uses 5 segments as an example. You may use as many as needed or desired.

  1. A pget command using segmentation is pget -n 5 Video2.mp4, where 5 is the number of segments.

  2. A mirror command using segmentation is mirror --use-pget-n=5 Directory, where 5 is the number of segments.

  3. You can use jobs -v to see the speeds of the individual segments as well as the total speed.

Parallel Downloading

  1. A mirror command that downloads multiple files in parallel is mirror -P5 Directory or mirror --parallel=5, where 5 is the max number of files to download simultaneously.

  2. A mirror command that downloads multiple files at once and uses segmentation is mirror -P 5 --use-pget-n=2 Directory or mirror --parallel=5 --use-pget-n=2, where 5 is the max number of files to download simultaneously and 2 is the number of segments to use per file.

    queue mirror -P 5 --use-pget-n=2 Videos
 [0] Done (queue (sftp://user@server.whatbox.ca))
 
    jobs -v
 [1] queue (sftp://user@server.whatbox.ca)
         sftp://user@server.whatbox.ca/home/user/files
         Now executing: [2] mirror -P 5 --use-pget-n=2 Videos
  [2] mirror -P 5 --use-pget-n=2 Videos
   \transfer Video1.mp4
         'Video1.mp4', got 1540096 of 199674133 (0%) 430.5K/s eta:8m
   \chunk 0-99837067
         'Video1.mp4' at 753664 (0%) 206.9K/s eta:8m [Receiving data]
   \chunk 99837067-199674132
         'Video1.mp4' at 100623499 (0%) 223.5K/s eta:7m [Receiving data]
   \transfer Video2.mp4
         'Video2.mp4', got 1277952 of 199674133 (0%) 387.5K/s eta:9m
   \chunk 0-99837067
         'Video2.mp4' at 720896 (0%) 205.8K/s eta:8m [Receiving data]
   \chunk 99837067-199674132
         'Video2.mp4' at 100394123 (0%) 181.7/s eta:9m [Receiving data]
   \transfer Video3.mp4
         'Video3.mp4', got 917504 of 199674133 (0%) 257.1K/s eta:14m
   \chunk 0-99837067
         'Video3.mp4' at 262144 (0%) 119.3K/s eta:14m [Receiving data]
   \chunk 99837067-199674132
         'Video3.mp4' at 100492427 (0%) 137.7K/s eta:12m [Receiving data]

Queues and Jobs

  1. The queue command will allow you to transfer files/directories while still being able to browse. The basic command to queue a file transfer will look like queue pget Video1.mp4. To transfer entire directories, queue mirror Directory. This adds them to your transfer queue and automatically starts the transfer if ready.

  2. Use queue alone to display your transfer queue. The jobs command displays your current transfers and transfer queue with more detail.

  3. You can stop your queue from automatically transferring by using queue stop. This can be used before adding anything to your queue to prevent immediate transfer. queue start will start your queued transfers. To delete a transfer from your queue, add the -d switch like queue -d #, where # is the queue number listed by jobs or queue

  4. To stop and delete a running transfer, use the kill command. If you set lftp to allow multiple transfers simultaneously, kill #, where # is the number listed by jobs, will stop and delete the specified transfer. kill all will stop and delete all transfers.

    queue stop
    kill 1
    jobs
 [0] queue (sftp://user@server.whatbox.ca)
     Queue is stopped.
     Now executing: pget Video1.mp4
     Commands queued:
      1. pget Video2.mp4
  pget Video1.mp4
    queue -d 1
 Deleted job: pget Video2.mp4
    jobs
 [0] queue (sftp://user@server.whatbox.ca)

Configuration and Aliases

Now to make it quick and easy, we'll set up some aliases. Aliases allow you to make shortcuts to commands.

  • In Linux, find lftp.conf, .lftprc, or /.lftp/rc file and open for editing.
  • In Windows, open your user's lftp config directory (C:\Cygwin\home\username\.lftp by default) and create/edit a file named rc with no extension. If you haven't run lftp previously, you will need to run the following command in Cygwin to create the directory and config file touch ~/.lftp/rc.
  • Add any configuration you would like to your config file. These example aliases are based on this article.

Some examples from this guide with those aliases applied:
lcd /cygdrive/c/Download becomes down
queue pget -cn 5 Video1.mp4 becomes q pg Video1.mp4
queue mirror -c --use-pget-n=5 Directory becomes q m Directory

For full command help and settings, see the lftp man page.

Tips and Tricks

  • The Tab key autocompletes file and directory names. It also autocompletes commands.
  • Files and directories with spaces or special characters need special handling. The easiest method is to put quotes around the file/directory names.
  • You can cycle through previously used commands with the Up and Down arrow keys.
  • The list of remote files can be refreshed with rels.
  • To quit lftp, type exit. Then type exit again to close the terminal.