LoRS Command Line Tool Tutorial
The LoRS command line tools let you do three primary things:
They also give you the ability to see where the pieces of the file are stored (List) and to extend the allocations' expiration times (Refresh).
- Store a file into the wide-area network (Upload),
- Change where the file is located in the network (Augment or Trim), and
- Retrieve the file (Download).
The tools have built-in defaults which can be over-ridden by settings in the .xndrc preferences file.
Let's get a file back from the network. The lors_download tool tries to download the file from beginning to end in order (to a degree). The simplest usage is:
lors_download foo.xnd -f
The tells the tool to download the file stored in foo.xnd to its original name (foo) and use the defaults for threads, blocksize, etc. This works but you want to alter the performance by adding a few parameters:
lors_download foo.xnd -f -t 8 -b 512k -C 8
In this case, we specified 8 threads, a transfer blocksize of 512 K and a working cache of 8 blocks. As you can see, you can explicitly control the amount of threads used. You are also controlling the amount of RAM used during the download. The amount used is equal to (the number of threads + the number of cache blocks) times the blocksize or:
RAM = (t + C) x b
To optimize your downloads, keep the RAM level below your physical RAM to avoid constant thrashing of your swap file. We have found that in WAN downloads, that a blocksize of 512 KB or 1 MB is ideal. I try to keep the cache parameter equal to (or twice) the thread count. We found that downloads do benefit from increasing threads, but most see little additional benefit over 24 threads. Note, the IETF views using more than 1 or 2 threads per depot as TCP unfriendly and system administrators may perceive your download as a denial of service (DOS) attack.
The lors_download tool offers several other options. You can choose to only download a portion of a file by specifying the beginning offset (-O) and its length (-L). If you are piping the output to a media player (mplayer, mpg123, sox, etc), you would want to use the prebuffer flag (-q number_of_blocks). This tells the download tool to wait until the first set of blocks are downloaded before releasing any to the player. This reduces the chance that the player will start and immediately stall because the first block was ready but the second (or third, etc) was not. Typically when playing the multimedia, I prebuffer enough blocks to play a few seconds.
The download tool features an adaptive algorithm as outlined in Algorithms for High Performance, Wide-Area, Distributed File Downloads. If the exNode has multiple copies of the data, the Progress-Driven Redundancy algorithm tries downloading blocks from any depot that has the data. If the data comes in at similar rates, then the algorithm simply works in a round-robin manner when selecting from which depot to get the data. If some depots send their data much faster than others, the threads calling on those depots will run ahead of the threads calling on the slower depots. If the faster threads are working on blocks that are too far ahead of the lagging threads, the algorithm will begin to duplicate effort and assign additional threads to download the lagging blocks from different depots. To use this algorithm, you can specify the number of blocks that can separate the lagging threads from the leading threads with -p (the progress number). You would then need to set the amount of redundancy (number of duplicate downloads) with -r. Keep in mind that if your exNode has two copies, then specifying a -r 3 will only be able to download form the two copies and not three. You might use something like:
lors_download foo.xnd -t 10 -b 512k -C 20 -p 15 -r 3
Also, in this example, I did not specify -f or -o to set the output file name. In this case, the file will be sent to stdout which is usually the terminal screen. This is helpful if you want to play the music or movie without storing it on disk. To do so, you will need a multimedia player that can read on stdin such as mpg123 and mplayer. Usage is something like:
lors_download foo.xnd -t 10 -b 512k -C 20 -p 15 -r 3 | mplayer -
See the mpg123 and mplayer man pages for usage. If you forgot to specify -f or -o and your screen gets the data dump and whatever you tupe looks like garbage characters, type 'reset' and it should clear it up.