Torrents from the command line

2016-12-13

Scientists are mirroring data from government sites, ensuring it does not vanish under the next U.S. administration.

As part of that effort, I've downloaded 11GB of NLCD data.

Now, how to make that available?

My plan is to spin up a node on DigitalOcean, which provides cheap cloud hosting, and make the data available via torrent.

Why Torrents?

They ensure the integrity of the file being downloaded, share bandwidth costs, facilitate speedy downloads, and enable users to easily participate in archiving data.

TODO

Instructions

Install the Transmission torrent client and daemon:

sudo apt install transmission-daemon transmission-cli

You'll now need a non-root user to run the daemon and handle the data. If you don't already have one, you can create one as follows:

sudo adduser --disabled-password transmission
sudo mkdir -p /home/transmission/Downloads
#Move the data files to the user's `Downloads` directory:
sudo mv TODO_FILES_HERE /home/transmission/Downloads
#Ensure permissions are set correctly
chown -R transmission /home/transmission/Downloads
chgrp -R transmission /home/transmission/Downloads

Create torrents for each file with transmission-create:

ls * | xargs -n 1 transmission-create

Move these torrents out of the directory:

mv *.torrent ~/

You can also create torrents of entire directories, ala:

:::bash transmission-create directory_name/

We now need to add trackers to the torrents. Below is a list of trackers which may work for you until dedicated Earth Science trackers can be set up. (TODO) The following command adds them to the torrent files:

echo "http://9.rarbg.com:2710/announce
http://announce.torrentsmd.com:6969/announce
http://bt.careland.com.cn:6969/announce
http://explodie.org:6969/announce
http://mgtracker.org:2710/announce
http://tracker.tfile.me/announce
http://tracker.torrenty.org:6969/announce
http://tracker.trackerfix.com/announce
http://www.mvgroup.org:2710/announce
udp://9.rarbg.com:2710/announce
udp://9.rarbg.me:2710/announce
udp://9.rarbg.to:2710/announce
udp://coppersurfer.tk:6969/announce
udp://exodus.desync.com:6969/announce
udp://glotorrents.pw:6969/announce
udp://open.demonii.com:1337/announce
udp://tracker.coppersurfer.tk:6969/announce
udp://tracker.glotorrents.com:6969/announce
udp://tracker.leechers-paradise.org:6969/announce
udp://tracker.openbittorrent.com:80/announce
udp://tracker.opentrackr.org:1337/announce
udp://tracker.publicbt.com:80/announce
udp://tracker4.piratux.com:6969/announce" | xargs -n 1 -I {} transmission-edit -a {} *.torrent

Start the daemon with transmission-daemon:

transmission-daemon -er -GSR -T

The -er flag enforces peer encryption. The -GSR flag seeds regardless of the ratio of upload to download, which is important since I hope to do quite a bit of seeding. -T says we don't require authentication.

(For debugging the above use the -f foreground flag. You can indicate an alternative downloads directory with -w DIRNAME.)

Add the torrents:

ls *.torrent | xargs -n 1 transmission-remote --add

If you check on the status of the daemon:

transmission-remote -l

It should have recognized that the torrents were already present in the Downloads directory and be ready to seed them, like so:

ID     Done       Have  ETA           Up    Down  Ratio  Status       Name
   1     0%       None  Unknown      0.0     0.0   None  Idle         NLCD_canopy_legend_2013.jpg
   2   100%   254.6 MB  Done         0.0     0.0    0.0  Idle         ak_nlcd_2011_landcover_1_15_15.zip
   3   100%   268.3 MB  Done         0.0     0.0    0.0  Idle         AK_nlcd_2011_USFS_tree_canopy_2011_edition_2016_02_08_analytical.zip
   4   100%   89.55 MB  Done         0.0     0.0    0.0  Idle         AK_nlcd_2011_USFS_tree_canopy_2011_edition_2016_02_08_cartographic.zip
   5   100%    1.21 MB  Done         0.0     0.0    0.0  Idle         ak_nlcd_2011_zone8_impervious_1_15_15.zip

If all of the files show Done as being 0%, then you may have the files located in the wrong place. Ensure they are in the directory indicated by

transmission-remote -si

The output of which includes this:

CONFIG
  Download directory: /home/rick/Downloads   <-Ensure your files are here

Alternatively, you may need to recreate and re-add one or all of the torrents as occasionally, for reasons unknown to me, this does not work the first time. Torrents can be removed with:

transmission-remote -t <ID#> -r

Magnet Links

Time was when you had to share torrent files in order for someone to get access to your torrent. Now, we have magnet links. They have have the same information, but without the clumsy files. To generate magnet links for your torrents, use the following:

ls *.torrent | xargs -n1 sh -c 'echo "\n$0"; transmission-show -m $0'

But You Wanted A Server

Here's an easy way to set up a file-sharing server:

sudo apt install nginx

Edit the configuration like so:

sudo pico /etc/nginx/nginx.conf

So that it includes the following indicated line:

http {

        ##
        # Basic Settings
        ##

        autoindex on;   #This is the import bit

Delete the default landing page:

rm -f /var/www/html/*

Move the torrent files, data, and other directories into:

mv *.torrent /var/www/html/

Ensure that they can be read by the webserver:

chmod -R a+r /var/www/html/*

And you're ready to roll.

Richbits

Why Torrents?

Instructions

Magnet Links

But You Wanted A Server