protocol.txt

libbt-1.04

BitTorrent <index.html> download <download.html> FAQ <FAQ.html>
documentation <doc.html> press <press.html> Donate! <donate.html>

BitTorrent is a protocol for distributing files. It identifies content
by url and is designed to integrate seamlessly with the web. Its
advantage over plain http is that when multiple downloads of the same
file happen concurrently, the downloaders upload to each other, making
it possible for the file source to support very large numbers of
downloaders with only a modest increase in its load.

The life cycle of a BitTorrent file distribution.

A BitTorrent file distribution consists of these entities -

    * An ordinary web server
    * A static 'metainfo' file
    * A BitTorrent tracker
    * An 'origin' downloader
    * The end user web browsers
* The end user downloaders

There are ideally many end users for a single file.

To start serving, a host goes through the following steps -

   1. Start running a tracker (or, more likely, have one running already).
   2. Start running an ordinary web server, such as apache, or have one
      already.
   3. Associate the extension .torrent with mimetype
      application/x-bittorrent on their web server (or have done so
      already).
   4. Generate a metainfo file using the complete file to be served and
      the url of the tracker.
   5. Put the metainfo file on the web server.
   6. Link to the metainfo file from some other web page.
   7. Start a downloader which already has the complete file (the
      'origin'). 

To start downloading, a user does the following -

   1. Run a BitTorrent installer (or have done so already).
   2. Web surf.
   3. Click on a link to a .torrent file.
   4. Select where to save the file locally, or select a partial
      download to resume.
   5. Wait for download to complete.
   6. Tell downloader to exit (it keeps uploading until this happens). 

The connectivity is as follows -

    * The web site is serving up static files as normal, but kicking off
      the BitTorrent helper app on the clients.
    * The tracker is receiving information from all downloaders and
      giving them random lists of peers. This is done over http or https.
    * Downloaders are periodically checking in with the tracker to keep
      it informed of their progress, and are uploading to and
      downloading from each other via direct connections. These
      connections use the BitTorrent peer protocol, which operates over
      TCP.
    * The origin is uploading but not downloading at all, since it has
      the entire file. The origin is necessary to get the entire file
      into the network. Often for popular downloads the origin can be
      taken down after a while since several downloads may have
completed and been left running indefinitely.

Metainfo file and tracker responses are both sent in a simple,
efficient, and extensible format called bencoding (pronounced 'bee
encoding'). Bencoded messages are nested dictionaries and lists, which
can contain strings and integers. Extensibility is supported by ignoring
unexpected dictionary keys, so additional optional ones can be added later.

Bencoding is done as follows -

    * Strings are length-prefixed base ten followed by a colon and the
      string. For example '4:spam' corresponds to 'spam'.
    * Integers are represented by an 'i' followed by the number in base
      10 followed by an 'e'. For example 'i3e' corresponds to 3 and
      'i-3e' corresponds to -3. Integers have no size limitation. 'i-0e'
      is invalid. All encodings with a leading zero, such as 'i03e', are
      invalid, other than 'i0e', which of course corresponds to 0.
    * Lists are encoded as an 'l' followed by their elements (also
      bencoded) followed by an 'e'. For example 'l4:spam4:eggse'
      corresponds to ['spam', 'eggs'].
    * Dictionaries are encoded as a 'd' followed by a list of
      alternating keys and their corresponding values followed by an
      'e'. For example, 'd3:cow3:moo4:spam4:eggse' corresponds to
      {'cow': 'moo', 'spam': 'eggs'} and 'd4:spaml1:a1:bee' corresponds
      to {'spam': ['a', 'b']} . Keys must be strings and appear in
sorted order (sorted as raw strings, not alphanumerics).


* Metainfo files are bencoded dictionaries with the following keys -
*
*    md={
*      announce=>'url',
*      info=>{
*          name=>'top-level-file-or-directory-name',
*          piece length=>12345,
*          pieces=>'md5sums',
*
*          length=>12345,
*            -or-
*          files=>[
*              {
*                  length=>12345,
*                  path=>['sub','directory','path','and','filename']
*
*              }, ... {}
*          ]
*      }


Metainfo files are bencoded dictionaries with the following keys -

'announce'
    The url of the tracker.

'info'
    This maps to a dictionary, with keys described below.

    The 'name' key maps to a string which is the suggested name to save
    the file (or directory) as. It is purely advisory.

    'piece length' maps to the number of bytes in each piece the file is
    split into. For the purposes of transfer, files are split into
    fixed-size pieces which are all the same length except for possibly
    the last one which may be truncated. Piece length is almost always a
    power of two, most commonly 2^20 .

    'pieces' maps to a string whose length is a multiple of 20. It is to
    be subdivided into strings of length 20, each of which is the sha1
    hash of the piece at the corresponding index.

    There is also a key 'length' or a key 'files', but not both or
    neither. If 'length' is present then the download represents a
    single file, otherwise it represents a set of files which go in a
    directory structure.

    In the single file case, 'length' maps to the length of the file in
    bytes.

    For the purposes of the other keys, the multi-file case is treated
    as only having a single file by concatenating the files in the order
    they appear in the files list. The files list is the value 'files'
    maps to, and is a list of dictionaries containing the following keys -

'length'
The length of the file, in bytes. 'path'
    A list of strings corresponding to subdirectory names, the last of
which is the actual file name (a zero length list is an error case).

In the single file case, the 'name' key is the name of a file, in the
muliple file case, it's the name of a directory.

Tracker queries are two way. The tracker receives information via GET
parameters and returns a bencoded message. Note that although the
current tracker implementation has its own web server, the tracker could
run very nicely as, for example, an apache module.


* Tracker GET requests have the following keys urlencoded -
*   req = {
*      info_hash => 'hash'
*      peer_id => 'random-20-character-name'
*      ip => 'ip-address' -or- 'dns-name'
*      port => '12345'
*      uploaded => '12345'
*      downloaded => '12345'
*      left => '12345'
*      event => 'started', 'completed' -or- 'stopped'
*   }

Tracker GET requests have the following keys -

'info_hash'
    The 20 byte sha1 hash of the bencoded form of the 'info' value from
    the metainfo file. Note that this is a substring of the metainfo
    file. This value will almost certainly have to be escaped.

'peer_id'
    A string of length 20 which this downloader uses as its id. Each
    downloader generates its own id at random at the start of a new
    download. This value will also almost certainly have to be escaped.

'ip'
    An optional parameter giving the ip (or dns name) which this peer is
    at. Generally used for the origin if it's on the same machine as the
    tracker.

'port'
    The port number this peer is listening on. Common behavior is for a
    downloader to try to listen on port 6881 and if that port is taken
    try 6882, then 6883, etc. and give up after 6889.

'uploaded'
    The total amount uploaded so far, encoded in base ten ascii.

'downloaded'
    The total amount downloaded so far, encoded in base ten ascii.

'left'
    The number of bytes this peer still has to download, encoded in base
    ten ascii. Note that this can't be computed from downloaded and the
    file length since it might be a resume, and there's a chance that
    some of the downloaded data failed an integrity check and had to be
    re-downloaded.

'event'
    This is an optional key which maps to 'started', 'completed', or
    'stopped' (or '', which is the same as not being present). If not
    present, this is one of the announcements done at regular intervals.
    An announcement using 'started' is sent when a download first
    begins, and one using 'completed' is sent when the download is
    complete. No 'completed' is sent if the file was complete when
    started. Downloaders send an announcement using 'stopped' when they
    cease downloading.

* Tracker responses are bencoded dictionaries.
*