config.txt

HADOOP 0.18.0安装源代码头文件

                     HOD Configuration
                     =================

1. Introduction:
================

Configuration options for HOD are organized as sections and options 
within them. They can be specified in two ways: a configuration file 
in the INI format, and as command line options to the HOD shell, 
specified in the format --section.option[=value]. If the same option is 
specified in both places, the value specified on the command line 
overrides the value in the configuration file.

To get a simple description of all configuration options, you can type
    hod --verbose-help

This document explains some of the most important or commonly used
configuration options in some more detail.

2. Sections:
============

The following are the various sections in the HOD configuration:

    * hod:                  Options for the HOD client
    * resource_manager:     Options for specifying which resource 
                            manager to use, and other parameters for 
                            using that resource manager
    * ringmaster:           Options for the RingMaster process, 
    * hodring:              Options for the HodRing processes
    * gridservice-mapred:   Options for the MapReduce daemons
    * gridservice-hdfs:     Options for the HDFS daemons.

The following are some of the important options in the HOD 
configuration:

3. Important / Commonly Used Configuration Options:
===================================================

3.1. Common configuration options:
----------------------------------

Certain configuration options are defined in most of the sections of 
the HOD configuration. Options defined in a section, are used by the
process for which that section applies. These options have the same
meaning, but can have different values in each section.

* temp-dir: Temporary directory for usage by the HOD processes. Make 
            sure that the users who will run hod have rights to create 
            directories under the directory specified here.

* debug: A numeric value from 1-4. 4 produces the most log information,
         and 1 the least.

* log-dir: Directory where log files are stored. By default, this is
           <install-location>/logs/. The restrictions and notes for the
           temp-dir variable apply here too.

* xrs-port-range: A range of ports, among which an available port shall
                  be picked for use to run an XML-RPC server.

* http-port-range: A range of ports, among which an available port shall
                   be picked for use to run an HTTP server.

* java-home: Location of Java to be used by Hadoop.

3.2 hod options:
----------------

* cluster: A descriptive name given to the cluster. For Torque, this is
           specified as a 'Node property' for every node in the cluster.
           HOD uses this value to compute the number of available nodes.

* client-params: A comma-separated list of hadoop config parameters
                 specified as key-value pairs. These will be used to
                 generate a hadoop-site.xml on the submit node that 
                 should be used for running MapReduce jobs.

3.3 resource_manager options:
-----------------------------

* queue: Name of the queue configured in the resource manager to which
         jobs are to be submitted.

* batch-home: Install directory to which 'bin' is appended and under 
              which the executables of the resource manager can be 
              found. 

* env-vars: This is a comma separated list of key-value pairs, 
            expressed as key=value, which would be passed to the jobs 
            launched on the compute nodes. 
            For example, if the python installation is 
            in a non-standard location, one can set the environment
            variable 'HOD_PYTHON_HOME' to the path to the python 
            executable. The HOD processes launched on the compute nodes
            can then use this variable.

3.4 ringmaster options:
-----------------------

* work-dirs: These are a list of comma separated paths that will serve
             as the root for directories that HOD generates and passes
             to Hadoop for use to store DFS / MapReduce data. For e.g.
             this is where DFS data blocks will be stored. Typically,
             as many paths are specified as there are disks available
             to ensure all disks are being utilized. The restrictions
             and notes for the temp-dir variable apply here too.

3.5 gridservice-hdfs options:
-----------------------------

* external: If false, this indicates that a HDFS cluster must be 
            bought up by the HOD system, on the nodes which it 
            allocates via the allocate command. Note that in that case,
            when the cluster is de-allocated, it will bring down the 
            HDFS cluster, and all the data will be lost.
            If true, it will try and connect to an externally configured
            HDFS system.
            Typically, because input for jobs are placed into HDFS
            before jobs are run, and also the output from jobs in HDFS 
            is required to be persistent, an internal HDFS cluster is 
            of little value in a production system. However, it allows 
            for quick testing.

* host: Hostname of the externally configured NameNode, if any

* fs_port: Port to which NameNode RPC server is bound.

* info_port: Port to which the NameNode web UI server is bound.

* pkgs: Installation directory, under which bin/hadoop executable is 
        located. This can be used to use a pre-installed version of
        Hadoop on the cluster.

* server-params: A comma-separated list of hadoop config parameters
                 specified key-value pairs. These will be used to
                 generate a hadoop-site.xml that will be used by the
                 NameNode and DataNodes.

* final-server-params: Same as above, except they will be marked final.


3.6 gridservice-mapred options:
-------------------------------

* external: If false, this indicates that a MapReduce cluster must be
            bought up by the HOD system on the nodes which it allocates
            via the allocate command.
            If true, if will try and connect to an externally 
            configured MapReduce system.

* host: Hostname of the externally configured JobTracker, if any

* tracker_port: Port to which the JobTracker RPC server is bound

* info_port: Port to which the JobTracker web UI server is bound.

* pkgs: Installation directory, under which bin/hadoop executable is 
        located

* server-params: A comma-separated list of hadoop config parameters
                 specified key-value pairs. These will be used to
                 generate a hadoop-site.xml that will be used by the
                 JobTracker and TaskTrackers

* final-server-params: Same as above, except they will be marked final.

4. Known Issues:
================

HOD does not currently handle special characters such as space, comma
and equals in configuration values.