admin_guide.txt

openPBS的开放源代码

back-to-back.  The portion of a job that fits within primetime must be
no longer than PRIME_TIME_WALLT_LIMIT (represented in HH:MM:SS).

#PRIME_TIME_START		9:00:00
#PRIME_TIME_END			17:00:00
#PRIME_TIME_WALLT_LIMIT		1:00:00

The next option allows the site to choose an action to take upon scheduler
startup.  The default is to do no special processing (NONE). In some
instances, a job can end up queued in one of the batch queues, since it
was running before but was stopped by PBS. If the argument is RESUBMIT,
these jobs will be moved back to the queue the job was originally submitted
to, and scheduled as if they had just arrived. If the argument is RERUN,
the scheduler will have PBS run any jobs found enqueued on the execution
queues. This may cause the machine to get somewhat confused, as no limits
checking is done (the assumption being that they were checked when they
were enqueued).

SCHED_RESTART_ACTION		RESUBMIT

Define how long a job should be forced to wait in the queue before being
given "extra" priority to run. The priority given is exceeded only by the
priority of the Express queue jobs. Note that this extra priority is
ignored for jobs from an over-fairshare queue, or if the job owner has
exceeded his/her max running job limit. Value is express in HH:MM:SS
(default is 5 days)

MAX_WAIT_TIME                   120:00:00

If specified, this directive will tell the scheduler to dump an ordered
listing of the jobs to the named file. Useful for users and debugging,
but an expensive operation with LOTS of jobs queued, since the file is
rewritten for each run of the scheduler.

SORTED_JOB_DUMPFILE             /PBS/sched_priv/sorted_jobs

The Fair Access Directives allow the specification, on a per-architecture
basis, of a per-group limit on the maximum number of CPUs and the maximum
amount of memory simultaneously used by running jobs.
Format is: FAIR_ACCESS ARCH:arch1:groupA:num_cpus:MB_of_memory

FAIR_ACCESS ARCH:arch1:groupA:30:800
FAIR_ACCESS ARCH:arch1:default:40:100
FAIR_ACCESS ARCH:arch2:groupB:30:100
FAIR_ACCESS ARCH:arch2:default:20:100


* Lazy Commenting

Because changing the job comment for each of a large group of jobs can be
very time intensive, there is a notion of lazy comments. The function that
sets the comment on a job takes a flag that indicates whether or not the
comment is optional. Most of the "can't run because ..." comments are
considered to be optional.

When presented with an optional comment, the job will only be altered if
the job was enqueued after the last run of the scheduler, if it does not
already have a comment, or the job's 'mtime' (modification time) attribute
indicates that the job has not been touched in MIN_COMMENT_AGE seconds.

This should provide each job with a comment at least once per scheduler
lifetime.  It also provides an upper bound (MIN_COMMENT_AGE seconds + the
scheduling iteration) on the time between comment updates.
This compromise seemed reasonable because the comments themselves are some-
what arbitrary, so keeping them up-to-date is not a high priority.


Installing The UMN-Cluster Scheduler
------------------------------------

The UMN-Cluster scheduler is packaged as an optional scheduler for OpenPBS
v.2.3.  Basic steps are as follows (note that $PBSSRC is the directory into
which you extracted the PBS source tree; this is the directory that contains
the configure and configure.in file, amoung others); $PBSOBJ is the top of
your object tree.


Rebuilding PBS to use UMN-Cluster scheduler
--------------------------------------

While it is not necessary to rebuilt all of PBS, it is necessary
to rerun "configure" and then build the scheduler:

	cd $PBSOBJ
	$PBSSRC/configure [your options] --set-sched-code=umn_cluster
	make
	make install

Note: To run this scheduler on system that have more than 2gb of 
memory, you should build the scheduler in 64-bit mode, using the
configure options specific for your compiler.

Required modifications to existing PBS configuration
----------------------------------------------------

There are several changes that will need to be made to the PBS configuration.

The UMN-Cluster scheduler takes advantage of the server nodes file, which
contains one line per node. (For a detailed explaination of the format of
the "nodes" file, see the PBS Administrator Guide).

1. Edit $PBSHOME/server_priv/nodes, and add one line for each execution
   host, as the following example shows:

   pbsnode1:ts  np=8
   pbsnode2:ts  np=8
   pbsnode3:ts  np=8
   pbsnode4:ts  np=8

Where the first column is the hostname of the node, appended with a
":ts" denoting that it is a timeshared node, and the second column
is a number of processor specification. (This NP value is 
used as the maximum number of cpus on a given node. This allows
the server to reject a job immediately that requests more cpus
than can be provided by the current configuration.)

2. Create an execution queue that will become a holding queue from which
   the scheduler will pull jobs. This queue will need certain minimum
   attributes set, as indicated below:

   first start the server, if not already running:
   #pbs_server

   then create and set the queue attributes (example SUBMIT queue
   called "pending"):

   #qmgr
   create queue pending
   set queue pending queue_type = Execution
   set queue pending resources_default.ncpus = 1
   set queue pending resources_default.walltime = 00:05:00
   set queue pending enabled = True
   set queue pending started = True

3. Create an execution queue that corresponds to each execution host.
   Set the default and maximum attributes for each execution queue. 
   It is suggested you dump the qmgr output to a file, and then edit
   the file (ie  'qmgr -c "p s" > /tmp/somefile' ; edit the file; and
   then load the changed info back into the server: 'qmgr < /tmp/somefile').
   The example below shows the recommended attributes (and changes via qmgr):
   Note that the "from_route_only" directive prevents users from submitting
   directly to the backend execution queues. This is necessary since
   priorities are calculated based on the originating queues (i.e. the
   route queues defined below).

   #qmgr
   create queue pbsnode1
   set queue pbsnode1 queue_type = Execution
   set queue pbsnode1 from_route_only = True
   set queue pbsnode1 resources_max.ncpus = 8
   set queue pbsnode1 resources_max.walltime = 08:00:00
   set queue pbsnode1 resources_max.memory = 240mb
   set queue pbsnode1 resources_default.ncpus = 1
   set queue pbsnode1 resources_default.walltime = 00:05:00
   set queue pbsnode1 enabled = True
   set queue pbsnode1 started = True
   ...

4. Create as many "originating" Route queues as needed by your local
   configuration. These should be route queues with one destination:
   the above defined SUBMIT queue. Jobs will carry their originating
   queue name with them.

   # Create and define queue groupA
   #
   create queue groupA
   set queue groupA queue_type = Route
   set queue groupA route_destinations = pending
   set queue groupA resources_default.mem = 10mb
   set queue groupA enabled = True
   set queue groupA started = True
   #
   # Create and define queue groupB
   #
   create queue groupB
   set queue groupB queue_type = Route
   set queue groupB route_destinations = pending
   set queue groupB resources_default.mem = 20mb
   set queue groupB enabled = True
   set queue groupB started = True


   # and then set the default queue on the server. 
   #
   set server default_queue = pending


 5. Set the cluster-wide maximum number of cpus (e.g. the count of all the
    CPUs on all nodes within the cluster. This info will be used in addition
    to the dynamically queried per-node CPU counts. You will need to update
    this value if you remove or add nodes to your cluster. Do not worry about
    updating it for a temporarily unavailable node.

    #qmgr
    set server resources_max.ncpus = 48  (or whatever the correct value is)
    quit


Configuring the UMN-Cluster scheduler
--------------------------------------

The scheduler configuration file (as discussed above) will need to be
modified. Edit $PBSHOME/sched_priv/sched_config changing in particular
the BATCH_QUEUES line. This should contain the list of all the queues
you have defined, and the associated execution host, eg:

Host "hostA.pbs.com" has an associated queue named "hostA" and
"pbsnode1.pbs.com" is fed by queue "pbsnode1":

BATCH_QUEUES	[e-mail],[e-mail]

However, the full hostname is not required, so for brevity, one could enter:

BATCH_QUEUES	hostA@hostA,pbsnode1@pbsnode1

The FAIR_ACCESS directive will also need to be updated, as described above.

Review the other configuration parameters, and change any as needed.
They are currently set to recommended defaults.


General Notes
-------------

This section has some general comments about this scheduler, and things
to be aware of.

Since this scheduler supports the PBS nodes file, you can use the "pbsnodes"
commands to view node status, take nodes offline, etc. Here is a short
summary of handy

	pbsnodes -l      # lists all down/offline/unavailable nodes

	pbsnodes -a 	 # lists all info for all nodes

	pbsnodes -o node # mark the named node OFFLINE, running jobs will
			 # will continue to run on that node, but no new
			 # jobs will be started on it. Be sure to list all
			 # OFFLINE nodes as any not listed will be assumed
			 # to be up.

	pbsnodes -c node # clear or remove the OFFLINE status on node, 
			 # making it available for running jobs again.