readme

openPBS的开放源代码

	ENFORCE_DEDICATED_TIME			<boolean>
	DEDICATED_TIME_COMMAND			<pathname>
	DEDICATED_QUEUE				<queue_spec>
	SYSTEM_NAME				<string>
	DEDICATED_TIME_CACHE_SECS [optional]	<integer>

Downtimes for all of the NAS supercomputers are scheduled by a locally-
created program called 'schedule'.  The schedule client queries a database
through the server, returning a list of scheduled downtimes in chronological
order.  The scheduler parses through this output, looking for information
about any upcoming dedicated times for the machine or cluster named by
the configuration variable 'SYSTEM_NAME'.

The scheduler runs the DEDICATED_TIME_COMMAND with a single argument of the
string given by the SYSTEM_NAME option.  As shipped, the scheduler expects
the output of this command to be in the following format:

    TURING       07/07/1998 16:00-19:00 07/07/1998  Administrative period 
						    (weekly turing downtime)
    HOPPER       07/09/1998 16:00-19:00 07/09/1998  Administrative period 
						    (weekly hopper downtime)
    PIGLET       07/14/1998 16:00-19:00 07/14/1998  Install latest compiler.
    PIGLET       07/28/1998 10:00-16:00 07/28/1998  Preventive maintenance
    SLICKY       08/20/1998 09:00-09:00 08/21/1998  All day outage for power
                                                    supply and CPU board swap.

The DEDICATED_TIME_COMMAND may be a binary executable or a shell script.

If DEDICATED_TIME_CACHE_SECS is specified, the DEDICATED_TIME_COMMAND will
be queried at most once each DEDICATED_TIME_CACHE_SECS seconds.  This feature
is intended to prevent overloading a centralized server with queries.

* Fragmentation Avoidance Heuristics

    Related configuration options:
        AVOID_FRAGMENTATION			<boolean>

A natural side-effect of packing jobs into a queue is that the queues tend
to become "fragmented".  As jobs are packed into the queue, the available
resources become smaller, decreasing the possibilities of a large job
being able to run.  Without some mechanism for recovering from queue
fragmentation, it is possible that the large job will be starved forever. 

Assuming there is a relatively small job running in the queue, the larger
job will not be able to run.  However, to improve utilization, smaller
jobs will be packed into the remaining nodes.  These jobs may run longer
than the original small job.  When the original job exits, there will
still not be enough nodes available to run the large job, so more small
jobs will be packed into the remaining space.  This "juggling" effect can
continue indefinitely in a naive implementation.

This problem was addressed in an early version of the scheduler (before
the Waiting Job Support (see below) was added) by introducing a heuristic
algorithm to attempt to recover from fragmentation.  The algorithm works
by examining the relationship between queue and job resources, as follows: 

If the queue is empty, allow the job to run, and attempt to recover from
fragmentation later, if necessary.  This avoids unduly preventing any job
from running.

Otherwise, compute the "fragment size"  of the queue.  This is the number
of nodes assigned to the queue divided by the maximum number of jobs that
can be run on the queue.  If the maxrun (see qmgr(1)) is not defined, or
the fragment size is less than 2 nodes, the heuristic doesn't apply, so
allow the job to run. 

If the requesting job is at least as large as the fragment size, allow it
to run - it will not create a fragmented queue itself.

If the job is smaller than the fragment size, it could cause the queue to
become or continue to be fragmented.  Compute the average size of the jobs
running in the queue.  If the average job size is smaller than the
fragment size then the queue is already fragmented.  If the job is
expected to complete before the queue is empty, allow it to run. 
Otherwise, do not run it, as it would perpetuate the state of
fragmentation.

Finally, if the job would allocate the last fragment, and it would extend
past the expected time at which the queue will empty, don't allow it.

If the job has not been disallowed by the previous tests, allow it to run.

This heuristic appears to do a reasonable job of controlling the effects
of queue fragmentation without unduly delaying jobs.

* Submit Queue

    Related configuration options:   
        SUBMIT_QUEUE				<queue_spec>

The "normal" mode of operation for this scheduler is to look for jobs that
are enqueued on the SUBMIT_QUEUE.  Assuming the system resources and policy 
permit it, jobs are moved from the SUBMIT_QUEUE to one of the BATCH_QUEUES
(see below), and started.

Note that the limits on the submit queue (i.e. resources_max.*) must be the
union of the limits on the BATCH_QUEUES.  This is required by PBS, since it
will reject requests to submit to the SUBMIT_QUEUE if the job does not fit
the SUBMIT_QUEUE limits.  See the "Configuring The Scheduler" section for
more information.

* Execution Queues
    
    Related configuration options:   
        BATCH_QUEUES				<queue_spec>[,<queue_spec,...]

* "Special" Queue Support

    Related configuration options:   
        SPECIAL_QUEUE				<queue_spec>

The "special" queue provides a way for the batch system administrator to
selectively allow high-priority access to the system, by providing a user
ACL (see qmgr(1)) on the queue named by the SPECIAL_QUEUE option.  Any
jobs queued on the special queue will be placed at the top of the list of
jobs to be scheduled.  This means that they will be considered for
execution before any other jobs. 

Additionally, jobs found on the SPECIAL_QUEUE will be internally treated
as high-priority jobs.  Most of the policy checks do not apply to jobs
that are marked as high-priority.  This can be confusing to users and
admins alike, so special jobs are noted as such in the job comment :

   "Started on Fri Aug  7 17:24:26 1998 (special access job)"

Internally, there is only one list of jobs (the submit queue's job list) 
that is considered for scheduling.  The "special" jobs are marked with a
HIGH_PRIORITY flag and placed at the head of the list by the function
fixup_special() in schedule.c.

* Externally Routed Queue Support
    
    Related configuration options:   
        EXTERN_QUEUES				<queue_spec>[,<queue_spec,...]

The scheduler provides support for "externally routed" queues.  An externally
routed queue is defined as a queue into which jobs are enqueued by some 
external agent (the user, an admin, a "metascheduler", etc).  At the end
of each scheduling run, the scheduler checks for EXTERN_QUEUES that have
jobs queued upon them.  If jobs are found, the scheduler attempts to pack
as many jobs into the queue as possible.

The code path used to run the jobs in the external queue is the same as that
used by the "normal" submit->execution queue scheduling.  This means that 
jobs on the external queues are subject to the same policy rules as the
normal queues.  We currently are investigating ways to decouple policy from
the implementation, such that the individual queues can have distinct policies. 

* Primetime Support
* Primetime Specification (syntax)
* Job Sorting Algorithms (policy)
* Data Structures (Job, Queue, QueueList, UserACL, Resources, etc)
* Queue Definitions
* Handling of Job Lists From Server (get/demux)
* Lazy Commenting

Because changing the job comment for each of a large group of jobs can be
very expensive, there is a notion of lazy jobs.  The function that sets the
comment on a job takes a flag that indicates whether or not the comment is
optional.  Most of the "can't run because ..." comments are considered to
be optional.

When presented with an optional comment, the job will only be altered if
the job was enqueued after the last run of the scheduler, if it does not
already have a comment, or the job's 'mtime' (modification time) attribute
indicates that the job has not been touched in MIN_COMMENT_AGE seconds.

This should provide each job with a comment at least once per scheduler
lifetime.  It also provides an upper bound (MIN_COMMENT_AGE seconds + the
scheduling iteration) on the time between comment updates.

This compromise seemed reasonable because the comments themselves are some-
what arbitrary, so keeping them up-to-date is not a high priority.

* Waiting Job Support (batch/interactive)

    Related configuration options:   
	MAX_QUEUED_TIME				<time_spec>
        INTERACTIVE_LONG_WAIT			<time_spec>
        SMALL_QUEUED_TIME			<time_spec>

Although some effort is made to prevent it happening, it is very possible
for a job to be starved for a very long period of time.  This is especially
true when large numbers of "special" jobs have been run, preventing the
other "normal" jobs from being eligible.  The scheduler provides a simple
mechanism by which these jobs may be prevented from starving indefinitely.

When a job has not been run for some period of time (see below), that job
will be marked as "waiting".  Waiting jobs are pushed to the top of the job
list (but are still preempted by "special" jobs).  When a job is marked as
waiting, a queue will be drained if necessary to allow the job to run as
soon as possible.

Jobs become waiting when their eligible time ("etime" attribute) (or, if 
not defined, their creation time ("ctime" attribute)) exceeds one of two
values, depending upon the value of the "interactive" job attribute.

    Job Type		Becomes waiting when etime (or ctime) exceeds...
    ---------------	----------------------------------------------------
    Non-interactive	MAX_QUEUED_TIME
    Interactive		(INTERACTIVE_LONG_WAIT + requested walltime)

(Setting either option to "00:00:00" disables the appropriate functionality.)

The theory is that an interactive job should become "waiting" much earlier
than a batch job, as there is probably a person at the other end of the 
'qsub' in the interactive case.  However, to provide a sort of "ordering"
for interactive jobs, the "long time" is dependent upon their walltime.

Waiting jobs are sorted using the following criteria:

    If both jobs are either over or under SMALL_QUEUED_TIME, sort by:
	- Largest to smallest (reqeuested nodes)
	- Oldest to youngest (eligible time)

    If one job requests more than SMALL_QUEUED_TIME, and the other doesn't,
    sort them by:
	- Longest to shortest (requested walltime)
	- Youngest to oldest (eligible time)

* Past-Usage Support
* Scheduler Restart Action
* Deletion of Illegally-sized Jobs
* Big Jobs (concept)
* Origin2000 Architecture
* What Nodemask Is/Does
* The Need For Queues
* History Of The Code
* Configuration Options (list with descriptions)
* User ACL's on Queues
* Signal Handlers
* Standards Compliance
* Allocations
* System Resources
* Queue Limits
* Scheduling CPU/Memory vs. Nodes
* Policy (implementation of)
* Dynamic Nodemask
* Multiple Execution Hosts
* Future Work
* Queue Sanity