readme

#ifdef INTEGER #include "ibp.h" #else #include "rbp.h" #endif /* built-in C functions */

   Some CPU time can be saved by altering certain parameter settings
that skip some of the forward passes used to determine the current set
of statistics.  The more often you print the statistics the more time
you can save by altering these parameter settings.  The penalty is that
the statistics will be out of date by one iteration with the quickprop
delta-bar-delta, supersab and regular periodic update methods and only
approximate for both the right and wrong continuous update methods.

   In all the update methods the training set statistics are computed
when the program passes back the error.  However then an update of the
weights is done and these numbers are out of date.  So if 100 iterations
have been done the program only has the statistics on iteration 99, the
values that were true before the weights were changed.  When it is time
to print out the program statistics the default is to do another forward
pass through the training set to get up to date statistics.  This can be
stopped by setting the off by 1 option in the format command like so:

f O+

The results that print out will now look like this:

   10 -1   0.00 %  64.42 % 0.42463       0.00 %  62.50 % 0.38234 
   20 -1   0.00 %  73.08 % 0.39058       0.00 %  77.88 % 0.38108 
   30 -1   8.65 %  75.00 % 0.35052       4.81 %  79.81 % 0.33285 

where the string "-1" comes right after the number of iterations done
and of course it means the numbers shown are for the previous iteration.
For the test set patterns one pass through the training set has to be
made to get up to date statistics on them so they are always up to date.
Most of the time you will probably be more interested in the test set
results than in the training set results so setting the off by 1 option
saves a little time and getting off by 1 results on the training set
is not important.

   The situation when you use the "right" and "wrong" continuous update
methods is even more complicated.  After the forward pass for one
pattern is done it is checked to see if it is right and the error is
added in to a sum of errors.  Then weights are changed.  Then another
pattern is processed in the same way.  When the weight changes are done
for this second pattern they may well ruin the right/wrong decision for
all the previous patterns.  Thus the number of right and wrong patterns
and the average error can be off by quite a lot.  With the off by 1
option off ("f O-") the program still does a forward pass to get the
up to date statistics on the training set.  However when the off by 1
option is on the statistics look like this:

   10 -1   1.92 %  66.35 % 0.42093 ?    31.73 %  40.38 % 0.46316 
   20 -1  12.50 %  64.42 % 0.38829 ?    40.38 %  44.23 % 0.53761 
   30 -1  34.62 %  75.00 % 0.30939 ?    50.96 %  63.46 % 0.37728 

where the ? after the training set error flags the fact that the numbers
are very suspect.

   Another option in the program is to do an extra forward pass through
the training set even when there are no statistics to print out.  The
option to give you up to date statistics is:

f u+

If you are using the periodic update method, quickprop or dbd you don't
need "f u+" as the program will report the correct values anyway.

   The one line form of the summary is the default but it can be turned
off using:

f s-

With this you get nothing whatsoever and normally you won't want this
unless perhaps you are producing your own customized output.

5. The Format Command (f)
-------------------------
   There are several ways to input and output patterns, numbers and
other values and there is one format command, `f', that is used to set
these options.  In the format command a number of options can be given
on a single line as for example in:

f b+ ir oc wB

Input Patterns

   The programs are able to read pattern values in two different
formats.  Real numbers follow the C language notation and must be
separated by a space.  The letters `H' used in recurrent networks is
also allowed.  The letter `x' with a default value of 0.5 is also
allowed.  The `x' character has a default value of 0.5.  The value of
`x' can be changed, for example to make `x' -1 use:

f x -1

Real input format is now the default but if you use the other format
(a compressed binary format) you can re-set the format to real with:

f ir

   The other format is the compressed format, a format consisting of 1s,
0s and the letters `x' and `H'.  In compressed format each value is one
character and it is not necessary to have blanks between the characters.
For example, in compressed format the patterns for xor could be written
out in either of the following ways:

101      10 1
000      00 0
011      01 1
110      11 0

The second example is preferable because it makes it easier to see the
input and the output patterns.

To change to compressed format use:

f ic

Output of Patterns

   Output format is controlled with the `f' command as in:

f or   * output node values using real (the C %f) format
f oc   * output node values using compressed format
f oa   * output node values using analog compressed format
f oe   * output values with e notation

The first sets the output to real numbers.  The second sets the output
to be compressed mode where the value printed will be a `1' when the
unit value is greater than 1.0 - tolerance, a `^' when the value is
above 0.5 but less than 1.0 - tolerance, a `v' when the value is less
than 0.5 but greater than the tolerance.  Below the tolerance value a
`0' is printed.  The tolerance can be changed using the `t' command (not
a part of the format command).  For example, to make all values greater
than 0.8 print as `1' and all values less than 0.2 print as `0' use:

t 0.2

Of course this same tolerance value is also used to check to see if all
the patterns have converged.  The third output format is meant to give
"analog compressed" output.  In this format a `c' is printed when a
value is close enough to its target value.  Otherwise, if the answer is
close to 1, a `1' is printed, if the answer is close to 0, a `0' is
printed, if the answer is above the target but not close to 1, a `^' is
printed and if the answer is below the target but not close to 0, a `v'
is printed.  This output format is designed for problems where the
output is a real number, as for instance, when the problem is to make a
network learn sin(x).  The format "e" writes out node values using
exponential notation with four places to the right of the decimal point.

Breaking up the Output Values

   In the compressed formats the default is to print a blank after every
10 values.  This can be altered using the `B' (for inserting breaks)
option within the format ('f') command.  The use for this command is to
separate output values into logical groups to make the output more
readable.  For instance, you may have 24 output units where it makes
sense to insert blanks after the 4th, 7th and 19th positions.  To do
this, specify:

f B 4 7 19

Then for example the output will look like:

  1 10^0 10^ ^000v00000v0 01000 e 0.17577
  2 1010 01v 0^0000v00000 ^1000 e 0.16341
  3 0101 10^ 00^00v00000v 00001 e 0.16887
  4 0100 0^0 000^00000v00 00^00 e 0.19880

The break option allows up to 20 break positions to be specified.  The
default output format is the real format with 10 numbers per line.  For
the output of real values the option specifies when to print a carriage
return rather than when to print a blank.

Pattern Formats

   There are two different types of problems that back-propagation can
handle, the general type of problem where every output unit can take on
an arbitrary value and the classification type of problem where the goal
is to turn on output unit i and turn off all the other output units when
the pattern is of class i.  The xor problem is an example of the general
type of problem.  For an example of a classification problem, suppose
you have a number of data points scattered about through two-dimensional
space and you have to classify the points as either class 1, class 2 or
class 3.  For a pattern of class 1 you can always set up the output:
"1 0 0", for class 2: "0 1 0" and for class 3: "0 0 1", however doing
the translation to bit patterns can be annoying so another notation can
be used.  Instead of specifying the bit patterns you can set the pattern
format option to classification (as opposed to the default value of
general) like so:

f pc

and then the program will read data in the form:

   1.33   3.61   1   *  shorthand for 1 0 0
   0.42  -2.30   2   *  shorthand for 0 1 0
  -0.31   4.30   3   *  shorthand for 0 0 1

and translate it to the bit string form.  To switch to the general form
use "f pg".  Another benefit of the classification format is that when
the program outputs a status line it will also include the percentage of
correct patterns based on the maximum value rather than just on
tolerance.


Controlling Summaries

   When the program is learning patterns you normally want to have it
print out the status of the learning process at regular intervals.  The
default is to print out a one-line summary of how learning is going
and this is set by using "f s+".  However if you want to customize
exactly what is printed out and you don't want the standard summary, use
"f s-".

Skipping the "running . . ." Message

   Normally whenever you run more training iterations the message,
"running . . ." prints out to reassure you that something is in fact
being done, however this can also be annoying at times.  To get rid of
this message use "f R-" and to bring it back use "f R+".

Ringing the Bell

   To ring the bell when the learning has been completed use "f b+" and
to turn off the bell use "f b-".

Echoing Input

   When you are reading commands from a file it is sometimes worthwhile
to see those commands echoed on the screen.  To do this, use "f e+" and
to turn off the echoing, use "f e-".

Paging

   To set the page size to some value, say, 25, use "f P 25" or to skip
paging use "f P 0".

Making a Copy of Your Session

   To make a copy of what appears on the screen use "f c+" to start
writing to the file "copy" and "f c-" to stop writing to this file.
Ending the session automatically closes this file as well.

Up-To-Date Statistics

   During the ith pass thru the network the program will collect
statistics on how many patterns are correct and how much error there is.
It does this so that it will know when to stop the training.  But it
gets these numbers BEFORE the weights are changed in the ith pass.  In
the case of periodic update methods (the periodic, delta-bar-delta,
quickprop and supersab) this is not much of a problem.  If the off by 1
flag is off ("f O-") there is another forward pass done whenever the
statistics are printed out so you get up to date statistics anyway.  If
the off by 1 flag is on ("f O+") you get the string "-1" after the
number of iterations is printed on the summary line.  Getting the
statistics in the off by 1 form is harmless and it saves a little CPU
time.  When the network converges the "-1" flag will not be shown.

   However with the continuous update methods the weights are changed
after each pattern and this skews the statistics gathered by the
training process by quite a lot.  To get an accurate assessment of how
well the training is going when results are printed on the summary line
you either need to have the off by 1 flag set to "f O+" or you need to
set the up to date statistics flag by: "f u+".  The default is to leave
this flag off: "f u-".  Furthermore, if you are training to get an
accurate assessment of how many iterations it takes to learn the
training set you need to set "f u+" (NOT JUST "f O+"!).  The "u+"
setting makes a check after every complete pass through the training
set.  The "f O+" setting only makes a check when it is time to print
the status line.


6. Taking Training and Testing Patterns from Files (rt,rx,tf)
-------------------------------------------------------------
   In the xor example given above the four patterns were part of the
data file and to read them in the following lines were used:

rt {
1 0 1
0 0 0
0 1 1
1 1 0 }

However it is also convenient to take patterns from a file that contains
nothing but a list of patterns (and possibly comments).  To read a new
set of patterns from some file, patterns, use:

rt patterns

To add an extra group of patterns to the current set you can use:

rx patterns

To read in test patterns from say the file, xtest, do the following:

tf xtest

To evaluate all the test patterns without listing them do "t0".  To list
them, use "t".  To list one particular test pattern, say pattern 3, do
"t 3".


7. Saving and Restoring Weights and Related Values (sw,rw,sw+,swe,swem)
--------------------------------------------------------------
   Sometimes the amount of time and effort needed to produce a set of
weights to solve a problem is so great that it is more convenient to
save the weights rather than constantly recalculate them.  To save the
weights to the current weights file use "sw".  The weights are then
written on a file called "weights" or to the last file name you have
specified.  The weights file looks like: