hw97.tex

Mitchell的《机器学习〉随书源码

weights of hidden unit {\it n}, type:

{\tt hidtopgm pose.net} {\it image-filename} {\tt 32 30} {\it n}

Invoking {\tt xv} on the image {\it image-filename} 
should then display the range of weights,
with the lowest weights mapped to pixel values of zero, and the highest
mapped to 255.  If the images just look like noise, try retraining using
{\tt facetrain\_init0} (compile with {\tt make facetrain\_init0}), which
initializes the hidden unit weights of a new network to zero, rather
than random values.

\item Do the hidden units seem to weight particular regions of the image
greater than others?  Do particular hidden units seem to be tuned
to different features of some sort?

\end{enumerate}

\subsection{Part II. (optional!)}

Now that you know your way around {\tt facetrain}, it's time to have some fun.
Form a team with one or two other students, and pick some interesting topic of
your own choice -- be creative!!  Run some experiments, and prepare a short
write-up of what your group's idea, experimental results, and any conclusions
you draw (a few pages should be sufficient).  For this part of the assignment,
your group can turn in a single group writeup.

%We'll reserve a class period just after this assignment is due, for brief
%presentations from each of the groups.  So also prepare a one-slide, three
%minute presentation to summarize your group's work, and pick a group member to
%present it to the class.

Some possibilities for experimentation are (but please don't let this list
limit you in any way if you want to try something else):

\begin{itemize}

\item Use the output of the pose recognizer as input to the face
recognizer, and see how this affects performance.  To do this, you will need
to add a mechanism for saving the output units of the pose recognizer and a
mechanism for loading this data into the face recognizer.

\item Learn the {\it location} of some feature in the image, such
as eyes.  You can use {\tt xv} to tell you the coordinates of the feature in
question for each image, which you can then use as your target values.

\item Take a look at the additional data from an earlier year's class in


{\tt /afs/cs.cmu.edu/project/theo-8/ml94faces}.  

What techniques can you
employ to train nets to generalize better from one dataset to the other?

\item How do nets perform if trained on more than one concept at once?
Do representations formed for multiple concepts interfere with each other in
the hidden layer, or perhaps augment each other?

\item Use the image package, weight visualization utility, and/or
anything else you might have available to try to understand better what the
network has actually learned.  Using this information, what do you think the
network is learning?  Can you exploit this information to improve
generalization?

\item Change the input or output encodings to try to improve
generalization accuracy.

\item Vary the number of hidden units, the number of training examples,
the number of epochs, the momentum and learning rate, or whatever else you
want to try, with the goal of getting the greatest possible discrepancy
between train and test set accuracy (i.e., how badly can you make the network
overfit), and the smallest possible discrepancy (i.e., what is the best
performance you can achieve).

\end{itemize}

\section{Documentation}
\label{docs}

The code for this assignment is broken into several modules:

\begin{itemize}
\item {\tt pgmimage.c}, {\tt pgmimage.h}: the image package.  Supports
read/write of PGM image files and pixel access/assignment.  Provides
an {\tt IMAGE} data structure, and an {\tt IMAGELIST} data structure
(an array of pointers to images; useful when handling many images). 
{\bf You will not need to modify any code in this module to complete
the assignment.}

\item {\tt backprop.c}, {\tt backprop.h}: the neural network package.
Supports three-layer fully-connected feedforward networks, using the
backpropagation algorithm for weight tuning.  Provides high level
routines for creating, training, and using networks.  {\bf You will not
need to modify any code in this module to complete the assignment.}

\item {\tt imagenet.c}: interface routines for loading images into
the input units of a network, and setting up target vectors for training.
You will need to modify the routine {\tt load\_target}, when
implementing the face recognizer and the pose recognizer, to set
up appropriate target vectors for the output encodings you choose.

\item {\tt facetrain.c}: the top-level program which uses all of the
modules above to implement a ``TA'' recognizer.  You will need to
modify this code to change network sizes and learning parameters,
both of which are trivial changes.  The performance evaluation routines
{\tt performance\_on\_imagelist()} and {\tt evaluate\_performance()} are
also in this module; you will need to modify these for your face
and pose recognizers.

\item {\tt hidtopgm.c}: the hidden unit weight visualization utility.
It's not necessary modify anything here, although it may be interesting
to explore some of the numerous possible alternate visualization schemes.
\end{itemize}

Although you'll only need to modify code in {\tt imagenet.c} and
{\tt facetrain.c}, feel free to modify anything you want in any of
the files if it makes your life easier or if it allows you to do
a nifty experiment.

\subsection {\tt facetrain}

\subsubsection{Running {\tt facetrain}}
\label{RUNFACE}

{\tt facetrain} has several options which can be specified on the
command line.  This section briefly describes how each option
works.  A very short summary of this information can be obtained
by running {\tt facetrain} with no arguments.

\begin{description}
\item {\tt -n <network file>} - this option either loads an existing
network file, or creates a new one with the given name.  At the end
of training, the neural network will be saved to this file.

\item {\tt -e <number of epochs>} - this option specifies the number
of training epochs which will be run.  If this option is not
specified, the default is 100.

\item {\tt -T} - for test-only mode (no training).  Performance
will be reported on each of the three datasets specified, and
those images misclassified will be listed, along with the corresponding
output unit levels.

\item {\tt -s <seed>} - an integer which will be used as the
seed for the random number generator.  The default seed is 102194
(guess what day it was when I wrote this document).  This allows you to
reproduce experiments if necessary, by generating the same sequence of
random numbers.  It also allows you to try a different set of random
numbers by changing the seed.

\item {\tt -S <number of epochs between saves>} - this option specifies
the number of epochs between saves.  The default is 100, which means
that if you train for 100 epochs (also the default), the network is
only saved when training is completed.

\item {\tt -t <training image list>} - this option specifies a text
file which contains a list of image pathnames, one per line, that
will be used for training.  If this option is not specified, it
is assumed that no training will take place ($epochs = 0$), and
the network will simply be run on the test sets.  In this case,
the statistics for the training set will all be zeros.

\item {\tt -1 <test set 1 list>} - this option specifies a text
file which contains a list of image pathnames, one per line, that
will be used as a test set.  If this option is not specified,
the statistics for test set 1 will all be zeros.

\item {\tt -2 <test set 2 list>} - same as above, but for test set 2.
The idea behind having two test sets is that one can be used as part
of the train/test paradigm, in which training is stopped when performance
on the test set begins to degrade.  The other can then be used as a
``real'' test of the resulting network.

\end{description}

\subsubsection{Interpreting the output of {\tt facetrain}}

When you run {\tt facetrain}, it will first read in
all the data files and print a bunch of lines regarding
these operations. Once all the data is loaded, it will
begin training. At this point, the network's training 
and test set performance is outlined in one line per epoch.
For each epoch, the following 
performance measures are output:

{\tt <epoch> <delta> <trainperf> <trainerr> <t1perf> <t1err>
<t2perf> <t2err>}

These values have the following meanings:

\begin{description}
\item {\tt epoch} is the number of the epoch just completed; it follows
that a value of 0 means that no training has yet been performed.

\item {\tt delta} is the sum of all $\delta$ values on the hidden and
output units as computed during backprop, over all training examples
for that epoch.

\item {\tt trainperf} is the percentage of examples in the training set
which were correctly classified.

\item {\tt trainerr} is the average, over all training examples,
of the error function $\frac{1}{2} \sum (t_{i} - o_{i})^{2}$, where
$t_{i}$ is the target value for output unit $i$ and $o_{i}$ is the actual
output value for that unit.

\item {\tt t1perf} is the percentage of examples in test set 1
which were correctly classified.

\item {\tt t1err} is the average, over all examples in test set 1,
of the error function described above.

\item {\tt t2perf} is the percentage of examples in test set 2
which were correctly classified.

\item {\tt t2err} is the average, over all examples in test set 2,
of the error function described above.
\end{description}

\subsection{Tips}

Although you do not have to modify the image or network packages,
you will need to know a little bit about the routines and data structures
in them, so that you can easily implement new output encodings for
your networks.  The following sections describe each of the packages
in a little more detail.  You can look at {\tt imagenet.c},
{\tt facetrain.c}, and {\tt facerec.c} to see how the routines are
actually used.  

In fact, it is probably a good idea to look over {\tt facetrain.c}
first, to see how the training process works.  You will notice
that {\tt load\_target()} from {\tt imagenet.c} is called to set
up the target vector for training.  You will also notice the
routines which evaluate performance and compute error statistics,
{\tt performance\_on\_imagelist()} and {\tt evaluate\_performance()}.
The first routine iterates through a set of images, computing the
average error on these images, and the second routine computes
the error and accuracy on a single image.

You will almost certainly not need to use all of the information
in the following sections, so don't feel like you need to know
everything the packages do.  You should view these sections
as reference guides for the packages, should you need information
on data structures and routines.