nn.faq

本例是一个神经网络仿真应用程序

  "You are almost not knowing what you are actually doing
   when using backpropagation"  :-)
it has pretty much success on practical applications and is
relatively easy to apply.

It is for the training of layered (i.e., nodes are grouped
in layers) feedforward (i.e., the arcs joining nodes are
unidirectional, and there are no cycles) nets.

Back-propagation needs a teacher that knows the correct output for any
input ("supervised learning") and uses gradient descent on the error
(as provided by the teacher) to train the weights.  The activation
function is (usually) a sigmoidal (i.e., bounded above and below, but
differentiable) function of a weighted sum of the nodes inputs.

The use of a gradient descent algorithm to train its weights makes it
slow to train; but being a feedforward algorithm, it is quite rapid during
the recall phase.

Literature:
  Rumelhart, D. E. and McClelland, J. L. (1986): 
  Parallel Distributed Processing: Explorations in the
  Microstructure of Cognition (volume 1, pp 318-362). 
  The MIT Press. 
(this is the classic one) or one of the dozens of other books 
or articles on backpropagation  :->

------------------------------------------------------------------------

-A7.)  How many learning methods for NNs exist ?  Which ?

There are many many learning methods for NNs by now. Nobody can know 
exactly how many.
New ones (at least variations of existing ones) are invented every
week. Below is a collection of some of the most well known methods;
not claiming to be complete.

The main categorization of these methods is the distiction of 
supervised from unsupervised learning:

- In supervised learning, there is a "teacher" who in the learning
  phase "tells" the net how well it performs ("reinforcement learning")
  or what the correct behavior would have been ("fully supervised learning").

- In unsupervised learning the net is autonomous: it just looks at
  the data it is presented with, finds out about some of the
  properties of the data set and learns to reflect these properties 
  in its output. What exactly these properties are, that the network
  can learn to recognise, depends on the particular network model and
  learning method.
  
Many of these learning methods are closely connected with a certain
(class of) network topology.

Now here is the list, just giving some names:
  
1. UNSUPERVISED LEARNING (i.e. without a "teacher"):
       1). Feedback Nets:
	   a). Additive Grossberg (AG)
	   b). Shunting Grossberg (SG)
	   c). Binary Adaptive Resonance Theory (ART1)
	   d). Analog Adaptive Resonance Theory (ART2, ART2a)
	   e). Discrete Hopfield (DH)
	   f). Continuous Hopfield (CH)
	   g). Discrete Bidirectional Associative Memory (BAM)
	   h). Temporal Associative Memory (TAM)
	   i). Adaptive Bidirectional Associative Memory (ABAM)
           j). Kohonen Self-organizing Map (SOM)
           k). Kohonen Topology-preserving Map (TPM)
       2). Feedforward-only Nets:
	   a). Learning Matrix (LM)
	   b). Driver-Reinforcement Learning (DR)
	   c). Linear Associative Memory (LAM)
	   d). Optimal Linear Associative Memory (OLAM)
	   e). Sparse Distributed Associative Memory (SDM)
	   f). Fuzzy Associative Memory (FAM)
	   g). Counterprogation (CPN)

2. SUPERVISED LEARNING (i.e. with a "teacher"):
       1). Feedback Nets:
	   a). Brain-State-in-a-Box (BSB)
	   b). Fuzzy Congitive Map (FCM)
	   c). Boltzmann Machine (BM)
           d). Mean Field Annealing (MFT)
           e). Recurrent Cascade Correlation (RCC)
	   f). Learning Vector Quantization (LVQ)
       2). Feedforward-only Nets:
	   a). Perceptron
	   b). Adaline, Madaline
	   c). Backpropagation (BP)
	   d). Cauchy Machine (CM)
	   e). Adaptive Heuristic Critic (AHC)
           f). Time Delay Neural Network (TDNN)
	   g). Associative Reward Penalty (ARP)
	   h). Avalanche Matched Filter (AMF)
           i). Backpercolation (Perc)
           j). Artmap
           k). Adaptive Logic Network (ALN)
           l). Cascade Correlation (CasCor)

------------------------------------------------------------------------

-A8.)  What about Genetic Algorithms ?

[preliminary]
[Who will write a better introduction?]

There are a number of definitions of GA (Genetic Algorithm).
A possible one is

  A GA is an optimization program 
  that starts with some encoded procedure,  (Creation of Life :-> )
  mutates it stochastically,                (Get cancer or so :-> )
  and uses a selection process              (Darwinism)
  to prefer the mutants with high fitness
  and perhaps a recombination process       (Make babies :-> )
  to combine properties of (preferably) the succesful mutants.

There is a newsgroup that is dedicated to Genetic Algorithms 
called comp.ai.genetic.
Some GA discussion also tends to happen in comp.ai.neural-nets.
Another loosely relevant group is comp.theory.self-org-sys.
There is a GA mailing list which you can subscribe to by
sending a request to [e-mail]
You can also try anonymous ftp to
  ftp.aic.nrl.navy.mil
in the /pub/galist directory. There are papers and some software.

For more details see (for example):

"Genetic Algorithms in Search Optimisation and Machine Learning"
by David Goldberg (Addison-Wesley 1989, 0-201-15767-5) or 

"Handbook of Genetic Algorithms"
edited by Lawrence Davis (Van Nostrand Reinhold 1991 0-442-00173-8) or

"Classifier Systems and Genetic Algorithms"
L.B. Booker, D.E. Goldberg and J.H. Holland, Techreport No. 8 (April 87),
Cognitive Science and Machine Intelligence Laboratory, University of Michigan
  also reprinted in :
Artificial Intelligence, Volume 40 (1989), pages 185-234

------------------------------------------------------------------------

-A9.)  What about Fuzzy Logic ?

[preliminary]
[Who will write an introduction?]

Fuzzy Logic is an area of research based on the work of L.A. Zadeh.
It is a departure from classical two-valued sets and logic, that uses
"soft" linguistic (e.g. large, hot, tall) system variables and a
continuous range of truth values in the interval [0,1], rather than
strict binary (True or False) decisions and assignments.
Fuzzy logic is used where a system is difficult to model, is
controlled by a human operator or expert, or where ambiguity or
vagueness is common.  A typical fuzzy system consists of a rule base,
membership functions, and an inference procedure.

Most Fuzzy Logic discussion takes place in the newsgroup comp.ai.fuzzy,
but there is also some work (and discussion) about combining fuzzy
logic with Neural Network approaches in comp.ai.neural-nets.

For more details see (for example):

Klir, G.J. and Folger, T.A., Fuzzy Sets, Uncertainty, and 
Information, Prentice-Hall, Englewood 
Cliffs, N.J., 1988.

Kosko, B., Neural Networks and Fuzzy Systems, Prentice Hall, 
Englewood Cliffs, NJ, 1992.

------------------------------------------------------------------------

-A10.) Good introductory literature about Neural Networks ?

0.) The best (subjectively, of course -- please don't flame me):

Hecht-Nielsen, R. (1990). Neurocomputing. Addison Wesley.
Comments: "A good book", "comprises a nice historical overview and a chapter
about NN hardware. Well structured prose. Makes important concepts clear."

Hertz, J., Krogh, A., and Palmer, R. (1991). Introduction to the Theory of 
Neural Computation. Addison-Wesley: Redwood City, California.
ISBN 0-201-50395-6 (hardbound) and 0-201-51560-1 (paperbound)
Comments: "My first impression is that this one is by far the best book on 
the topic. And it's below $30 for the paperback."; "Well written, theoretical
(but not overwhelming)"; It provides a good balance of model development, 
computational algorithms, and applications. The mathematical derivations 
are especially well done"; "Nice mathematical analysis on the mechanism of 
different learning algorithms"; "It is NOT for mathematical beginner.
If you don't have a good grasp of higher level math, this book can
be really tough to get through."


1.) Books for the beginner:

Aleksander, I. and Morton, H. (1990). An Introduction to Neural Computing.
Chapman and Hall. (ISBN 0-412-37780-2).
Comments: "This book seems to be intended for the first year of university 
education."

Beale, R. and Jackson, T. (1990). Neural Computing, an Introduction.
Adam Hilger, IOP Publishing Ltd : Bristol. (ISBN 0-85274-262-2).
Comments: "It's clearly written.  Lots of hints as to how to get the
adaptive models covered to work (not always well explained in the
original sources).  Consistent mathematical terminology.  Covers
perceptrons, error-backpropagation, Kohonen self-org model, Hopfield
type models, ART, and associative memories."

Dayhoff, J. E. (1990). Neural Network Architectures: An Introduction.
Van Nostrand Reinhold: New York.
Comments: "Like Wasserman's book, Dayhoff's book is also very easy to 
understand".

McClelland, J. L. and Rumelhart, D. E. (1988).
Explorations in Parallel Distributed Processing: Computational Models of 
Cognition and Perception (software manual). The MIT Press.
Comments: "Written in a tutorial style, and includes 2 diskettes of NN 
simulation programs that can be compiled on MS-DOS or Unix (and they do 
too !)"; "The programs are pretty reasonable as an introduction to some 
of the things that NNs can do."; "There are *two* editions of this book.  
One comes with disks for the IBM PC, the other comes with disks for the 
Macintosh".

McCord Nelson, M. and Illingworth, W.T. (1990). A Practical Guide to Neural 
Nets. Addison-Wesley Publishing Company, Inc. (ISBN 0-201-52376-0).
Comments: "No formulas at all( ==> no good)"; "It does not have much 
detailed model development (very few equations), but it does present many 
areas of application.  It includes a chapter on current areas of research. 
A variety of commercial applications is discussed in chapter 1.  It also 
includes a program diskette with a fancy graphical interface (unlike the 
PDP diskette)".

Orchard, G.A. & Phillips, W.A. (1991). Neural Computation: A
Beginner's Guide. Lawrence Earlbaum Associates: London.
Comments: "Short user-friendly introduction to the area, with a 
non-technical flavour. Apparently accompanies a software package, but I 
haven't seen that yet".

Wasserman, P. D. (1989). Neural Computing: Theory & Practice.
Van Nostrand Reinhold: New York. (ISBN 0-442-20743-3)
Comments: "Wasserman flatly enumerates some common architectures from an 
engineer's perspective ('how it works') without ever addressing the underlying 
fundamentals ('why it works') - important basic concepts such as clustering, 
principal components or gradient descent are not treated.  It's also full of 
errors, and unhelpful diagrams drawn with what appears to be PCB board layout 
software from the '70s. For anyone who wants to do active research in the
field I consider it quite inadequate"; "Okay, but too shallow"; "Quite
easy to understand";
"The best bedtime reading for Neural Networks.  I have given
this book to numerous collegues who want to know NN basics, but who never
plan to implement anything.  An excellent book to give your manager."


2.) The classics:

Kohonen, T. (1984). Self-organization and Associative Memory. Springer-Verlag:
New York. (2nd Edition: 1988; 3rd edition: 1989). 
Comments: "The section on Pattern mathematics is excellent."

Rumelhart, D. E. and McClelland, J. L. (1986). Parallel Distributed 
Processing: Explorations in the Microstructure of Cognition (volumes 1 & 2). 
The MIT Press. 
Comments: "As a computer scientist I found the two Rumelhart and McClelland 
books really heavy going and definitely not the sort of thing to read if you 
are a beginner."; "It's quite readable, and affordable (about $65 for both 
volumes)."; "THE Connectionist bible.".


3.) Introductory journal articles:

Hinton, G. E. (1989). Connectionist learning procedures.
Artificial Intelligence, Vol. 40, pp. 185--234.
Comments: "One of the better neural networks overview papers, although the
distinction between network topology and learning algorithm is not always
very clear.  Could very well be used as an introduction to neural networks."

Knight, K. (1990). Connectionist, Ideas and Algorithms. Communications of 
the ACM. November 1990. Vol.33 nr.11, pp 59-74.
Comments:"A good article, while it is for most people easy to find a copy of
this journal."

Kohonen, T. (1988). An Introduction to Neural Computing. Neural Networks,
vol. 1, no. 1. pp. 3-16.