neuralnetrecognition.html

基于神经网络的手写体识别程序

	void Backpropagate( double *actualOutput, double *desiredOutput, 
		UINT count );

	VectorLayers m_Layers;
};


// Layer class

class NNLayer
{
public:
	NNLayer( LPCTSTR str, NNLayer* pPrev = NULL );
	virtual ~NNLayer();
	
	void Calculate();
	
	void Backpropagate( std::vector< double >& dErr_wrt_dXn /* in */, 
		std::vector< double >& dErr_wrt_dXnm1 /* out */, 
		double etaLearningRate );

	NNLayer* m_pPrevLayer;
	VectorNeurons m_Neurons;
	VectorWeights m_Weights;
};


// Neuron class

class NNNeuron
{
public:
	NNNeuron( LPCTSTR str );
	virtual ~NNNeuron();

	void AddConnection( UINT iNeuron, UINT iWeight );
	void AddConnection( NNConnection const & conn );

	double output;

	VectorConnections m_Connections;
};


// Connection class

class NNConnection
{
public: 
	NNConnection(UINT neuron = ULONG_MAX, UINT weight = ULONG_MAX);
	virtual ~NNConnection();

	UINT NeuronIndex;
	UINT WeightIndex;
};


// Weight class

class NNWeight
{
public:
	NNWeight( LPCTSTR str, double val = 0.0 );
	virtual ~NNWeight();

	double value;
};
</PRE>

<P>As you can see from the above, class <CODE>NeuralNetwork</CODE> stores a vector of pointers to layers in the neural network, which are represented by class <CODE>NNLayer</CODE>.  There is no special function to add a layer (there probably should be one); simply use the <CODE>std::vector::push_back()</CODE> function.  The <CODE>NeuralNetwork</CODE> class also provides the two primary interfaces with the outside world, namely, a function to forward propagate the neural network (the <CODE>Calculate()</CODE> function) and a function to <CODE>Backpropagate()</CODE> the neural network so as to train it.</P>

<P>Each <CODE>NNLayer</CODE> stores a pointer to the previous layer, so that it knows where to look for its input values.  In addition, it stores a vector of pointers to the neurons in the layer, represented by class <CODE>NNNeuron</CODE>, and a vector of pointers to weights, represented by class <CODE>NNWeight</CODE>.  Similar to the <CODE>NeuralNetwork</CODE> class, the pointers to the neurons and to the weights are added using the <CODE>std::vector::push_back()</CODE> function.  Finally, the <CODE>NNLayer</CODE> class includes functions to <CODE>Calculate()</CODE> the output values of neurons in the layer, and to <CODE>Backpropagate()</CODE> them; in fact, the corresponding function in the <CODE>NeuralNetwork</CODE> class simply iterate through all layers in the network and call these functions.</P>

<P>Each <CODE>NNNeuron</CODE> stores a vector of connections that tell the neurons where to get its inputs.  Connections are added using the <CODE>NNNeuron::AddConnection()</CODE> function, which takes an index to a neuron and an index to a weight, constructs a <CODE>NNConnection</CODE> object, and <CODE>push_back()</CODE>'s the new connection onto the vector of connections.  Each neuron also stores its own output value, even though it's the <CODE>NNLayer</CODE> class that is responsible for calculating the actual value of the output and storing it there.  The <CODE>NNConnection</CODE> and <CODE>NNWeight</CODE> classes respectively store obviously-labeled information.</P>

<P>One legitimate question about the class structure is, why are there separate classes for the weights and the connections?  According to the diagram above, each connection has a weight, so why not put them in the same class?  The answer lies in the fact that weights are often shared between connections.  In fact, the convolutional neural network of this program specifically shares weights amongst its connections.  So, for example, even though there might be several hundred neurons in a layer, there might only be a few dozen weights due to sharing.  By making the <CODE>NNWeight</CODE> class separate from the <CODE>NNConnection</CODE> class, this sharing is more readily accomplished.</P>


<BR><A HREF="#topmost"><FONT SIZE="-6" COLOR="">go back to top</FONT></A>

<BR><BR>
<A name="ForwardPropagation"/>
<h3>Forward Propagation</h2>

<P>Forward propagation is the process whereby each of all of the neurons calculates its output value, based on inputs provided by the output values of the neurons that feed it.</P>

<P>In the code, the process is initiated by calling <CODE>NeuralNetwork::Calculate()</CODE>.  <CODE>NeuralNetwork::Calculate()</CODE> directly sets the values of neurons in the input layer, and then iterates through the remaining layers, calling each layer's <CODE>NNLayer::Calculate()</CODE> function.  This results in a forward propagation that's completely sequential, starting from neurons in the input layer and progressing through to the neurons in the output layer.  A sequential calculation is not the only way to forward propagate, but it's the most straightforward.  Here's simplified code, which takes a pointer to a C-style array of <CODE>double</CODE>s representing the input to the neural network, and stores the output of the neural network to another C-style array of <CODE>double</CODE>s:</P>

<PRE>// simplified code

void NeuralNetwork::Calculate(double* inputVector, UINT iCount, 
			double* outputVector /* =NULL */, UINT oCount /* =0 */)
							  
{
	VectorLayers::iterator lit = m_Layers.begin();
	VectorNeurons::iterator nit;
	
	// first layer is input layer: directly set outputs of all of its neurons
	// to the given input vector
	
	if ( lit &lt; m_Layers.end() )  
	{
		nit = (*lit)-&gt;m_Neurons.begin();
		int count = 0;
		
		ASSERT( iCount == (*lit)-&gt;m_Neurons.size() );  // there should be exactly one neuron per input
		
		while( ( nit &lt; (*lit)-&gt;m_Neurons.end() ) && ( count &lt; iCount ) )
		{
			(*nit)-&gt;output = inputVector[ count ];
			nit++;
			count++;
		}
	}
	
	// iterate through remaining layers, calling their Calculate() functions
	
	for( lit++; lit&lt;m_Layers.end(); lit++ )
	{
		(*lit)-&gt;Calculate();
	}
	
	// load up output vector with results
	
	if ( outputVector != NULL )
	{
		lit = m_Layers.end();
		lit--;
		
		nit = (*lit)-&gt;m_Neurons.begin();
		
		for ( int ii=0; ii&lt;oCount; ++ii )
		{
			outputVector[ ii ] = (*nit)-&gt;output;
			nit++;
		}
	}
}
</PRE>

<P>Inside the layer's <CODE>Calculate()</CODE> function, the layer iterates through all neurons in the layer, and for each neuron the output is calculated according to the feed-forward formula given above, namely</P>

<IMG SRC="Images/ForwardPropagationEquation.gif" WIDTH="227" HEIGHT="53" BORDER="0" ALT="General feed-forward equation">

<P>This formula is applied by iterating through all connections for the neuron, and for each connection, obtaining the corresponding weight and the corresponding output value from a neuron in the previous layer:</P>

<PRE>// simplified code

void NNLayer::Calculate()
{
	ASSERT( m_pPrevLayer != NULL );
	
	VectorNeurons::iterator nit;
	VectorConnections::iterator cit;
	
	double dSum;
	
	for( nit=m_Neurons.begin(); nit&lt;m_Neurons.end(); nit++ )
	{
		NNNeuron& n = *(*nit);  // to ease the terminology
		
		cit = n.m_Connections.begin();
		
		ASSERT( (*cit).WeightIndex &lt; m_Weights.size() );
		
		// weight of the first connection is the bias; its neuron-index is ignored

		dSum = m_Weights[ (*cit).WeightIndex ]-&gt;value;  
		
		for ( cit++ ; cit&lt;n.m_Connections.end(); cit++ )
		{
			ASSERT( (*cit).WeightIndex &lt; m_Weights.size() );
			ASSERT( (*cit).NeuronIndex &lt; m_pPrevLayer-&gt;m_Neurons.size() );
			
			dSum += ( m_Weights[ (*cit).WeightIndex ]-&gt;value ) * 
				( m_pPrevLayer-&gt;m_Neurons[ (*cit).NeuronIndex ]-&gt;output );
		}
		
		n.output = SIGMOID( dSum );
		
	}
	
}
</PRE>

<P>In this code, <CODE>SIGMOID</CODE> is <CODE>#define</CODE>d to the activation function, which is described in the next section.
</P>

<BR><A HREF="#topmost"><FONT SIZE="-6" COLOR="">go back to top</FONT></A>

<BR><BR>
<A name="ActivationFunction"/>
<h3>The Activation Function (or, &quot;Sigmoid&quot; or &quot;Squashing&quot; Function)</h2>

<P>Selection of a good activation function is an important part of the design of a neural network.  Generally speaking, the activation function should be symmetric, and the neural network should be trained to a value that is lower than the limits of the function.</P>

<P>One function that should <I><B>never</B></I> be used as the activation function is the classical sigmoid function (or &quot;logistic&quot; function), defined as <IMG SRC="Images/LogisticFunction.gif" WIDTH="110" HEIGHT="42" BORDER="0" ALT="Logisitc function">.  It should never be used since it is not symmetric: its value approaches +1 for increasing x, but for decreasing x its value approaches zero (i.e., it does not approach -1 which it should for symmetry).  The reason the logistic function is even mentioned here is that there are many articles on the web that recommend its use in neural networks, for example, <A HREF="http://en.wikipedia.org/wiki/Sigmoid_function" target=_newwin>Sigmoid function&nbsp;<IMG SRC="Images/ExternalLink.gif" WIDTH="14" HEIGHT="14" BORDER="0" ALT="External Link"></A> in the Wikipedia.  In my view, this is a poor recommendation and should be avoided.</P>

<P>One good selection for the activation function is the hyperbolic tangent, or <IMG SRC="Images/HyperbolicTangent.gif" WIDTH="117" HEIGHT="22" BORDER="0" ALT="Hyperbolic tangent">.  This function is a good choice because it's completely symmetric, as shown in the following graph.  If used, then do not train the neural network to &plusmn;1.0.  Instead, choose an intermediate value, like &plusmn;0.8.</P>

<IMG SRC="Images/HyperbolicTangentGraph.gif" WIDTH="327" HEIGHT="220" BORDER="0" ALT="Graph of hyperbolic tangent">

<P>Another reason why hyperbolic tangent is a good choice is that it's easy to obtain its derivative.  Yes, sorry, but a bit of calculus is needed in neural networks.  Not only is it easy to obtain its derivative, but also the value of derivative can be expressed in terms of the output value (i.e., as opposed to the input value).  More specifically, given that</P>

<P><IMG SRC="Images/DerivativeEquation1.gif" ALIGN="center" WIDTH="215" HEIGHT="50" BORDER="0" ALT="Basic tanh function">&nbsp;, where (using the notation established above) y is the input to the function (corresponding to the activation value of a neuron) and x is the output of the neuron.</P>