opencvref_ml.htm

Simple ellipse fitting example on C++Builder6 + OpenCV1.0.

in the method <code>predict</code>.</p>
<p>The suffix "const" means that prediction does not affect the internal model state, so
the method can be safely called from within different threads.</p>

<!-- *****************************************************************************************
     *****************************************************************************************
     ***************************************************************************************** -->

<hr><h2><a name="ch_nbayes">Normal Bayes Classifier</a></h2>
This is a simple classification model assuming that feature vectors from each class
are normally distributed (though, not necessarily independently distributed),
so the whole data distribution function is assumed to be a Gaussian mixture, one component per a class. 
Using the training data the algorithm estimates mean vectors and covariation matrices for every class,
and then it uses them for prediction.

<p><a name="NBC_paper"></a><b>
[Fukunaga90] K. Fukunaga.
Introduction to Statistical Pattern Recognition. second ed., New York: Academic Press, 1990.
</b>
</p>

<hr><h3><a name="decl_CvNormalBayesClassifier">CvNormalBayesClassifier</a></h3>
<p class="Blurb">Bayes classifier for normally distributed data</p>
<pre>
class CvNormalBayesClassifier : public CvStatModel
{
public:
    CvNormalBayesClassifier();
    virtual ~CvNormalBayesClassifier();

    CvNormalBayesClassifier( const CvMat* _train_data, const CvMat* _responses,
        const CvMat* _var_idx=0, const CvMat* _sample_idx=0 );

    virtual bool train( const CvMat* _train_data, const CvMat* _responses,
        const CvMat* _var_idx = 0, const CvMat* _sample_idx=0, bool update=false );

    virtual float predict( const CvMat* _samples, CvMat* results=0 ) const;
    virtual void clear();

    virtual void save( const char* filename, const char* name=0 );
    virtual void load( const char* filename, const char* name=0 );

    virtual void write( CvFileStorage* storage, const char* name );
    virtual void read( CvFileStorage* storage, CvFileNode* node );
protected:
    ...
};
</pre>

<hr><h3><a name="decl_CvNormalBayesClassifier_train">CvNormalBayesClassifier::train</a></h3>
<p class="Blurb">Trains the model</p>
<pre>
bool CvNormalBayesClassifier::train( const CvMat* _train_data, const CvMat* _responses,
               const CvMat* _var_idx = 0, const CvMat* _sample_idx=0, bool update=false );
</pre>
<p>
The method trains the Normal Bayes classifier. It follows the conventions of
generic <a href="#decl_CvStatModel_train">train</a> "method" with the following limitations:
only CV_ROW_SAMPLE data layout is supported; the input variables are all ordered;
the output variable is categorical (i.e. elements of <code>_responses</code>
must be integer numbers, though the vector may have <code>32fC1</code> type),
missing measurements are not supported.</p>
<p>In addition, there is <code>update</code> flag that identifies, whether the model should
be trained from scratch (<code>update=false</code>) or should be updated using new training data
(<code>update=true</code>).
</p>


<hr><h3><a name="decl_CvNormalBayesClassifier_predict">CvNormalBayesClassifier::predict</a></h3>
<p class="Blurb">Predicts the response for sample(s)</p>
<pre>
float CvNormalBayesClassifier::predict( const CvMat* samples, CvMat* results=0 ) const;
</pre>
<p>
The method <code>predict</code> estimates the most probable classes for the input vectors.
The input vectors (one or more) are stored as rows of the matrix <code>samples</code>.
In case of multiple input vectors, there should be output vector <code>results</code>.
The predicted class for a single input vector is returned by the method.</p>

<!-- *****************************************************************************************
     *****************************************************************************************
     ***************************************************************************************** -->

<hr>
<h2><a name="ch_knn">K Nearest Neighbors</a></h2>

<p>
The algorithm caches all the training samples, and it predicts the response for
a new sample by analyzing a certain number (<em>K</em>) of the nearest neighbors 
of the sample (using voting, calculating weighted sum etc.) The method is 
sometimes referred to as &quot;learning by example&quot;, i.e. for prediction it looks for 
the feature vector
with a known response that is closest to the given vector.
</p>

<hr><h3><a name="decl_CvKNearest">CvKNearest</a></h3>
<p class="Blurb">K Nearest Neighbors model</p>
<pre>
class CvKNearest : public CvStatModel
{
public:

    CvKNearest();
    virtual ~CvKNearest();

    CvKNearest( const CvMat* _train_data, const CvMat* _responses,
                const CvMat* _sample_idx=0, bool _is_regression=false, int max_k=32 );

    virtual bool train( const CvMat* _train_data, const CvMat* _responses,
                        const CvMat* _sample_idx=0, bool is_regression=false,
                        int _max_k=32, bool _update_base=false );

    virtual float find_nearest( const CvMat* _samples, int k, CvMat* results,
        const float** neighbors=0, CvMat* neighbor_responses=0, CvMat* dist=0 ) const;

    virtual void clear();
    int get_max_k() const;
    int get_var_count() const;
    int get_sample_count() const;
    bool is_regression() const;

protected:
    ...
};
</pre>


<hr><h3><a name="decl_CvKNearest_train">CvKNearest::train</a></h3>
<p class="Blurb">Trains the model</p>
<pre>
bool CvKNearest::train( const CvMat* _train_data, const CvMat* _responses,
                        const CvMat* _sample_idx=0, bool is_regression=false,
                        int _max_k=32, bool _update_base=false );
</pre>
<p>
The method trains the K-Nearest model. It follows the conventions of
generic <a href="#decl_CvStatModel_train">train</a> "method" with the following limitations:
only CV_ROW_SAMPLE data layout is supported, the input variables are all ordered,
the output variables can be either categorical (<code>is_regression=false</code>)
or ordered (<code>is_regression=true</code>), variable subsets (<code>var_idx</code>) and
missing measurements are not supported.</p>
<p>The parameter <code>_max_k</code> specifies the number of maximum neighbors that may be
passed to the method <a href="decl_CvKNearest_find_nearest">find_nearest</a>.</p>
<p>The parameter <code>_update_base</code> specifies, whether the model is trained from scratch (<code>_update_base=false</code>), or
it is updated using the new training data
(<code>_update_base=true</code>). In the latter case the parameter <code>_max_k</code>
must not be larger than the original value.</p>


<hr><h3><a name="decl_CvKNearest_find_nearest">CvKNearest::find_nearest</a></h3>
<p class="Blurb">Finds the neighbors for the input vectors</p>
<pre>
float CvKNearest::find_nearest( const CvMat* _samples, int k, CvMat* results=0,
        const float** neighbors=0, CvMat* neighbor_responses=0, CvMat* dist=0 ) const;
</pre>
<p>
For each input vector (which are rows of the matrix <code>_samples</code>)
the method finds <code>k&le;get_max_k()</code> nearest neighbor. In case of regression,
the predicted result will be a mean value of the particular vector's neighbor responses.
In case of classification the class is determined by voting.</p>
<p>
For custom classification/regression prediction, the method can optionally return
pointers to the neighbor vectors themselves (<code>neighbors</code>, array of
<code>k*_samples-&gt;rows</code> pointers), their corresponding output values
(<code>neighbor_responses</code>, a vector of <code>k*_samples-&gt;rows</code> elements)
and the distances from the input vectors to the neighbors
(<code>dist</code>, also a vector of <code>k*_samples-&gt;rows</code> elements).</p>
<p>For each input vector the neighbors are sorted by their distances to the vector.</p>
<p>If only a single input vector is passed, all output matrices are optional and the
predicted value is returned by the method.</p>


<hr>
<h3>Example. Classification of 2D samples from a gaussian mixture with k-nearest classifier</h3>
<pre>
#include "ml.h"
#include "highgui.h"

int main( int argc, char** argv )
{
    const int K = 10;
    int i, j, k, accuracy;
    float response;
    int train_sample_count = 100;
    CvRNG rng_state = cvRNG(-1);
    CvMat* trainData = cvCreateMat( train_sample_count, 2, CV_32FC1 );
    CvMat* trainClasses = cvCreateMat( train_sample_count, 1, CV_32FC1 );
    IplImage* img = cvCreateImage( cvSize( 500, 500 ), 8, 3 );
    float _sample[2];
    CvMat sample = cvMat( 1, 2, CV_32FC1, _sample );
    cvZero( img );

    CvMat trainData1, trainData2, trainClasses1, trainClasses2;

    // form the training samples
    cvGetRows( trainData, &trainData1, 0, train_sample_count/2 );
    cvRandArr( &rng_state, &trainData1, CV_RAND_NORMAL, cvScalar(200,200), cvScalar(50,50) );

    cvGetRows( trainData, &trainData2, train_sample_count/2, train_sample_count );
    cvRandArr( &rng_state, &trainData2, CV_RAND_NORMAL, cvScalar(300,300), cvScalar(50,50) );

    cvGetRows( trainClasses, &trainClasses1, 0, train_sample_count/2 );
    cvSet( &trainClasses1, cvScalar(1) );

    cvGetRows( trainClasses, &trainClasses2, train_sample_count/2, train_sample_count );
    cvSet( &trainClasses2, cvScalar(2) );

    // learn classifier
    CvKNearest knn( trainData, trainClasses, 0, false, K );
    CvMat* nearests = cvCreateMat( 1, K, CV_32FC1);

    for( i = 0; i &lt; img-&gt;height; i++ )
    {
        for( j = 0; j &lt; img-&gt;width; j++ )
        {
            sample.data.fl[0] = (float)j;
            sample.data.fl[1] = (float)i;

            // estimates the response and get the neighbors' labels
            response = knn.find_nearest(&sample,K,0,0,nearests,0);

            // compute the number of neighbors representing the majority
            for( k = 0, accuracy = 0; k &lt; K; k++ )
            {
                if( nearests-&gt;data.fl[k] == response)
                    accuracy++;
            }
            // highlight the pixel depending on the accuracy (or confidence)
            cvSet2D( img, i, j, response == 1 ?
                (accuracy &gt; 5 ? CV_RGB(180,0,0) : CV_RGB(180,120,0)) :
                (accuracy &gt; 5 ? CV_RGB(0,180,0) : CV_RGB(120,120,0)) );
        }
    }

    // display the original training samples
    for( i = 0; i &lt; train_sample_count/2; i++ )
    {
        CvPoint pt;
        pt.x = cvRound(trainData1.data.fl[i*2]);
        pt.y = cvRound(trainData1.data.fl[i*2+1]);
        cvCircle( img, pt, 2, CV_RGB(255,0,0), CV_FILLED );
        pt.x = cvRound(trainData2.data.fl[i*2]);
        pt.y = cvRound(trainData2.data.fl[i*2+1]);
        cvCircle( img, pt, 2, CV_RGB(0,255,0), CV_FILLED );
    }

    cvNamedWindow( "classifier result", 1 );
    cvShowImage( "classifier result", img );
    cvWaitKey(0);

    cvReleaseMat( &trainClasses );
    cvReleaseMat( &trainData );
    return 0;
}

</pre>

<!-- *****************************************************************************************
     *****************************************************************************************
     ***************************************************************************************** -->
<hr><h2><a name="ch_svm">Support Vector Machines</a></h2>

<p>Originally, support vector machines (SVM) was a technique for building an optimal (in some sense)
binary (2-class) classifier. Then the technique has been extended to regression and clustering problems.
SVM is a partial case of kernel-based methods,
it maps feature vectors into higher-dimensional space using some kernel function, and then it builds