node2.html

高斯混合模型算法

<P>

<H3><A NAME="SECTION00023300000000000000">
Example:</A>
</H3>
Use function <TT>gloglike(point,mu,sigma)</TT> to compute the
log-likelihoods <!-- MATH
 $\log p(\ensuremath\mathbf{x}_i|q_k,\ensuremath\boldsymbol{\Theta})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="90" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img126.gif"
 ALT="$ \log p(\ensuremath\mathbf{x}_i\vert q_k,\ensuremath\boldsymbol{\Theta})$"></SPAN>.  Don't forget to add the log
of the prior probability <!-- MATH
 $P(q_k|\ensuremath\boldsymbol{\Theta})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="55" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img98.gif"
 ALT="$ P(q_k\vert\ensuremath\boldsymbol{\Theta})$"></SPAN>!
E.g., for the feature set <SPAN CLASS="MATH"><IMG
 WIDTH="18" HEIGHT="25" ALIGN="MIDDLE" BORDER="0"
 SRC="img127.gif"
 ALT="$ x_1$"></SPAN> and class /a/ use
<BR>
<TT>&#187; gloglike([400,1800],mu_a,sigma_a) + log(Pa)</TT>

<P>

<P><P>
<BR>

<H2><A NAME="SECTION00024000000000000000"></A>
<A NAME="sec:discr"></A>
<BR>
Discriminant surfaces
</H2>
For the Bayesian classification in the last section we made use of the
<SPAN  CLASS="textit">discriminant functions</SPAN> <!-- MATH
 $f_k(\ensuremath\mathbf{x}_i) = \log p(\ensuremath\mathbf{x}_i|q_k,\ensuremath\boldsymbol{\Theta}) +
\log P(q_k|\ensuremath\boldsymbol{\Theta})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="236" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img114.gif"
 ALT="$ f_k(\ensuremath\mathbf{x}_i) = \log
p(\ensuremath\mathbf{x}_i\vert q_k,\ensuremath\boldsymbol{\Theta}) + \log P(q_k\vert\ensuremath\boldsymbol{\Theta})$"></SPAN> to classify data points <!-- MATH
 $\ensuremath\mathbf{x}_i$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="16" HEIGHT="25" ALIGN="MIDDLE" BORDER="0"
 SRC="img44.gif"
 ALT="$ \ensuremath\mathbf{x}_i$"></SPAN>.  This corresponds to
establishing <SPAN  CLASS="textit">discriminant surfaces</SPAN> of dimension <SPAN CLASS="MATH"><IMG
 WIDTH="35" HEIGHT="26" ALIGN="MIDDLE" BORDER="0"
 SRC="img128.gif"
 ALT="$ d-1$"></SPAN> in the
vector space for <!-- MATH
 $\ensuremath\mathbf{x}$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="12" HEIGHT="13" ALIGN="BOTTOM" BORDER="0"
 SRC="img20.gif"
 ALT="$ \ensuremath\mathbf{x}$"></SPAN> (dimension <SPAN CLASS="MATH"><IMG
 WIDTH="11" HEIGHT="14" ALIGN="BOTTOM" BORDER="0"
 SRC="img5.gif"
 ALT="$ d$"></SPAN>) to separate regions for the
different classes.

<P>

<H3><A NAME="SECTION00024100000000000000">
Useful formulas and definitions:</A>
</H3>

<UL>
<LI><EM>Discriminant function</EM>: a set of functions <!-- MATH
 $f_k(\ensuremath\mathbf{x})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="36" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img129.gif"
 ALT="$ f_k(\ensuremath\mathbf{x})$"></SPAN> allows
  to classify a sample <!-- MATH
 $\ensuremath\mathbf{x}$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="12" HEIGHT="13" ALIGN="BOTTOM" BORDER="0"
 SRC="img20.gif"
 ALT="$ \ensuremath\mathbf{x}$"></SPAN> into <SPAN CLASS="MATH"><IMG
 WIDTH="11" HEIGHT="14" ALIGN="BOTTOM" BORDER="0"
 SRC="img130.gif"
 ALT="$ k$"></SPAN> classes <SPAN CLASS="MATH"><IMG
 WIDTH="16" HEIGHT="25" ALIGN="MIDDLE" BORDER="0"
 SRC="img91.gif"
 ALT="$ q_k$"></SPAN> if:
  <!-- MATH
 \begin{displaymath}
\ensuremath\mathbf{x}\in q_k \quad \Leftrightarrow \quad f_k(\ensuremath\mathbf{x},\ensuremath\boldsymbol{\Theta}_k) \geq f_l(\ensuremath\mathbf{x},\ensuremath\boldsymbol{\Theta}_l),
  \quad \forall l \neq k
  
\end{displaymath}
 -->
<P></P>
<DIV ALIGN="CENTER" CLASS="mathdisplay">
<IMG
 WIDTH="281" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img131.gif"
 ALT="$\displaystyle \ensuremath\mathbf{x}\in q_k \quad \Leftrightarrow \quad f_k(\ens...
...suremath\mathbf{x},\ensuremath\boldsymbol{\Theta}_l),
\quad \forall l \neq k
$">
</DIV><P></P>
In this case, the <SPAN CLASS="MATH"><IMG
 WIDTH="11" HEIGHT="14" ALIGN="BOTTOM" BORDER="0"
 SRC="img130.gif"
 ALT="$ k$"></SPAN> functions <!-- MATH
 $f_k(\ensuremath\mathbf{x})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="36" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img129.gif"
 ALT="$ f_k(\ensuremath\mathbf{x})$"></SPAN> are called discriminant
  functions.
</LI>
</UL>

<P>
The a-posteriori probability <!-- MATH
 $P(q_k|\ensuremath\mathbf{x}_i)$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="55" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img132.gif"
 ALT="$ P(q_k\vert\ensuremath\mathbf{x}_i)$"></SPAN> that a sample <!-- MATH
 $\ensuremath\mathbf{x}_i$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="16" HEIGHT="25" ALIGN="MIDDLE" BORDER="0"
 SRC="img44.gif"
 ALT="$ \ensuremath\mathbf{x}_i$"></SPAN>
belongs to class <SPAN CLASS="MATH"><IMG
 WIDTH="16" HEIGHT="25" ALIGN="MIDDLE" BORDER="0"
 SRC="img91.gif"
 ALT="$ q_k$"></SPAN> is itself a discriminant function:
<BR>
<DIV ALIGN="CENTER" CLASS="mathdisplay">
<!-- MATH
 \begin{eqnarray*}
\ensuremath\mathbf{x}\in q_k
  & \Leftrightarrow & P(q_k|\ensuremath\mathbf{x}_i) \geq P(q_l|\ensuremath\mathbf{x}_i),\quad \forall l \neq k \\
  & \Leftrightarrow & p(\ensuremath\mathbf{x}_i|q_k)\; P(q_k) \geq p(\ensuremath\mathbf{x}_i|q_l)\; P(q_l),\quad
  \forall l \neq k \\
  & \Leftrightarrow & \log p(\ensuremath\mathbf{x}_i|q_k)+\log P(q_k) \geq \log
  p(\ensuremath\mathbf{x}_i|q_l)+\log P(q_l),\quad \forall l \neq k
\end{eqnarray*}
 -->
<TABLE CELLPADDING="0" ALIGN="CENTER" WIDTH="100%">
<TR VALIGN="MIDDLE"><TD NOWRAP ALIGN="RIGHT"><IMG
 WIDTH="42" HEIGHT="25" ALIGN="MIDDLE" BORDER="0"
 SRC="img133.gif"
 ALT="$\displaystyle \ensuremath\mathbf{x}\in q_k$"></TD>
<TD WIDTH="10" ALIGN="CENTER" NOWRAP><IMG
 WIDTH="17" HEIGHT="25" ALIGN="MIDDLE" BORDER="0"
 SRC="img134.gif"
 ALT="$\displaystyle \Leftrightarrow$"></TD>
<TD ALIGN="LEFT" NOWRAP><IMG
 WIDTH="182" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img135.gif"
 ALT="$\displaystyle P(q_k\vert\ensuremath\mathbf{x}_i) \geq P(q_l\vert\ensuremath\mathbf{x}_i),\quad \forall l \neq k$"></TD>
<TD CLASS="eqno" WIDTH=10 ALIGN="RIGHT">
&nbsp;</TD></TR>
<TR VALIGN="MIDDLE"><TD NOWRAP ALIGN="RIGHT">&nbsp;</TD>
<TD WIDTH="10" ALIGN="CENTER" NOWRAP><IMG
 WIDTH="17" HEIGHT="25" ALIGN="MIDDLE" BORDER="0"
 SRC="img134.gif"
 ALT="$\displaystyle \Leftrightarrow$"></TD>
<TD ALIGN="LEFT" NOWRAP><IMG
 WIDTH="249" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img136.gif"
 ALT="$\displaystyle p(\ensuremath\mathbf{x}_i\vert q_k)\; P(q_k) \geq p(\ensuremath\mathbf{x}_i\vert q_l)\; P(q_l),\quad
\forall l \neq k$"></TD>
<TD CLASS="eqno" WIDTH=10 ALIGN="RIGHT">
&nbsp;</TD></TR>
<TR VALIGN="MIDDLE"><TD NOWRAP ALIGN="RIGHT">&nbsp;</TD>
<TD WIDTH="10" ALIGN="CENTER" NOWRAP><IMG
 WIDTH="17" HEIGHT="25" ALIGN="MIDDLE" BORDER="0"
 SRC="img134.gif"
 ALT="$\displaystyle \Leftrightarrow$"></TD>
<TD ALIGN="LEFT" NOWRAP><IMG
 WIDTH="357" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img137.gif"
 ALT="$\displaystyle \log p(\ensuremath\mathbf{x}_i\vert q_k)+\log P(q_k) \geq \log
p(\ensuremath\mathbf{x}_i\vert q_l)+\log P(q_l),\quad \forall l \neq k$"></TD>
<TD CLASS="eqno" WIDTH=10 ALIGN="RIGHT">
&nbsp;</TD></TR>
</TABLE></DIV>
<BR CLEAR="ALL">

<P>
As in our case the samples <!-- MATH
 $\ensuremath\mathbf{x}$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="12" HEIGHT="13" ALIGN="BOTTOM" BORDER="0"
 SRC="img20.gif"
 ALT="$ \ensuremath\mathbf{x}$"></SPAN> are two-dimensional vectors, the
discriminant surfaces are one-dimensional, i.e., lines at equal values
of the discriminant functions for two distinct classes.

<P>

<H3><A NAME="SECTION00024200000000000000">
Experiment:</A>
</H3>

<DIV ALIGN="CENTER"><A NAME="iso"></A><A NAME="836"></A>
<TABLE>
<CAPTION ALIGN="BOTTOM"><STRONG>Figure 1:</STRONG>
Iso-likelihood lines for the Gaussian pdfs
  <!-- MATH
 ${\cal N}(\ensuremath\boldsymbol{\mu}_{\text{/i/}},\ensuremath\boldsymbol{\Sigma}_{\text{/i/}})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="85" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img1.gif"
 ALT="$ {\cal N}(\ensuremath \boldsymbol {\mu }_{\text {/i/}},\ensuremath \boldsymbol {\Sigma }_{\text {/i/}})$"></SPAN> and <!-- MATH
 ${\cal
N}(\ensuremath\boldsymbol{\mu}_{\text{/e/}},\ensuremath\boldsymbol{\Sigma}_{\text{/e/}})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="89" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img2.gif"
 ALT="$ {\cal N}(\ensuremath \boldsymbol {\mu }_{\text {/e/}},\ensuremath \boldsymbol {\Sigma }_{\text {/e/}})$"></SPAN> (top), and <!-- MATH
 ${\cal
N}(\ensuremath\boldsymbol{\mu}_{\text{/i/}},\ensuremath\boldsymbol{\Sigma}_{\text{/e/}})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="87" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img3.gif"
 ALT="$ {\cal N}(\ensuremath \boldsymbol {\mu }_{\text {/i/}},\ensuremath \boldsymbol {\Sigma }_{\text {/e/}})$"></SPAN> and <!-- MATH
 ${\cal
N}(\ensuremath\boldsymbol{\mu}_{\text{/e/}},\ensuremath\boldsymbol{\Sigma}_{\text{/e/}})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="89" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img2.gif"
 ALT="$ {\cal N}(\ensuremath \boldsymbol {\mu }_{\text {/e/}},\ensuremath \boldsymbol {\Sigma }_{\text {/e/}})$"></SPAN> (bottom).</CAPTION>
<TR><TD>
<DIV CLASS="centerline" ID="par3870" ALIGN="CENTER">
<IMG
 WIDTH="338" HEIGHT="723" ALIGN="BOTTOM" BORDER="0"
 SRC="img138.gif"
 ALT="\includegraphics[height=0.95\textheight]{iso}"></DIV></TD></TR>
</TABLE>
</DIV>
The iso-likelihood lines for the Gaussian pdfs <!-- MATH
 ${\cal
N}(\ensuremath\boldsymbol{\mu}_{\text{/i/}},\ensuremath\boldsymbol{\Sigma}_{\text{/i/}})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="85" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img1.gif"
 ALT="$ {\cal N}(\ensuremath \boldsymbol {\mu }_{\text {/i/}},\ensuremath \boldsymbol {\Sigma }_{\text {/i/}})$"></SPAN> and <!-- MATH
 ${\cal
N}(\ensuremath\boldsymbol{\mu}_{\text{/e/}},\ensuremath\boldsymbol{\Sigma}_{\text{/e/}})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="89" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img2.gif"
 ALT="$ {\cal N}(\ensuremath \boldsymbol {\mu }_{\text {/e/}},\ensuremath \boldsymbol {\Sigma }_{\text {/e/}})$"></SPAN>, which we used before to
model the class /i/ and the class /e/, are plotted in
figure&nbsp;<A HREF="#iso">1</A>, first graph. On the second graph in
figure&nbsp;<A HREF="#iso">1</A>, the iso-likelihood lines for <!-- MATH
 ${\cal
N}(\ensuremath\boldsymbol{\mu}_{\text{/i/}},\ensuremath\boldsymbol{\Sigma}_{\text{/e/}})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="87" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img3.gif"
 ALT="$ {\cal N}(\ensuremath \boldsymbol {\mu }_{\text {/i/}},\ensuremath \boldsymbol {\Sigma }_{\text {/e/}})$"></SPAN> and <!-- MATH
 ${\cal
N}(\ensuremath\boldsymbol{\mu}_{\text{/e/}},\ensuremath\boldsymbol{\Sigma}_{\text{/e/}})$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="89" HEIGHT="28" ALIGN="MIDDLE" BORDER="0"
 SRC="img2.gif"
 ALT="$ {\cal N}(\ensuremath \boldsymbol {\mu }_{\text {/e/}},\ensuremath \boldsymbol {\Sigma }_{\text {/e/}})$"></SPAN> (two pdfs with the
<SPAN  CLASS="textit">same</SPAN> covariance matrix <!-- MATH
 $\ensuremath\boldsymbol{\Sigma}_{\text{/e/}}$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="32" HEIGHT="26" ALIGN="MIDDLE" BORDER="0"
 SRC="img139.gif"
 ALT="$ \ensuremath\boldsymbol{\Sigma}_{\text{/e/}}$"></SPAN>) are represented.  

<P>
On these figures, use a colored pen to join the intersections of the
level lines that correspond to equal likelihoods.  Assume that the
highest iso-likelihood lines (smallest ellipses) are of the same
height. (You can also use <TT>isosurf</TT> in M<SMALL>ATLAB</SMALL> to create a
color plot.)

<P>

<H3><A NAME="SECTION00024300000000000000">
Question:</A>
</H3>
What is the nature of the surface that separates class /i/ from class
/e/ when the two models have <EM>different</EM> variances? Can you
explain the origin of this form?

<P>
What is the nature of the surface that separates class /i/ from class
/e/ when the two models have the <EM>same</EM> variances? Why is it
different from the previous discriminant surface?

<BR>
<P>
Show that in the case of two Gaussian pdfs with <SPAN  CLASS="textit">equal covariance
  matrices</SPAN>, the separation between class 1 and class 2 does not
depend upon the covariance <!-- MATH
 $\ensuremath\boldsymbol{\Sigma}$
 -->
<SPAN CLASS="MATH"><IMG
 WIDTH="15" HEIGHT="14" ALIGN="BOTTOM" BORDER="0"
 SRC="img10.gif"
 ALT="$ \ensuremath\boldsymbol{\Sigma}$"></SPAN> any more.

<BR>
<P>
As a summary, we have seen that Bayesian classifiers with
Gaussian data models separate the classes with combinations of
parabolic surfaces. If the covariance matrices of the models are
equal, the parabolic separation surfaces become simple hyper-planes.

<P>

<P><P>
<BR>

<P>

<DIV CLASS="navigation"><br>
<table border=0 cellspacing=0 callpadding=0 width=100% class="tut_nav">
<tr valign=middle class="tut_nav">
<td valign=middle align=left width=1% class="tut_nav">
<A NAME="tex2html25"
  HREF="node1.html">
<IMG  ALIGN="absmiddle" BORDER="0" ALT="previous" SRC="prev.gif"></A></td><td valign=middle align=left class="tut_nav">&nbsp;<A NAME="tex2html26"
  HREF="node1.html">Gaussian statistics</A></td>
<td align=right valign=middle class="tut_nav"><A NAME="tex2html34"
  HREF="node3.html">Unsupervised training</A>&nbsp;
<A NAME="tex2html33"
  HREF="node3.html">
<IMG  ALIGN="absmiddle" BORDER="0" ALT="next" SRC="next.gif"></A></td>
</tr></table>
</DIV>
<!--End of Navigation Panel-->

</BODY>
</HTML>