labman1.tex

一本关于dsp的原著教材

\documentclass[twoside,a4paper,titlepage]{article}
\usepackage{array,amsmath,amssymb,rotating,epsfig,fancyheadings}


\newcommand{\mat}[1]{{\tt >> #1} \\}
\newcommand{\com}[1]{{\tt #1}}
\newcommand{\expl}[1]{%
\begin{turn}{180}%
\parbox{\textwidth}{\em #1}%
\end{turn}%
}
\newcommand{\tit}[1]{{\noindent \bf #1 \\}}
\newcommand{\tab}{\hspace{1em}}

\setlength{\hoffset}{-1in}
\setlength{\voffset}{-1in}

\setlength{\topskip}{0cm}
\setlength{\headheight}{0cm}
\setlength{\headsep}{0cm}

\setlength{\textwidth}{16cm}
\setlength{\evensidemargin}{2.5cm}
\setlength{\oddsidemargin}{2.5cm}

\setlength{\textheight}{24.7cm}
\setlength{\topmargin}{1.5cm}
\setlength{\headheight}{0.5cm}
\setlength{\headsep}{0.5cm}

\pagestyle{fancyplain}

\begin{document}

\begin{titlepage}
\setcounter{page}{-1}

\centerline{Ecole Polytechnique F\'ed\'erale de Lausanne}

\vspace{5cm}

\begin{center}
\huge
Session 1/2\,: \\
Introduction to Gaussian Statistics \\
and Statistical Pattern Recognition
\end{center}

\vfill

\centerline{\includegraphics[width=7cm]{cover.eps.gz}}

\vfill

\noindent
\begin{tabular}{lll}
Course\,: & Speech processing and speech recognition & \\
& & \\
Teacher\,: & Prof. Herv\'e Bourlard \hspace{1cm} & {\tt [e-mail]} \\
& & \\
Assistants\,: & Sacha Krstulovi\'c & {\tt [e-mail]}\\
 & Mathew Magimai-Doss & {\tt [e-mail]} 
\end{tabular}

\end{titlepage}

\thispagestyle{empty}
%%%%%%%%%
%%%%%%%%%
\section*{Guidelines}
%%%%%%%%%
%%%%%%%%%
The following lab manual is structured as follows\,:
\begin{itemize}
\item each section corresponds to a theme
\item each subsection corresponds to a separate experiment.
\end{itemize}
The subsections begin with useful formulas and definitions that will be put
in practice during the experiments. These are followed by the description
of the experiment and by an example of how to realize it in {\sc Matlab}.

If you follow the examples literally, you will be able to progress into the
lab session without worrying about the experimental implementation
details. If you have ideas for better {\sc Matlab} implementations, you are
welcome to put them in practice provided you don't loose too much time\,:
remember that a lab session is no more than 3 hours long.

The subsections also contain questions that you should think about.
Corresponding answers are given right after, in case of problem. You can
read them right after the question, {\em but}\,: the purpose of this lab is
to make you

\medskip
\centerline{\LARGE \bf Think !}
\medskip

If you get lost with some of the questions or some of the explanations, DO
ASK the assistants or the teacher for help\,: they are here to make the course
understood. There is no such thing as a stupid question, and the only
obstacle to knowledge is laziness.

\bigskip
Have a nice lab;

\hfill Teacher \& Assistants \hspace{2cm}


\vfill
%%%%%%%%%
%%%%%%%%%
\section*{Before you begin...}
%%%%%%%%%
%%%%%%%%%
If this lab manual has been handed to you as a hardcopy\,:
\begin{enumerate}
\item get the lab package from \\
	\hspace{2cm}{\tt ftp.idiap.ch/pub/sacha/labs/Session1.tgz}
\item un-archive the package\,: \\
	{\tt \% gunzip Session1.tgz \\
	\% tar xvf Session1.tar}
\item change directory\,: \\
	{\tt \% cd session1}
\item start {\sc Matlab}\,: \\
	{\tt \% matlab }
\end{enumerate}
Then go on with the experiments...


\vspace{1cm}

{\scriptsize
\noindent
This document was created by\,: Sacha Krstulovi\'c ({\tt [e-mail]}).

\noindent
This document is currently maintained by\,: Sacha Krstulovi\'c ({\tt [e-mail]}). Last modification on \today.

\noindent
This document is part of the package {\tt Session4.tgz} available by ftp as\,: {\tt ftp.idiap.ch/pub/sacha/labs/Session1.tgz} .
}

\clearpage

\tableofcontents

\bigskip
%\clearpage
%%%%%%%%%
%%%%%%%%%
\section{Gaussian statistics}
%%%%%%%%%
%%%%%%%%%

%%%%%%%%%
\subsection{Samples from a Gaussian density}
\label{samples}
%%%%%%%%%

\subsubsection*{Useful formulas and definitions\,:}
\begin{itemize}
\item[-] The {\em Gaussian probability density function} (Gaussian pdf) for
the $d$-dimensional random variable $x \circlearrowleft {\cal
N}(\mu,\Sigma)$ (i.e. variable $x$ following the Gaussian, or Normal,
probability law) is given by\,:
\[
g_{(\mu,\Sigma)}(x) = \frac{1}{\sqrt{2\pi}^d \sqrt{\det\left(\Sigma\right)}}
\, e^{-\frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu)}
\]
where $\mu$ is the mean vector and $\Sigma$ is the variance-covariance
matrix. $\mu$ and $\Sigma$ are the {\em parameters} of the Gaussian
distribution. {\em Speech features} (also referred as {\em acoustic
vectors}) are examples of $d$-dimensional variables, and it is usually
assumed that they follow a Gaussian distribution.
%
\item[-] If $x \circlearrowleft {\cal N}(0,I)$ ($x$ follows a normal law
with zero mean and unit variance; $I$ denotes the identity matrix), and if
$y = \sqrt{\Sigma} \, x + \mu$, then $y \circlearrowleft {\cal
N}(\mu,\Sigma)$.
%
\item[-] $\sqrt{\Sigma}$ defines the {\em standard deviation} of the random
variable $x$ ({\em \'ecart-type} in French). Beware\,: this square root is
meant in the {\em matrix sense}.
\end{itemize}

\subsubsection*{Experiment\,:}
Generate a sample $X$ of $N$ points, i.e. $X=\{x_1, x_2,\cdots,x_N\}$, with
$N=10000$, coming from a 2-dimensional Gaussian process that has mean\,:
\[
\mu = \left( \begin{array}{c} 730 \\ 1090 \end{array} \right)
\]
and variance\,:
	\begin{enumerate}
	%%%%%
	\item 8000 for both dimensions ({\em spherical process}) (sample $X_1$)\,:
		\[
		\Sigma_1 = \left[ \begin{array}{cc}
						8000 & 0 \\
						0    & 8000
						\end{array} \right]
		\]
	%%%%%
	\item expressed as a {\em diagonal} covariance matrix (sample $X_2$)\,:
		\[
		\Sigma_2 = \left[ \begin{array}{cc}
						8000 & 0 \\
						0    & 18500
						\end{array} \right]
		\]
	%%%%%
	\item expressed as a {\em full} covariance matrix (sample $X_3$)\,:
		\[
		\Sigma_3 = \left[ \begin{array}{cc}
						8000 & 8400 \\
						8400 & 18500
						\end{array} \right]
		\]
	%%%%%
	\end{enumerate}
%
Use the function \com{gausview} (\com{>> help gausview}) to plot the
results as clouds of points in the 2-dimensional plane, and to view the
corresponding 2-dimensional probability density functions (pdfs) in 2D and
3D.

\subsubsection*{Example\,:}
\mat{N = 10000;}
\mat{mu = [730 1090]; sigma\_1 = [8000 0; 0 8000];}
\mat{X1 = randn(N,2) * sqrtm(sigma\_1) + repmat(mu,N,1);}
\mat{gausview(X1,mu,sigma\_1,'Sample X1');}
%
Repeat the three previous steps for the two other Gaussians. Use the radio
buttons to switch the plots on/off. Use the ``view'' buttons to switch
between 2D and 3D. Use the mouse to rotate the plot.


{\bf \noindent Note\,:} if you don't know what one of the cited {\sc Matlab}
command does, use \\
\mat{help {\it command}}
to get some help. If the help doesn't make it clearer, ask the assistants.

\subsubsection*{Question\,:}
By simple inspection of 2D views of the data and of the corresponding pdf
contours, how can you tell which sample corresponds to a spherical process,
which sample corresponds to a pdf with a diagonal covariance, and which to
a pdf with a full covariance~?

\subsubsection*{Answer\,:}
\expl{The cloud of points and the pdf contours corresponding to $\Sigma_1$
are circular, because in this case the first and the second dimension of
the vectors are independent. As a matter of fact, they have a null
covariance\,:
\[ {\cal E}\left[(x_{d1}-\mu_{d1})^T (x_{d2}-\mu_{d2})\right] = 0 \]
They are {\em orthogonal} in the statistical sense, which transposes to a
geometric sense (the expectation is a scalar product of random variables; a
null scalar product means orthogonality). Intuitively, you can also
consider that a null covariance means no sharing of information between the
two dimensions\,: they can evolve independently in a Gaussian way along
their respective axes. Besides, the variance is the same in both
dimensions, which indicates an equivalent spread of the data along both
axes. Hence the circular blob of data points and the name ``spherical
process''.

\medskip

\tab The cloud of points and the pdf contours corresponding to $\Sigma_2$
are elliptic, with their axes parallel to the abscissa and ordinate
axes. This is because the first and the second dimension are still
independent (their covariance is null again), but this time the variance is
different along both dimensions.

\medskip

\tab For $\Sigma_3$, the covariance of both dimensions is not null, so the
principal axes of the ellipses are not aligned with the abscissa and
ordinate axes.}

\pagebreak
%%%%%%%%%
\subsection{Gaussian modeling\,: mean and variance of a sample}
%%%%%%%%%

\subsubsection*{Useful formulas and definitions\,:}
\begin{itemize}
\item[-] Mean estimator\,: $\hat{\mu} = \frac{1}{N} \sum_{i=1}^{N} x_i$
\item[-] Unbiased covariance estimator\,: $ \hat{\Sigma} = \frac{1}{N-1} \;
\sum_{i=1}^{N} (x_i-\mu)^T (x_i-\mu) $
\end{itemize}

\subsubsection*{Experiment\,:}
Take the set $X_3$ of 10000 points generated from ${\cal
N}(\mu,\Sigma_3)$. Compute an estimate $\hat{\mu}$ of its mean and an
estimate $\hat{\Sigma}$ of its variance\,:
\begin{enumerate}
\item with all the available points \hspace{1cm}$\hat{\mu}_{(10000)}
=$\hspace{3.5cm}$\hat{\Sigma}_{(10000)} =$
\vspace{0.8cm}
\item with only 1000 points \hspace{2cm}$\hat{\mu}_{(1000)}
=$\hspace{3.7cm}$\hat{\Sigma}_{(1000)} =$
\vspace{0.8cm}
\item with only 100 points \hspace{2.1cm}$\hat{\mu}_{(100)}
=$\hspace{3.9cm}$\hat{\Sigma}_{(100)} =$
\vspace{0.8cm}
\end{enumerate}
Compare the estimated value $\hat{\mu}$ with the original value of $\mu$ by
measuring the Euclidean distance that separates them. Compare the estimated
value $\hat{\Sigma}$ with the original value of $\Sigma_3$ by measuring the
matrix 2-norm of their difference ($\parallel A-B \parallel_2$ constitutes
a measure of similarity of two matrices $A$ and $B$; use {\sc Matlab}'s
\com{norm} command).

\subsubsection*{Example\,:}
In the case of 1000 points (case 2.)\,: \\
\mat{X = X3(1:1000,:);}
\mat{N = size(X,1)}
\com{>> mu\_1000 = sum(X)/N} \,\,{\it -or-}\,\, \mat{mu\_1000 = mean(X)}
\mat{sigma\_1000 = (X - repmat(mu\_1000,N,1))' * (X - repmat(mu\_1000,N,1)) / (N-1)}
\,\,{\it -or-}\,\, \mat{sigma\_1000 = cov(X)}

\noindent
\mat{\% Comparison of the values:}
\mat{e\_mu =  sqrt( (mu\_1000 - mu) * (mu\_1000 - mu)' )}
\mat{\% (This is the Euclidean distance between mu\_1000 and mu)}
\mat{e\_sigma = norm( sigma\_1000 - sigma\_3 )}
\com{>> \% (This is the 2-norm of the difference between sigma\_1000 and sigma\_3)}

\subsubsection*{Question\,:}
When comparing the estimated values $\hat{\mu}$ and $\hat{\Sigma}$ with the
original values of $\mu$ and $\Sigma_3$ (using the Euclidean distance and
the matrix 2-norm), what can you observe~?

\subsubsection*{Answer\,:}
\expl{The more points, the better the estimates. Furthermore, an accurate
mean estimate requires less points than an accurate variance estimate. In
general, in any data-based pattern classification technique (as opposed to
knowledge-based techniques or expert systems), it is very important to have
enough training examples to estimate some accurate models of the data.}

\pagebreak
%%%%%%%%%
\subsection{Likelihood of a sample with respect to a Gaussian model}
\label{like}
%%%%%%%%%
\subsubsection*{Useful formulas and definitions\,:}
\begin{itemize}
\item[-] {\em Likelihood}\,: the likelihood of a sample point given a
Gaussian model (i.e. given a set of parameters $\Theta = (\mu,\Sigma)$) is
the value of the probability density function for that point. In the case
of Gaussian models, this amounts to compute the value of the pdf expression
given at the beginning of section~\ref{samples}.
\item[-] {\em Joint likelihood}\,: for a set of independent identically
distributed (i.i.d.) points, say $X = \{ x_1, x_2, \cdots, x_N \}$, the
joint (or total) likelihood is the product of the likelihood for each
point. For instance, in the Gaussian case\,:
\[ p(X|\Theta) = 
\prod_{i=1}^{N} p(x_i|\Theta) =