eigenfisherface.tex

automatic face recognition

% Template for SLT-2006 paper; to be used with:
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage{spconf,amsmath,epsfig}
\usepackage{amsmath,amssymb,amsthm,amsfonts,graphicx,hyperref,subfigure,rotating}
%\renewcommand{\thefigure}{\arabic{figure}}

\usepackage{url}
%% Define a new 'leo' style for the package that will use a smaller font.
\makeatletter
\def\url@leostyle{%
  \@ifundefined{selectfont}{\def\UrlFont{\sf}}{\def\UrlFont{\small\ttfamily}}}
\makeatother
%% Now actually use the newly defined style.
\urlstyle{leo}

% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{Eigenfaces and Fisherfaces}
% 
% Single address.
% ---------------
\name{Naotoshi Seo}
\address{University of Maryland\\ENEE633 Pattern Recognition\\Project 2-1}
% 
% For example:
% ------------
% \address{School\\
%   Department\\
%   Address}
% 
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
% \twoauthors
% {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
%	{School A-B\\
%   Department A-B\\
%   Address A-B}
% {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
%     while at ...}}
%	{School C-D\\
%   Department C-D\\
%   Address C-D}
% 
\begin{document}
% \ninept
% 
\DeclareGraphicsExtensions{.pdf,.eps,.gif,.jpg,.eps, .png}
\maketitle
% 
\begin{abstract}
This project describes a study of two traditional face recognition
methods, the Eigenface \cite{Eigenface91} and the Fisherface \cite{Fisherface97}. 
The Eigenface is the first method considered as a successful technique of face recognition. 
The Eigenface method uses Principal Component Analysis (PCA) to linearly project the image space to 
a low dimensional feature space. 
The Fisherface method is an enhancement of the Eigenface method that it uses Fisher's Linear Discriminant Analysis (FLDA or LDA) for the dimensionality reduction. 
The LDA maximizes the ratio of between-class scatter to that of within-class scatter, therefore, 
it works better than PCA for purpose of discrimination. 
The Fisherface is especially useful when facial images have large variations in illumination and facial expression. 
In this project, a comparison of the Eigenface and the Fisherface methods respect to facial images having large illumination variations is examined. 

\end{abstract}
% 
\begin{keywords}
Eigenface, Fisherface, Face Recognition, PCA, LDA, Illumination Invariant

\end{keywords}
% 
\section{Introduction}
\label{sec:intro}
%
%婄夋憸偺擣幆偱傑偢峫偊傜傟傞偺偼丄栚丒旲丒岥側偳偺婄傪峔惉偡傞晹昳偺宍忬傗偦傟傜偺攝抲偺屄恖嵎偵拝栚偟丄偙傟傜偐傜摿挜揰傪拪弌偟偰擣幆偵棙梡偡傞曽朄偱偁傞丅偟偐偟丄婄夋憸偐傜偙傟傜偺晹昳傪惛搙傛偔拪弌偡傞偙偲偼偐側傝擄偟偄丅傑偨丄奺晹昳偑偆傑偔拪弌偱偒偨偲偟偰傕丄椶帡偟偨宍忬偺嵎傪擣幆偵棙梡偡傞偙偲偼偦傟傎偳梕堈偱偼側偄丅偦偙偱丄偙偺傛偆側婄偵摿桳偺抦幆傪梡偄傞偺偱偼側偔丄婄夋憸偦偺傕偺傪僷僞乕儞偲偟偰埖偄丄摑寁揑僷僞乕儞擣幆庤朄傪揔梡偡傞曽岦偺尋媶偑妶敪偵峴傢傟偰偄傞丅
%
%嵟傕娙扨側僷僞乕儞擣幆庤朄偼丄僷僞乕儞娫偺儅僢僠儞僌偵婎偯偔曽朄偱偁傞偑丄夋憸偦偺傕偺傪僷僞乕儞偲偟偰埖偭偨応崌偵偼丄僷僞乕儞偺師尦偑朿戝偵側偭偰偟傑偆丅偦偺偨傔丄僷僞乕儞傪忣曬埑弅偟偨屻偱儅僢僠儞僌傪峴偆曽朄偑偄偔偮偐採埬偝傟偰偄傞丅僷僞乕儞傪忣曬埑弅偡傟偽丄擖椡忦審偺曄摦偵懳偟偰婃寬側擣幆寢壥偑摼傜傟傞偲婜懸偱偒傞丅Turk摍 [77,78]偑採埬偟偨屌桳婄(eigenface)偵傛傞曽朄偱偼丄庡惉暘暘愅偵傛傝僷僞乕儞傪忣曬埑弅偟丄婄夋憸偺幆暿偵棙梡偟偰偄傞丅庤朄偺扨弮偝偲屌桳婄偲偄偆柤慜偺晅偗曽偺忋庤偝偐傜丄婄夋憸偺擣幆偵偍偄偰嵟傕桳柤側庤朄偺傂偲偮偲側偭偰偄傞丅 

Over the last couple of years, face recognition has become a popular area of research in computer vision
and one of the most successful applications of image analysis and understanding. 
The face recognition problem can generally be formulated as: 
Given still or video images of a scene, identify or verify one or more persons in the scene using store database of faces. 

The Eigenface \cite{sirovitch1987lowdimensional}, \cite{turk1991eigenfaces}, \cite{turk1991face} is the first method considered as a successful technique of face recognition. 
The Eigenface method uses Principal Component Analysis (PCA) to linearly project the image space to 
a low dimensional feature space. 

The Fisherface \cite{belhumeur1996eigenfaces} is an enhancement of the Eigenface method. 
The Eigenface method uses PCA for dimensionality reduction, thus, yields projection directions 
that maximize the total scatter across all classes, i.e., across all image s of all faces. 
The PCA projections are optimal for representation in a low dimensional basis, but
they may not be optional from a discrimination standpoint. 
In stead, the Fisherface method uses Fisher's Linear Discriminant Analysis (FLDA or LDA) which 
maximizes the ratio of between-class scatter to that of within-class scatter. 

In section \ref{S:Eigenface}, the Eigenface method is investigated. 
In section \ref{S:Fisherface}, the Fisherface method is examined.  
In section \ref{S:Comparison}, an empirical comparison of the Eigenface and the Fisherface methods subject to facial images having large illumination variations is performed. 
In section \ref{S:Conclusion}, conclusion of this study is provided.

\section{Eigenface}\label{S:Eigenface}

The {\it Eigenface} method is based on linearly projecting
the image space to a low dimensional feature space \cite{sirovitch1987lowdimensional}, \cite{turk1991eigenfaces}, \cite{turk1991face}. The Eigenface method, which uses principal
components analysis (PCA) for dimensionality reduction,
yields projection directions that maximize the total
scatter across all classes, i.e., across all images of all faces. 
% In choosing the projection which maximizes total scatter, PCA
% retains unwanted variations due to lighting and facial
% expression. As illustrated in Figs. 1 and 4 and stated by
% Moses et al., "the variations between the images of the same
% face due to illumination and viewing direction are almost
% always larger than image variations due to change in face
% identity" [9]. Thus, while the PCA projections are optimal
% for reconstruction from a low dimensional basis, they may
% not be optimal from a discrimination standpoint.

Let us consider a set of $N$ sample images $\{\mathbf{x}_{1},\ \mathbf{x}_{2},\ \ldots,\ \mathbf{x}_{N}\}$ taking values in an n-dimensional image space, and assume that each image belongs to one of $c$ classes $\{X_{1},X_{2},\ \ldots,X_{c}\}$. Let us also consider a linear transformation mapping the original n-dimensional image space into an m-dimensional feature space, where $m<n$. The new feature vectors $\mathbf{y}_{k}\in \mathbb{R}^{m}$ are defined by the following linear transformation:
\begin{equation}
\mathbf{y}_{k}=W^{T}\mathbf{x}_{k}~~~~~~~~~~~~~~~k=1,2,\ldots,N
\end{equation}
where $W\in \mathbb{R}^{n\times m}$ is a matrix with orthonormal columns.

  If the total scatter matrix $S_{T}$ is defined as
\begin{equation}
S_{T}=\sum_{k=1}^{N}(\mathbf{x}_{k}-\mathbf{\mu})(\mathbf{x}_{k}-\mathbf{\mu})^{T}
\end{equation}
where $\mathbf{\mu}\in \mathbb{R}^{n}$ is the mean image of all samples, then after applying the linear transformation $W^{T}$, the scatter of the transformed feature vectors $\{\mathbf{y}_{1},\ \mathbf{y}_{2},\ \ldots,\ \mathbf{y}_{N}\}$ is $W^{T}S_{T}W$. In PCA, the projection $W_{\text{opt}}$ is chosen to maximize the determinant of the total scatter matrix of the projected samples, i.e.,
\begin{equation}
W_{\text{opt}}=\arg \max_{W}|W^{T}S_{T}W|
\end{equation}
\begin{equation}
=[\mathbf{w}_{1}\mathbf{w}_{2}\text{ }...\text{ }\mathbf{w}_{m}]
\end{equation}
where $\{\mathbf{w}_i |i=1,2,\ \ldots,\ m\}$ is the set of $n$-dimensional eigenvectors of $S_{T}$ corresponding to the $m$ largest eigenvalues $\{\lambda_i |i=1,2,\ \ldots,\ m\}$ \cite{duda2001pattern}, i.e.,
\begin{equation}
S_T \mathbf{w}_i = \lambda_i \mathbf{w}_i,~~~~~~~i=1,2,\cdots,m.
\end{equation}

 Since these eigenvectors have the same dimension as the original images, they are referred to as Eigenpictures in \cite{sirovitch1987lowdimensional} and Eigenfaces in \cite{turk1991eigenfaces}, \cite{turk1991face}. Classification is performed using a nearest neighbor classifier in the reduced feature space.
%If classification is performed using a nearest neighbor classifier in the reduced feature space and $m$ is chosen to be the number of images $N$ in the training set, then the Eigenface method is equivalent to the correlation method in the previous section.

%   A drawback of this approach is that the scatter being maximized is due not only to the between-class scatter that is useful for classification, but also to the within-class scatter that, for classification purposes, is unwanted information. Recall the comment by Moses et al. \cite{moses1994face}: Much of the variation from one image to the next is due to illumination changes. Thus if PCA is presented with images of faces under varying illumination, the projection matrix $W_{\text{opt}}$ will contain principal components (i.e., Eigenfaces) which retain, in the projected feature space, the variation due lighting. Consequently, the points in the projected space will not be well clustered, and worse, the classes may be smeared together.

%   It has been suggested that by discarding the three most significant principal components, the variation due to lighting is reduced. The hope is that if the first principal components capture the variation due to lighting, then better clustering of projected samples is achieved by ignoring them. Yet, it is unlikely that the first several principal components correspond solely to variation in lighting; as a consequence, information that is useful for discrimination may be lost.

% For a data $\textbf{X}^T$ with zero empirical mean (the empirical mean of the distribution has been subtracted from the data set), where each row represents a different repetition of the experiment, and each column gives the results from a particular probe, the PCA transformation is given by:
% \begin{eqnarray}
% \mathbf{Y}&=&\mathbf{W}^T\mathbf{X}
% \end{eqnarray}
% where $\mathbf{V}\mathbf{\Sigma}\mathbf{W}^T$ is the singular value decomposition (svd) of $\mathbf{X}^T$.

\subsection{Further Observations}

To obtain deeper understandings of the Eigenface methods, 
a few extra experiments are conducted in this section.

The Fig. \ref{fig:dataset} shows a facial image set taken from ORL face database \cite{ORL}. The PCA analysis is applied to the image set, and the obtained meanface and eigenfaces are shown in the Fig. \ref{fig:eigenface}. 
% A projection of a feature vector into the PCA subspace can be considered as a computation of weights of linear transform of eigenfaces to reconstruct back into the full space. 

\begin{figure}[htbp]
\begin{tabular}{ccccc}
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=face01.eps,width=1.6cm}}
\end{minipage} &
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=face02.eps,width=1.6cm}}
\end{minipage} &
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=face03.eps,width=1.6cm}}
\end{minipage} &
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=face04.eps,width=1.6cm}}
\end{minipage} &
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=face05.eps,width=1.6cm}}
\end{minipage}
\end{tabular}
\caption{The input facial image set}
\label{fig:dataset}
\end{figure}


\begin{figure}[htbp]
\begin{tabular}{ccccc}
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=meanface.eps,width=1.6cm}}
  \centerline{(a)}\medskip
\end{minipage} &
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=eigenface01.eps,width=1.6cm}}
  \centerline{(b)}\medskip
\end{minipage} &
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=eigenface02.eps,width=1.6cm}}
  \centerline{(c)}\medskip
\end{minipage} &
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=eigenface03.eps,width=1.6cm}}
  \centerline{(d)}\medskip
\end{minipage} &
\begin{minipage}[b]{0.15\linewidth}
  \centering
  \centerline{\epsfig{figure=eigenface04.eps,width=1.6cm}}
  \centerline{(e)}\medskip
\end{minipage}
\end{tabular}
\caption{(a) The meanface (b-e) The eigenfaces. The left-most eigenface is the most principal one.}
\label{fig:eigenface}
\end{figure}

In Fig. \ref{fig:newset}, the top row shows another image set and the bottom row shows the reconstructed images which are created by projecting images into the PCA subspace once and reconstructing them back into the full original space. 
The pictures illustrate the reason why the Eigenface method works well. The reconstructed images each other are more similar than the original images each other, thus, it possibly achieves better recognition by comparing the reconstructed images than comparing the original images like a typical template matching method. 

Notice that we in fact do not need to compute the reconstructed images, but uses the projected points in the PCA subspace for the face recognition. 
A simple proof to show their equivalence is as follows: Without loss of generality, let $\mathbf{x} \in \mathbb{R}^n $ be a query face image vector to be classified and $\{\mathbf{x}_i \in \mathbb{R}^n, i = 1,\ldots,c\}$ \footnote{One representative from one class is the minimum requirement for the nearest neighbor classification.} be representatives from each face class $i$ with zero empirical mean where $n$ is the dimension of the vectors. Furthermore, let $\mathbf{y} \in \mathbb{R}^m$ and $\{\mathbf{y}_i \in \mathbb{R}^m, i=1,\ldots,c\}$ be the projected points into the PCA subspace of them, that is,
\begin{eqnarray}
\mathbf{y} &=& \mathbf{W}^T \mathbf{x},\\
\mathbf{y}_i &=& \mathbf{W}^T \mathbf{x}_i,~~~i=1,\ldots,c.
\end{eqnarray}
where $\mathbf{W} =[\mathbf{w}_{1}\mathbf{w}_{2}\text{ }...\text{ }\mathbf{w}_{m}]$ is a $n \times m$ matrix of a set of $n$ dimensional orthonormal eigenvectors and $m \ll n$. Furthermore, let $\hat{\mathbf{x}} \in \mathbb{R}^n$ and $\{\hat{\mathbf{x}}_i \in \mathbb{R}^n, i=1,\ldots,c\}$ be the reconstructed images of them, that is,
\begin{eqnarray}
\hat{\mathbf{x}} &=& \mathbf{W} \mathbf{y},\\
\hat{\mathbf{x}}_i &=& \mathbf{W} \mathbf{y}_i,~~~i=1,\ldots,c.
\end{eqnarray}
Notice that
\begin{eqnarray}
\mathbf{y} &=& \mathbf{W}^T \hat{\mathbf{x}},\\
\mathbf{y}_i &=& \mathbf{W}^T \hat{\mathbf{x}}_i,~~~~i=1,\ldots,c
\end{eqnarray}
are also satisfied and the orthonormal eigenvectors $\mathbf{W}$ is the orthonomal ``basis'' for the reconstructed image space. 
The recognition task using the reconstructed images is to identify $i^*$ by finding a representative $\hat{\mathbf{x}}_i$ with minimal distance to the query image $\hat{\mathbf{x}}$ in the reconstructed space, i.e., 
\begin{eqnarray}
i^* &=& \arg \min_i ||\hat{\mathbf{x}} - \hat{\mathbf{x}}_i||_2 \label{eq:reconstructed}