hlmtutorial.tex

该压缩包为最新版htk的源代码,htk是现在比较流行的语音处理软件,请有兴趣的朋友下载使用

Class map = 5k.wlist [Class mappings only]
 saving 75400 ngrams to file lm_5k/data.0
92918 out of 489516 ngrams stored in 1 files
\end{verbatim} % $

Because the {\tt -o} option was passed, all $n$-grams containing OOVs
will be extracted from the input files and the OOV words mapped to the
unknown symbol with the results stored in the files
\texttt{lm\_5k/data.*}.  A new word map containing the new class
symbols (\texttt{!!UNK} in this case) and only words in the vocabulary
will be saved to \texttt{lm\_5k/5k.wmap}.  Note how the newly produced
OOV $n$-gram files can no longer be decoded by the original word map
\texttt{holmes.0/wmap}:
\begin{verbatim}
$ LGList holmes.0/wmap lm_5k/data.0 |
  ERROR [+15330]  OpenNGramFile: Gram file map Holmes%%5k.wlist 
        inconsistent with Holmes
 FATAL ERROR - Terminating program LGList
\end{verbatim} % $
The error is due to the mismatch between the original map's name
(``Holmes'') and the name of the map stored in the header of the
$n$-gram file we attempted to list (``Holmes\%\%5k.wlist''). The latter
name indicates that the word map was derived from the original map
\texttt{Holmes} by resolving class membership using the class map
\texttt{5k.wlist}.  As a further consistency indicator, the original
map has a sequence count of 1 whilst the class-resolved map has a
sequence count of 2.

The correct command for listing the contents of the OOV $n$-gram
file is:
\begin{verbatim}
$ LGList lm_5k/5k.wmap lm_5k/data.0 | more

4-Gram File lm_5k/data.0[75400 entries]:
 Text Source: LGCopy
!!UNK       !!UNK       !!UNK       !!UNK        : 50
!!UNK       !!UNK       !!UNK       </s>         : 20
!!UNK       !!UNK       !!UNK       A            : 2
!!UNK       !!UNK       !!UNK       ACCOUNTS     : 1
!!UNK       !!UNK       !!UNK       ACROSS       : 1
!!UNK       !!UNK       !!UNK       AND          : 17
...
\end{verbatim} % $

At the same time the class resolved map \texttt{lm\_5k/5k.wmap} can
be used to list the contents of the $n$-gram, database files -- the
newer map can view the older grams, but not vice-versa.
\begin{verbatim}
$ LGList lm_5k/5k.wmap holmes.1/data.2 | more

4-Gram File holmes.1/data.2[89516 entries]:
 Text Source: LGCopy
THE         SUSSEX      MANOR       HOUSE        : 1
THE         SWARTHY     GIANT       GLARED       : 1
THE         SWEEP       OF          HIS          : 1
THE         SWEET       FACE        OF           : 1
THE         SWEET       PROMISE     OF           : 1
THE         SWINGING    DOOR        OF           : 1
...
\end{verbatim} % $
However, any $n$-grams containing OOV words will be discarded since
these are no longer in the word map.

Note that the required word map \texttt{lm\_5k/5k.wmap} can also be
produced also using the \htool{LSubset} tool:
\begin{verbatim}
$ LSubset -T 1 holmes.0/wmap 5k.wlist lm_5k/5k.wmap
\end{verbatim} % $

Note also that had the {\tt -o} option not been passed to
\htool{LGCopy} then the $n$-gram files built in {\tt lm\_5k} would have
contained not only those with OOV entries but also all the remaining
purely in-vocabulary words, the union of those shown by the two
preceding {\tt LGList} commands, in fact.  The method that you choose
to use depends on what experiments you are performing -- the \HTK\
tools allow you a degree of flexibility.


\mysect{Language model generation}{HLMlanmodgen}
Language models are built using the \htool{LBuild} command.  If you're
constructing a class-based model you'll also need the \htool{Cluster}
tool, but for now we'll construct a standard word $n$-gram model.

You'll probably want to accept the default of using Turing-Good
discounting for your $n$-gram model, so the first step in generating a
language model is to produce a frequency of frequency (FoF) table for
the chosen vocabulary list.  This is performed automatically by
\htool{LBuild}, but optionally you can generate this yourself using
the \htool{LFoF} tool and pass the result into \htool{LBuild}.  This
has only a negligable effect on computation time, but the result is
interesting in itself because it provides useful information for
setting cut-offs.  Cut-offs are where you choose to discard low
frequency events from the training text -- you might wish to do this
to decrease model size, or because you judge these infrequent events
to be unimportant.

In this example, you can generate a suitable table from the language
model databases and the newly generated OOV $n$-gram files:
\begin{verbatim}
$ LFoF -T 1 -n 4 -f 32 lm_5k/5k.wmap lm_5k/5k.fof
     holmes.1/data.* lm_5k/data.*
Input file holmes.1/data.0 added, weight=1.0000
Input file holmes.1/data.1 added, weight=1.0000
Input file holmes.1/data.2 added, weight=1.0000
Input file lm_5k/data.0 added, weight=1.0000
Calculating FoF table
\end{verbatim} % $

After executing the command, the FoF table will be stored in
\texttt{lm\_5k/5k.fof}.  It shows the number of times a word is found
with a given frequency -- if you recall the definition of Turing-Good
discounting you will see that this needs to be known.  See chapter
\ref{c:hlmfiles} for further details of the FoF file format.

You can also pass a configuration parameter to \htool{LFoF} to make it
output a related table showing the number of $n$-grams that will be
left if different cut-off rates are applied.  Rerun \htool{LFoF} and
also pass it the existing configuration file {\tt config}:
\begin{verbatim}
$ LFoF -C config -T 1 -n 4 -f 32 lm_5k/5k.wmap lm_5k/5k.fof
     holmes.1/data.* lm_5k/data.*
Input file holmes.1/data.0 added, weight=1.0000
Input file holmes.1/data.1 added, weight=1.0000
Input file holmes.1/data.2 added, weight=1.0000
Input file lm_5k/data.0 added, weight=1.0000
Calculating FoF table

cutoff  1-g     2-g     3-g     4-g
0       5001    128252  330433  471998 
1       5001    49014   60314   40602 
2       5001    30082   28646   15492 
3       5001    21614   17945   8801 
...
\end{verbatim} % $
The information can be interpreted as follows.  A bigram cut-off value
of 1 will leave 49014 bigrams in the model, whilst a trigram cut-off
of 3 will result in 17945 trigrams in the model.  The configuration
file \texttt{config} forces the tool to print out this extra
information by setting \texttt{LPCALC: TRACE=3}.  This is the trace
level for one of the library modules, and is separate from the trace
level for the tool itself (in this case we are passing {\tt -T 1} to
set trace level 1.  The trace field consists of a series of bits, so
setting trace 3 actually turns on two of those trace flags.

We'll now proceed to build our actual language model.  In this the
model will be generated in stages by executing the \htool{LBuild}
separately for each of the unigram, bigram and trigram sections of the
model (we won't build a 4-gram model in this example, although the
$n$-gram files we've build allow us to do so at a later date if we so
wish), but you can build the final trigram in one go if you like.  The
following command will generate the unigram model:
\begin{verbatim}
$ LBuild -T 1 -n 1 lm_5k/5k.wmap lm_5k/ug 
         holmes.1/data.* lm_5k/data.*
\end{verbatim} % $
Look in the {\tt lm\_5k} directory and you'll discover the model {\tt
ug} which can now be used on its own as a complete ARPA format
unigram language model.

We'll now build a bigram model with a cut-off of 1 and to save
regenerating the unigram component we'll include our existing unigram model:
\begin{verbatim}
$ LBuild -C config -T 1 -t lm_5k/5k.fof -c 2 1 -n 2
         -l lm_5k/ug lm_5k/5k.wmap lm_5k/bg1 
         holmes.1/data.* lm_5k/data.*
\end{verbatim} % $
Passing the {\tt config} file again means that we get given some
discount coefficient information.  Try rerunning the tool without the
{\tt -C config} to see the difference.  We've also passed in the
existing {\tt lm\_5k/5k.fof} file although this is not necessary --
try omitting this and you'll find that the resulting file is
identical.  What will be different, however, is that the tool will
print out the cut-off table seen when running \htool{LFoF} with the
{\tt LPCALC: TRACE = 3} parameter set; if you don't want to see this
then don't set {\tt LPCALC: TRACE = 3} in the configuration file (try
running the above command without {\tt -t} and {\tt -C}).

Note that this bigram model is created in \HTK\'s own binary version
of the ARPA format language model, with just the unigram component in
text format by default.  This makes the model more compact and faster
to load.  If you want to override this then simply add the {\tt -f
TEXT} parameter to the command.

Finally, the trigram model can be generated using the command:
\begin{verbatim}
$ LBuild -T 1 -c 3 1 -n 3 -l lm_5k/bg1
         lm_5k/5k.wmap lm_5k/tg1_1  
         holmes.1/data.* lm_5k/data.*
\end{verbatim} % $

Alternatively instead of the three stages above, you can also build
the final trigram in one step:
\begin{verbatim}
$ LBuild -T 1 -c 2 1 -c 3 1 -n 3 lm_5k/5k.wmap
         lm_5k/tg2-1_1 holmes.1/data.* lm_5k/data.*
\end{verbatim} % $
If you compare the two trigram models you'll see that they're the same
size -- there will probably be a few insignificant changes in
probability due to more cumulative rounding errors incorporated in the
three stage procedure.


\mysect{Testing the LM perplexity}{HLMtestingpp}
\index{Perplexity}
Once the language models have been generated, their ``goodness'' can
be evaluated by computing the perplexity of previously unseen text
data.  This won't necessarily tell you how well the language model
will perform in a speech recognition task because it takes no account
of acoustic similarities or the vagaries of any particular system, but
it will reveal how well a given piece of test text is modelled by your
language model.  The directory \texttt{test} contains a single story
which was withheld from the training text for testing purposes -- if
it had been included in the training text then it wouldn't be fair to
test the perplexity on it since the model would have already `seen' it.

Perplexity evaluation is carried out using \htool{LPlex}. The tool
accepts input text in one of two forms -- either as an HTK style MLF
(this is the default mode) or as a simple text stream. The text stream
mode, specified with the \texttt{-t} option, will be used to evaluate
the test material in this example.

\begin{verbatim}
$ LPlex -n 2 -n 3 -t lm_5k/tg1_1 test/red-headed_league.txt 
LPlex test #0: 2-gram
perplexity 131.8723, var 7.8744, utterances 556, words predicted 8588
num tokens 10408, OOV 665, OOV rate 6.75% (excl. </s>)

Access statistics for lm_5k/tg1_1:
Lang model  requested  exact backed    n/a     mean    stdev
    bigram       8588  78.9%  20.6%   0.4%    -4.88     2.81
   trigram          0   0.0%   0.0%   0.0%     0.00     0.00
LPlex test #1: 3-gram
perplexity 113.2480, var 8.9254, utterances 556, words predicted 8127
num tokens 10408, OOV 665, OOV rate 6.75% (excl. </s>)

Access statistics for lm_5k/tg1_1:
Lang model  requested  exact backed    n/a     mean    stdev
    bigram       5357  68.2%  31.1%   0.6%    -5.66     2.93
   trigram       8127  34.1%  30.2%  35.7%    -4.73     2.99
\end{verbatim} % $
The multiple \texttt{-n} options instruct \htool{LPlex} to perform two
separate tests on the data. The first test (\texttt{-n 2}) will use
only the bigram part of the model (and unigram when backing off),
whilst the second test (\texttt{-n 3}) will use the full trigram
model. For each test, the first part of the result gives general