hlmtutorial.tex

该压缩包为最新版htk的源代码,htk是现在比较流行的语音处理软件,请有兴趣的朋友下载使用

class.recovery} and {\tt class.recovery.cm} -- these are a recovery
status file and its associated class map which are exported at regular
intervals because the \htool{Cluster} tool can take so long to run.
In this way you can kill the tool before it has finished and resume
execution at a later date by using the {\tt -x} option; in this case
you would use {\tt -x holmes.2/class.recovery} for example (making
sure you pass the same word map and gram files -- the tool does
{\it not} currently check that you pass it the same files when restarting).

Once the tool finishes running you should see the file {\tt
holmes.2/class.1.cm} which is the resulting class map.  It is in plain
text format so feel free to examine it.  Note, for example, how {\tt
CLASS23} consists almost totally of verb forms ending in {\tt -ED},
whilst {\tt CLASS41} lists various general words for a person or
object.  Had you created more classes then you would be likely to see
more distinctive classes.  We can now use this file to build the class
$n$-gram component of our language model.
\begin{verbatim}
$ LGCopy -T 1 -d holmes.2 -m holmes.2/cmap -w holmes.2/class.1.cm
         lm_5k/5k.wmap holmes.1/data.* lm_5k/data.0
Input file holmes.1/data.0 added, weight=1.0000
Input file lm_5k/data.0 added, weight=1.0000
Copying 2 input files to output files with 2000000 entries
Class map = holmes.2/class.1.cm
 saving 162397 ngrams to file holmes.2/data.0 
330433 out of 330433 ngrams stored in 1 files
\end{verbatim} % $

The {\tt -w} option specifies an input class map which is applied when
copying the gram files, so we now have a class gram file in {\tt
holmes.2/data.0}.  It has an associated word map file {\tt
holmes.2/cmap} -- although this only contains class names it is
technically a word map since it is taken as input wherever a word map
is required by the \HTK\ language modelling tools; recall that word
maps can contain classes as witnessed by {\tt !!UNK} previously.

You can examine the class $n$-grams in a similar way to previously by
using \htool{LGList}
\begin{verbatim}
$ LGList holmes.2/cmap holmes.2/data.0 | more 
 
3-Gram File holmes.2/data.0[162397 entries]:
 Text Source: LGCopy
CLASS1      CLASS10     CLASS103     : 1
CLASS1      CLASS10     CLASS11      : 2
CLASS1      CLASS10     CLASS118     : 1
CLASS1      CLASS10     CLASS12      : 1
CLASS1      CLASS10     CLASS126     : 2
CLASS1      CLASS10     CLASS140     : 2
CLASS1      CLASS10     CLASS147     : 1
...
\end{verbatim} % $

And similarly the class $n$-gram component of the overall language
model is built using \htool{LBuild} as previously with
\begin{verbatim}
$ LBuild -T 1 -c 2 1 -c 3 1 -n 3 holmes.2/cmap
     lm_5k/cl150-tg_1_1.cc holmes.2/data.*
Input file holmes.2/data.0 added, weight=1.0000
\end{verbatim} % $

To build the word-given-class component of the model we must run
\htool{Cluster} again.
\begin{verbatim}
$ Cluster -l holmes.2/class.1.cm -i 0 -q lm_5k/cl150-counts.wc
     lm_5k/5k.wmap holmes.1/data.* lm_5k/data.0
\end{verbatim} % $

This is very similar to how we ran \htool{Cluster} earlier, except
that we now want to perform 0 iterations ({\tt -i 0}) and we start by
loading in the existing class map with {\tt -l holmes.2/class.1.cm}.
We don't need to pass {\tt -k} because we aren't doing any further
clustering and we don't need to specify the number of classes since
this is read from the class map along with the class contents.  The
{\tt -q lm\_5k/cl150-counts.wc} option tells the tool to write 
word-given-class counts to the specified file.  Alternatively we could
have specified {\tt -p} instead of {\tt -q} and written probabilities
as opposed to counts.  The file is in a plain text format, and either
the {\tt -p} or {\tt -q} version is sufficient for forming the
word-given-class component of a class language model.  Note that in
fact we could have simply added either {\tt -p} or {\tt -q} the
first time we ran \htool{Cluster} and generated both the class map and
language model component file in one go.

Given the two language model components we can now link them together
to make our overall class $n$-gram language model.
\begin{verbatim}
$ LLink lm_5k/cl150-counts.wc lm_5k/cl150-tg_1_1.cc
     lm_5k/cl150-tg_1_1
\end{verbatim} % $

The \htool{LLink} tool creates a simple text file pointing to the two
necessary components, auto-detecting whether a count or probabilities
file has been supplied.  The resulting file, {\tt lm\_5k/cl150-tg\_1\_1}
is the finished overall class $n$-gram model, which we can now assess
the performance of with \htool{LPlex}.
\begin{verbatim}
$ LPlex -n 3 -t lm_5k/cl150-tg_1_1 test/red-headed_league.txt
LPlex test #0: 3-gram
perplexity 125.9065, var 7.4139, utterances 556, words predicted 8127
num tokens 10408, OOV 665, OOV rate 6.75% (excl. </s>)

Access statistics for lm_5k/cl150-tg_1_1:
Lang model  requested  exact backed    n/a     mean    stdev
    bigram       2867  95.4%   4.6%   0.0%    -4.61     1.64
   trigram       8127  64.7%  24.1%  11.2%    -4.84     2.72
\end{verbatim} % $

The class trigram model performs worse than the word trigram (which
had a perplexity of 117.4), but this is not a surprise since this is
true of almost every reasonably-sized test set -- the class model is
less specific.  Interpolating the two often leads to further
improvements, however.  We can find out if this will happen in this
case by interpolating the models with \htool{LPlex}.
\begin{verbatim}
$ LPlex -u -n 3 -t -i 0.4 lm_5k/cl150-tg_1_1 lm_5k/tg1_1
      test/red-headed_league.txt
LPlex test #0: 3-gram
perplexity 102.6389, var 7.3924, utterances 556, words predicted 9187
num tokens 10408, OOV 665, OOV rate 6.75% (excl. </s>)
 
Access statistics for lm_5k/tg2-1_1:
Lang model  requested  exact backed    n/a     mean    stdev
    bigram       5911  68.5%  30.9%   0.6%    -5.75     2.94
   trigram       9187  35.7%  31.2%  33.2%    -4.77     2.98
 
Access statistics for lm_5k/cl150-tg_1_1:
Lang model  requested  exact backed    n/a     mean    stdev
    bigram       3104  95.5%   4.5%   0.0%    -4.67     1.62
   trigram       9187  66.2%  23.9%   9.9%    -4.87     2.75
\end{verbatim} % $
So a further gain is obtained -- the interpolated model performs
significantly better.  Further improvement might be possible by
attempting to optimise the interpolation weight.

Note that we could also have used \htool{LLink} to build a single
class language model file instead of producing a third file which
points to the two components.  We can do this by using the {\tt -s}
single file option.
\begin{verbatim}
$ LLink -s lm_5k/cl150-counts.wc lm_5k/cl150-tg_1_1.cc
     lm_5k/cl150-tg_1_1.all
\end{verbatim} % $
The file {\tt lm\_5k/cl150-tg\_1\_1.all} is now a standalone language
model, identical in performance to {\tt lm\_5k/cl150-tg\_1\_1} created
earlier.


\mysect{Problem solving}{HLMproblemSolving}
\index{Problem solving}
Sometimes a tool returns an error message which doesn't seem to make
sense when you check the files you've passed and the switches
you've given.  This section provides a few problem-solving hints.

\mysubsect{File format problems}{HLMfileproblems}
If a file which seems to be in the correct format is giving errors
such as `Bad header' then make sure that you are using the correct
input filter.  If the file is gzipped then ensure you are using a
suitable configuration parameter to decompress it on input; similarly
if it isn't compressed then check you're not trying to decompress it.
Also check to see if you have two files, one with and one without a
{\tt .gz} extension -- maybe you're picking up the wrong one and
checking the other file.

You might be missing a switch or configuration file to tell the tool
which format the file is in.  In general none of the \HTK\ language
modelling tools can auto-detect file formats -- unless you tell them
otherwise they will expect the file type they are configured to
default to and will give an error relevant to that type if it does not
match.  For example, if you omit to pass {\tt -t} to \htool{LPlex}
then it will treat an input text file as a
\HTK\ label file and you will get a `Too many columns' error if a line
has more than 100 words on it or a ridiculously high perplexity
otherwise.  Check the command documentation in chapter
\ref{c:toolref}.

\mysubsect{Command syntax problems}{HLMsyntaxproblems}
If a tool is giving unexpected syntax errors then check that you have
placed all the option switches {\it before} the compulsory parameters
-- the tools will not work if this rule is not followed.  You must
also place whitespace between switches and any options they expect.
The ordering of switches is not important, but the order of compulsory
parameters cannot be changed.  Check the switch syntax -- passing a
redundant parameter to one will cause problems since it will be
interpreted as the first compulsory parameter.

All \HTK\ tools assume that a parameter which starts with a digit is a
number of some kind -- you cannot pass filenames which start with a
digit, therefore.  This is a limitation of the routines in
\htool{HShell}. 

\mysubsect{Word maps}{HLMwordmapproblems}
If your word map and gram file combination is being rejected then make
sure they match in terms of their sequence number.  Although gram
files are mainly stored in a binary format the header is in plain text
and so if you look at the top of the file you can compare it
manually with the word map.  Note it is not a good idea to fiddle the
values to match since they are bound to be different for a good
reason!  Word maps must have the same or a higher sequence id than a
gram file in order to open that gram file -- the names must match too.

The tools might not behave as you expect.  For example, \htool{LGPrep}
will write its word map to the file {\tt wmap} unless you tell it
otherwise, irrespective of the input filename.  It will also place it
in the same directory as the gram files unless you changed its name
from {\tt wmap}(!) -- check you are picking up the correct word map
when building subsequent gram files.

The word ids start at 65536 in order to allow space for that many
classes below them -- anything lower is assumed to be a class.  In
turn the number of classes is limited to 65535.

\mysubsect{Memory problems}{HLMmemoryproblems}
Should you encounter memory problems then try altering the amount of
space reserved by the tools using the relevant tool switches such as
{\tt -a} and {\tt -b} for {\tt LGPrep} and {\tt LGCopy}.  You could
also try turning on memory tracing to see how much memory is used and
for what (use the configuration {\tt TRACE} parameters and the {\tt
-T} option as appropriate.  Language models can become very large,
however -- hundreds of megabytes in size, for example -- so it is
important to apply cut-offs and/or discounting as appropriate to keep
them to a suitable size for your system.

\mysubsect{Unexpected perplexities}{HLMperpproblems}
If perplexities are not what you expected, then there are many things
that could have gone wrong -- you may not have constructed a suitable
model -- but also some mistakes you might have made.  Check that you
passed all the switches you intended, and check that you have been
consistent in your use of {\tt *RAW*} configuration parameters --
using escaped characters in the language model without them in your
test text will lead to unexpected results.  If you have not escaped
words in your word map then check they're not escaped in any class
map.  When using a class model make sure you're passing the correct
input file of the three separate components.

Check the switches to {\tt LPlex} -- did you set {\tt -u} as you
intended?  If you passed a text file did you pass {\tt -t}?  Not doing
so will lead either to a format error or to extremely bizarre
perplexities!

Did you build the length of $n$-gram you meant to?  Check the final
language model by looking at the header of it, which is always stored
in plain text format.  You can easily see how many $n$-grams there are
for each size of $n$.