openviron.tex

该压缩包为最新版htk的源代码,htk是现在比较流行的语音处理软件,请有兴趣的朋友下载使用

Although these string conventions are unnecessary in \HLM, to maintain
compatibility with \HTK the same conventions are used. However, a
number of options are provided to allow a mix of escaped and unescaped
text files to be handled.  Word maps allow the type of escaping (HTK
or none) to be defined in their headers.  When a degenerate form of
word map is used (i.e. a map with no header), the \htool{LWMap}
configuration variable \texttt{INWMAPRAW} may be set to true to
disable \HTK\ escaping. By default, \HLM\ tools output word lists and
maps in HTK escaped form.  However, this can be overridden by setting
the configuration variable \texttt{OUTWMAPRAW} to true.  Similar
conventions apply to class maps.  A degenerate class map can be read
in raw mode by setting the \htool{LClass} configuration variable
\texttt{INCMAPRAW} to true, and a class map can be written in raw form
by setting \texttt{OUTCMAPRAW} to  true.

Input/output of N-gram language model files are handled by the \HLM\
module \texttt{LModel}. Hence, by default input/output of LMs stored
in the ARPA-MIT text format will assume \HTK\ escaping conventions.
This can be disabled for both input and output by setting 
\texttt{RAWMITFORMAT} to true.

% $

\mysect{Memory Management}{memman}

Memory management\index{memory management} is a very low level function and is
mostly invisible to \HTK\ users.  However, some applications require very large
amounts of memory.  For example, building the models for a large vocabulary
continuous speech dictation system might require 150MB or more.  Clearly, when
memory demands become this large, a proper understanding of the impact of
system design decisions on memory usage is important.  The first step in this
is to have a basic understanding of memory allocation in \HTK.

Many \HTK\ tools dynamically construct large and complex data structures in
memory.  To keep strict control over this and to reduce memory allocation
overheads to an absolute minimum, \HTK\ performs its own memory
management. Thus, every time that a module or tool wishes to allocate some
memory, it does so by calling routines in
\htool{HMem}\index{hmem@\htool{HMem}}. At a slightly higher level, math objects
such as vectors and matrices are allocated by \htool{HMath} but using the
primitives provided by \htool{HMem}.

To make memory allocation\index{memory!allocators} and de-allocation very fast,
tools create specific memory allocators for specific objects or groups of
objects.  These memory allocators are divided into a sequence of blocks, and
they are organised as either Stacks\index{stacks}, M-heaps\index{M-heaps} or
C-heaps\index{C-heaps}.  A Stack constrains the pattern of allocation and
de-allocation requests to be made in a last-allocated first-deallocated order
but allows objects of any size to be allocated. An M-heap allows an arbitrary
pattern of allocation and de-allocation requests to be made but all allocated
objects must be the same size.  Both of these memory allocation disciplines are
more restricted than the general mechanism supplied by the operating system,
and as a result, such memory operations are faster and incur no storage
overhead due to the need to maintain hidden housekeeping information in each
allocated object.  Finally, a C-heap uses the underlying operating system and
allows arbitrary allocation patterns, and as a result incurs the associated
time and space overheads.  The use of C-heaps is avoided wherever possible.

Most tools provide one or more trace options which show how
much memory has been allocated.  The following shows the form of
the output\index{memory!statistics}
\begin{verbatim}
  ---------------------- Heap Statistics ------------------------
  nblk=1, siz= 100000*1, used= 32056, alloc= 100000 : Global Stack[S]
  nblk=1, siz=   200*28, used=   100, alloc=   5600 : cellHeap[M]
  nblk=1, siz=  10000*1, used=  3450, alloc=  10000 : mlfHeap[S]
  nblk=2, siz=   7504*1, used=  9216, alloc=  10346 : nameHeap[S]
  ---------------------------------------------------------------
\end{verbatim}
Each line describes the status of each memory allocator and gives the number of
blocks allocated, the current block size (number of elements in block $\times$
the number of bytes in each element)\footnote{ Block sizes typically grow as
more blocks are allocated}, the total number of bytes in use by the tool and
the total number of bytes currently allocated to that allocator.  The end of
each line gives the name of the allocator and its type: Stack[S], M-heap[M] or
C-heap[M].  The element size for Stacks will always be 1 but will be variable
in M-heaps.\index{memory!element sizes} The documentation for the memory
intensive \HTK\ tools indicates what each of the main memory allocators are
used for and this information allows the effects of various system design
choices to be monitored.

\mysect{Input/Output via Pipes and Networks}{iopipes}

Most types of file in \HTK\ can be input or output via a pipe\index{pipes}
instead of directly from or to disk.  The mechanism for doing this is to assign
the required input or output filter\index{output filter} command to a
configuration parameter or to an environment variable, either can be used.
Within this command, any occurrence of the dollar symbol
\verb+$+ will be replaced by the name of the required file. The
output of the command will then be input to or output from the \HTK\ tool via a
pipe.\index{filters}

For example, the following command will
normally list the contents of the speech waveform file \texttt{spfile}
\begin{verbatim}
    HList spfile
\end{verbatim}
However, if the value of the environment variable \texttt{HWAVEFILTER}
is set as follows
\begin{verbatim}
    setenv HWAVEFILTER 'gunzip -c $'
\end{verbatim}
then the effect is to invoke the decompression filter\index{decompression
filter} \texttt{gunzip} with its input connected to the file \texttt{spfile}
and its output connected to \htool{HList} via a pipe.  Each different type of
file has a unique associated variable so that multiple input and/or filters can
be used.  The full list of these is given in the summary section at the end of
this chapter.

\HTK\ is often used to process large amounts of data and typically this
data is distributed across a network.  In many systems, an attempt to open a
file can fail because of temporary network \textit{glitches}.  In the majority
of cases, a second or third attempt to open the file a few seconds later will
succeed and all will be well.  To allow this to be done automatically, 
\HTK\ tools can be configured to retry opening a file several times before giving up.
This is done simply by setting the configuration parameter
\texttt{MAXTRYOPEN}\index{maxtryopen@\texttt{MAXTRYOPEN}} to the required
number of retries\footnote{ This does not work if input filters are used.  }.
\index{files!network problems}\index{files!opening}

\mysect{Byte-swapping of HTK data files}{byteswap}

\index{natreadorder@\texttt{NATURALREADORDER}}
\index{natwriteorder@\texttt{NATURALWRITEORDER}}
\index{byte swapping}
Virtually all \HTK\ tools can read and write data to and from binary files. The
use of binary format as opposed to text can speed up the performance of the
tools and at the same time reduce the file size when manipulating large
quantities of data.  Typical binary files used by the \HTK\ tools are speech
waveform/parameter files, binary master model files (MMF), binary accumulator
files used in HMM parameter estimation and binary lattice files. However, the
use of binary data format often introduces incompatibilities between different
machine architectures due to the different byte ordering conventions used to
represent numerical quantities. In such cases, byte swapping of the data is
required. To avoid incompatibilities across different machine architectures,
all \HTK\ binary data files are written out using big-endian (\texttt{NONVAX})
representation of numerical values. Similarly, during loading \HTK\ binary
format files are assumed to be in \texttt{NONVAX} byte order. The default
behavior can be altered using the configuration parameters
\texttt{NATURALREADORDER} and
\texttt{NATURALWRITEORDER}. Setting \texttt{NATURALREADORDER} to true will
instruct the \HTK\ tools to interpret the binary input data in the machine's
natural byte order (byte swapping will never take place). Similarly, setting
\texttt{NATURALWRITEORDER} to true will instruct the tools to write out data
using the machine's natural byte order. The default value of these two
configuration variables is false which is the appropriate setting when using
\HTK\ in a multiple machine architecture environment. In an environment
comprising entirely of machines with \texttt{VAX} byte order both configuration
parameters can be set true which will disable the byte swapping procedure
during reading and writing of data.

\mysect{Summary}{openvsum}

This section summarises the globally-used environment
variables\index{environment variables} and configuration
parameters\index{configuration parameters!operating environment}. It
also provides a list of all the standard command line options used
with \HTK.

Table~\href{t:openvcparms} lists all of the configuration parameters
along with a brief description.  A missing module name means that it
is recognised by more than one module. Table~\href{t:openvars} lists
all of the environment parameters used by these modules.  Finally,
table~\href{t:stdopts} lists all of the standard options.

\begin{center}
\begin{tabular}{|p{1.4cm}|p{3.0cm}|p{6.4cm}|} \hline
Module & Name  & Description  \\ \hline
\htool{HShell} & \texttt{ABORTONERR}      & Core dump on error (for debugging) \\
\htool{HShell} & \texttt{HWAVEFILTER}     & Filter for waveform file input\\
\htool{HShell} & \texttt{HPARMFILTER}     & Filter for parameter file input\\
\htool{HShell} & \texttt{HLANGMODFILTER}  & Filter for language model file input\\
\htool{HShell} & \texttt{HMMLISTFILTER}   & Filter for HMM list file input\\
\htool{HShell} & \texttt{HMMDEFFILTER}    & Filter for HMM definition file input\\
\htool{HShell} & \texttt{HLABELFILTER}    & Filter for Label file input\\
\htool{HShell} & \texttt{HNETFILTER}      & Filter for Network file input\\
\htool{HShell} & \texttt{HDICTFILTER}     & Filter for Dictionary file input \\ 
\htool{HShell} & \texttt{LGRAMFILTER}     & Filter for gram file input\\
\htool{HShell} & \texttt{LWMAPFILTER}     & Filter for word map file input\\
\htool{HShell} & \texttt{LCMAPFILTER}     & Filter for class map file input\\
\htool{HShell} & \texttt{LMTEXTFILTER}    & Filter for text file input\\
\htool{HShell} & \texttt{HWAVEOFILTER}    & Filter for waveform file output\\
\htool{HShell} & \texttt{HPARMOFILTER}    & Filter for parameter file output\\
\htool{HShell} & \texttt{HLANGMODOFILTER} & Filter for language model file output\\
\htool{HShell} & \texttt{HMMLISTOFILTER}  & Filter for HMM list file output\\
\htool{HShell} & \texttt{HMMDEFOFILTER}   & Filter for HMM definition file output\\
\htool{HShell} & \texttt{HLABELOFILTER}   & Filter for Label file output\\
\htool{HShell} & \texttt{HNETOFILTER}     & Filter for Network file output\\
\htool{HShell} & \texttt{HDICTOFILTER}    & Filter for Dictionary file output \\ 
\htool{HShell} & \texttt{LGRAMOFILTER}    & Filter for gram file output\\
\htool{HShell} & \texttt{LWMAPOFILTER}    & Filter for word map file output\\
\htool{HShell} & \texttt{LCMAPOFILTER}    & Filter for class map file output\\
\htool{HShell} & \texttt{MAXTRYOPEN}     & Number of file open retries \\
\htool{HShell} & \texttt{NONUMESCAPES}   & Prevent string output using \verb+\012+ format \\
\htool{HShell} & \texttt{NATURALREADORDER}  & Enable natural read order for HTK binary files \\
\htool{HShell} & \texttt{NATURALWRITEORDER} & Enable natural write order for HTK binary files \\
\htool{HMem} & \texttt{PROTECTSTAKS}   & Warn if stack is cut-back (debugging) \\
 & \texttt{TRACE}             & Trace control (default=0) \\
 & \texttt{STARTWORD}         & Set sentence start symbol ({\tt <s>}) \\
 & \texttt{ENDWORD}           & Set sentence end symbol   ({\tt </s>}) \\
 & \texttt{UNKNOWNNAME}       & Set OOV class symbol      ({\tt !!UNK}) \\
 & \texttt{RAWMITFORMAT}      & Disable \HTK\ escaping for LM tools\\
\htool{LWMap}  & \texttt{INWMAPRAW}  & Disable \HTK\ escaping for input word lists and maps \\
\htool{LWMap}  & \texttt{OUTWMAPRAW} & Disable \HTK\ escaping for output word lists and maps \\
\htool{LCMap}  & \texttt{INCMAPRAW}  & Disable \HTK\ escaping for input class lists and maps \\
\htool{LCMap}  & \texttt{OUTCMAPRAW} & Disable \HTK\ escaping for output class lists and maps \\
\hline
\end{tabular}
\tabcap{openvcparms}{Configuration Parameters used in Operating Environment}
\end{center}

\vspace*{1cm}
\begin{center}
\begin{tabular}{|p{2.6cm}|p{8.2cm}|} \hline
Env Variable &  Meaning  \\ \hline
\texttt{HCONFIG}     &   Name of default configuration file\\
\texttt{HxxxFILTER} & Input/Output filters as above \\ \hline
\end{tabular}
\tabcap{openvars}{Environment Variables used in Operating Environment}
\end{center}

\vspace*{1cm}
\begin{center}
\begin{tabular}{|p{2.6cm}|p{8.2cm}|} \hline
Standard Option &   Meaning  \\ \hline
\texttt{-A}     &   Print command line arguments\\
\texttt{-B}     &  Store output HMM macro files in binary \\
\texttt{-C cf}  &   Configuration file is cf \\
\texttt{-D}     &   Display configuration variables\\
\texttt{-F fmt}  &  Set source data file format to fmt \\
\texttt{-G fmt}  &  Set source label file format to fmt \\
\texttt{-H mmf}  &  Load HMM macro file mmf \\
\texttt{-I mlf}  &  Load master label file mlf \\
\texttt{-J tmf}  &  Load transform model file tmf \\
\texttt{-K tmf}  &  Save transform model file tmf \\
\texttt{-L dir}  &  Look for label files in directory dir \\
\texttt{-M dir}  &  Store output HMM macro files in directory dir \\
\texttt{-O fmt}  &  Set output data file format to fmt \\
\texttt{-P fmt}  &  Set output label file format to fmt \\
\texttt{-Q}     &   Print command summary info\\
\texttt{-S scp}  &   Use command line script file scp \\
\texttt{-T N}  &   Set trace level to N \\
\texttt{-V}     &   Print version information\\
\texttt{-X ext} &   Set label file extension to ext \\ \hline
\end{tabular}
\tabcap{stdopts}{Summary of Standard Options}
\end{center}\index{standard options!summary}


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "htkbook"
%%% End: