training-scripts.1

这是一款很好用的工具包

training-scripts(1)                           training-scripts(1)



NNAAMMEE
       training-scripts,    compute-oov-rate,   continuous-ngram-
       count,   get-gt-counts,   make-abs-discount,   make-batch-
       counts,   make-big-lm,  make-diacritic-map,   make-google-
       ngrams,  make-gt-discounts,  make-kn-counts,  make-kn-dis-
       counts,   merge-batch-counts,  replace-words-with-classes,
       reverse-ngram-counts,  split-tagged-ngrams,  reverse-text,
       uniform-classes,  vp2text - miscellaneous conveniences for
       language model training

SSYYNNOOPPSSIISS
       ggeett--ggtt--ccoouunnttss mmaaxx==_K oouutt==_n_a_m_e [_c_o_u_n_t_s...]
       mmaakkee--aabbss--ddiissccoouunntt _g_t_c_o_u_n_t_s
       mmaakkee--ggtt--ddiissccoouunnttss mmiinn==_m_i_n mmaaxx==_m_a_x _g_t_c_o_u_n_t_s
       mmaakkee--kknn--ccoouunnttss  oorrddeerr==_N   mmaaxx__ppeerr__ffiillee==_M   oouuttppuutt==_f_i_l_e   [
       nnoo__mmaaxx__oorrddeerr==11 ]
       mmaakkee--kknn--ddiissccoouunnttss mmiinn==_m_i_n _g_t_c_o_u_n_t_s
       mmaakkee--bbaattcchh--ccoouunnttss _f_i_l_e_-_l_i_s_t [_b_a_t_c_h_-_s_i_z_e [_f_i_l_t_e_r [_c_o_u_n_t_-_d_i_r
       [_o_p_t_i_o_n_s...]]]]
       mmeerrggee--bbaattcchh--ccoouunnttss _c_o_u_n_t_-_d_i_r [_f_i_l_e_-_l_i_s_t|_s_t_a_r_t_-_i_t_e_r]
       mmaakkee--ggooooggllee--nnggrraammss [ ddiirr==_D_I_R ] [ ppeerr__ffiillee==_N ] [  ggzziipp==00  ]
       [_c_o_u_n_t_s_-_f_i_l_e...]
       ccoonnttiinnuuoouuss--nnggrraamm--ccoouunntt [ oorrddeerr==_N ] [_t_e_x_t_f_i_l_e...]
       rreevveerrssee--nnggrraamm--ccoouunnttss [_c_o_u_n_t_s_-_f_i_l_e...]
       rreevveerrssee--tteexxtt [_t_e_x_t_f_i_l_e...]
       sspplliitt--ttaaggggeedd--nnggrraammss [ sseeppaarraattoorr==_S ] [_c_o_u_n_t_s_-_f_i_l_e...]
       mmaakkee--bbiigg--llmm  --nnaammee _n_a_m_e --rreeaadd _c_o_u_n_t_s [ --ttrruusstt--ttoottaallss --mmaaxx--
       ppeerr--ffiillee M ] --llmm _n_e_w_-_m_o_d_e_l [_o_p_t_i_o_n_s...]
       rreeppllaaccee--wwoorrddss--wwiitthh--ccllaasssseess ccllaasssseess==_c_l_a_s_s_e_s [oouuttffiillee==_c_o_u_n_t_s
       nnoorrmmaalliizzee==0|1    aaddddoonnee==_K    hhaavvee__ccoouunnttss==11   ppaarrttiiaall==11   ]
       [_t_e_x_t_f_i_l_e...]
       uunniiffoorrmm--ccllaasssseess _c_l_a_s_s_e_s >>_n_e_w_-_c_l_a_s_s_e_s
       mmaakkee--ddiiaaccrriittiicc--mmaapp _v_o_c_a_b
       vvpp22tteexxtt [_t_e_x_t_f_i_l_e...]
       ccoommppuuttee--oooovv--rraattee _v_o_c_a_b [_c_o_u_n_t_s...]

DDEESSCCRRIIPPTTIIOONN
       These scripts perform convenience  tasks  associated  with
       the  training  of  language  models.   They complement and
       extend the basic N-gram model estimator in nnggrraamm--ccoouunntt(1).

       Since  these  tools  are implemented as scripts they don't
       automatically input or output compressed data  files  cor-
       rectly,  unlike the main SRILM tools.  However, since most
       scripts work with data from standard input or to  standard
       output (by leaving out the file argument, or specifying it
       as ``-'') it is easy to combine  them  with  gguunnzziipp(1)  or
       ggzziipp(1) on the command line.

       Also note that many of the scripts take their options with
       the ggaawwkk(1) syntax _o_p_t_i_o_n==_v_a_l_u_e instead of the more common
       --_o_p_t_i_o_n _v_a_l_u_e.

       ggeett--ggtt--ccoouunnttss  computes  the  counts-of-counts  statistics
       needed  in  Good-Turing  smoothing.   The  frequencies  of
       counts  up to _K are computed (default is 10).  The results
       are  stored  in  a  series  of  files  with   root   _n_a_m_e,
       _n_a_m_e..ggtt11ccoouunnttss,  _n_a_m_e..ggtt22ccoouunnttss,  ..., _n_a_m_e..ggtt_Nccoouunnttss.  It
       is assumed  that  the  input  counts  have  been  properly
       merged, i.e., that there are no duplicated N-grams.

       mmaakkee--ggtt--ddiissccoouunnttss takes one of the output files of ggeett--ggtt--
       ccoouunnttss and computes  the  corresponding  Good-Turing  dis-
       counting factors.  The output can then be passed to nnggrraamm--
       ccoouunntt(1) via the --ggtt_n options  to  control  the  smoothing
       during  model estimation.  Precomputing the GT discounting
       in this fashion has the advantage that the  GT  statistics
       are  not  affected  by  restricting  N-grams  to a limited
       vocabulary.   Also,  ggeett--ggtt--ccoouunnttss/mmaakkee--ggtt--ddiissccoouunnttss   can
       process  arbitrarily  large count files, since they do not
       need to read the counts into memory (unlike  nnggrraamm--ccoouunntt).

       mmaakkee--aabbss--ddiissccoouunntt  computes  the absolute discounting con-
       stant needed  for  the  nnggrraamm--ccoouunntt  --ccddiissccoouunntt_n  options.
       Input is one of the files produced by ggeett--ggtt--ccoouunnttss.

       mmaakkee--kknn--ddiissccoouunntt  computes  the discounting constants used
       by the modified Kneser-Ney smoothing method.  Input is one
       of the files produced by ggeett--ggtt--ccoouunnttss.

       mmaakkee--bbaattcchh--ccoouunnttss  performs  the  first  stage in the con-
       struction of very large N-gram count files.  _f_i_l_e_-_l_i_s_t  is
       a  list  of  input  text files.  Lines starting with a `#'
       character are ignored.  These files will be  grouped  into
       batches of size _b_a_t_c_h_-_s_i_z_e (default 10) that are then pro-
       cessed in one run of nnggrraamm--ccoouunntt each.  For  maximum  per-
       formance,  _b_a_t_c_h_-_s_i_z_e should be as large as possible with-
       out triggering paging.  Optionally,  a  _f_i_l_t_e_r  script  or
       program can be given to condition the input texts.  The N-
       gram  count  files  are  left   in   directory   _c_o_u_n_t_-_d_i_r
       (``counts'' by default), where they can be found by a sub-
       sequent run of mmeerrggee--bbaattcchh--ccoouunnttss.  All following  _o_p_t_i_o_n_s
       are  passed to nnggrraamm--ccoouunntt, e.g., to control N-gram order,
       vocabulary, etc.  (no options triggering model  estimation
       should be included).

       mmeerrggee--bbaattcchh--ccoouunnttss  completes  the  construction  of large
       count files by merging the batched counts left  in  _c_o_u_n_t_-
       _d_i_r  until a single count file is produced.  Optionally, a
       _f_i_l_e_-_l_i_s_t of count files to combine can be specified; oth-
       erwise  all  count  files in _c_o_u_n_t_-_d_i_r from a prior run of
       mmaakkee--bbaattcchh--ccoouunnttss will be  merged.   A  number  as  second
       argument  restarts the merging process at iteration _s_t_a_r_t_-
       _i_t_e_r.  This is convenient if merging fails to complete for
       some reason (e.g., for temporary lack of disk space).

       mmaakkee--ggooooggllee--nnggrraammss  takes a sorted count file as input and
       creates an indexed directory structure, in a format devel-
       oped  by  Google  to  store very large N-gram collections.
       The resulting directory can then be used with  the  nnggrraamm--
       ccoouunntt(1)  --rreeaadd--ggooooggllee option.  Optional arguments specify
       the output directory _d_i_r and the size _N of  individual  N-
       gram  files (default is 10 million N-grams per file).  The
       ggzziipp==00 option writes  plain,  as  opposed  to  compressed,
       files.

       ccoonnttiinnuuoouuss--nnggrraamm--ccoouunntt  generates  N-grams  that span line
       breaks (which are usually  taken  to  be  sentence  bound-
       aries).  To count N-grams across line breaks use
            continuous-ngram-count _t_e_x_t_f_i_l_e | ngram-count -read -
       The argument _N  controls  the  order  of  N-grams  counted
       (default 3), and should match  the argument of nnggrraamm--ccoouunntt
       --oorrddeerr.

       rreevveerrssee--nnggrraamm--ccoouunnttss reverses the word order of N-grams in
       a counts file or stream.  For example, to recompute lower-
       order counts from higher-order ones, but do the  summation
       over  preceding  words (rather than following words, as in
       nnggrraamm--ccoouunntt(1)), use
            reverse-ngram-counts _c_o_u_n_t_-_f_i_l_e | \
            ngram-count -read - -recompute -write - | \
            reverse-ngram-counts > _n_e_w_-_c_o_u_n_t_s

       rreevveerrssee--tteexxtt reverses the word order in text files,  line-
       by-line.   Start-  and end-sentence tags, if present, will
       be preserved.  This reversal is  appropriate  for  prepro-
       cessing  training  data  for LMs that are meant to be used
       with the nnggrraamm --rreevveerrssee option.

       sspplliitt--ttaaggggeedd--nnggrraammss expands N-gram count of word/tag pairs
       into  mixed N-grams of words and tags.  The optional sseeppaa--
       rraattoorr==_S argument allows the  delimiting  character,  which
       defaults to "/", to be modified.

       mmaakkee--bbiigg--llmm  constructs large N-gram models in a more mem-
       ory-efficient way than nnggrraamm--ccoouunntt by itself.  It does  so
       by  precomputing  the  Good-Turing or Kneser-Ney smoothing
       parameters from the full set of counts, and then instruct-
       ing  nnggrraamm--ccoouunntt  to  store only a subset of the counts in
       memory, namely those of N-grams  to  be  retained  in  the
       model.   The _n_a_m_e parameter is used to name various auxil-
       iary files.  _c_o_u_n_t_s contains the raw N-gram counts; it may
       be (and usually is) a compressed file.  Unlike with nnggrraamm--
       ccoouunntt, the --rreeaadd option can  be  repeated  to  concatenate
       multiple  count  files,  but the arguments must be regular
       files; reading from stdin is not supported.  If  Good-Tur-
       ing  smoothing  is  used  and  the  file contains complete
       lower-order counts corresponding to the  sums  of  higher-
       order  counts, then the --ttrruusstt--ttoottaallss options may be given
       for efficiency.  All other _o_p_t_i_o_n_s are  passed  to  nnggrraamm--
       ccoouunntt  (only  options affecting model estimation should be
       given).  Smoothing methods other than Good-Turing and mod-
       ified   Kneser-Ney   are  not  supported  by  mmaakkee--bbiigg--llmm.
       Kneser-Ney smoothing also requires enough  disk  space  to
       compute  and store the modified lower-order counts used by
       the KN method.  This is done using the  mmeerrggee--bbaattcchh--ccoouunnttss
       command,  and  the  --mmaaxx--ppeerr--ffiillee option controls how many
       counts are to be stored per batch, and should be chosen so
       that these batches fit in real memory.

       mmaakkee--kknn--ccoouunnttss  computes  the  modified lower-order counts
       used by the KN smoothing  method.   It  is  invoked  as  a
       helper scripts by mmaakkee--bbiigg--llmm ..

       rreeppllaaccee--wwoorrddss--wwiitthh--ccllaasssseess  replaces  expansions  of  word
       classes with  the  corresponding  class  labels.   _c_l_a_s_s_e_s
       specifies class expansions in ccllaasssseess--ffoorrmmaatt(5).  Ambigui-
       ties are resolved in favor of the  longest  matching  word
       strings.  Ties are broken in favor of the expansion listed
       first  in  _c_l_a_s_s_e_s_.   Optionally,  the  file  _c_o_u_n_t_s  will
       receive  the  expansion counts resulting from the replace-
       ments.  nnoorrmmaalliizzee==00 or  11  indicates  whether  the  counts
       should be normalized to probabilities (default is 1).  The
       aaddddoonnee option may be used to smooth the  expansion  proba-
       bilities  by  adding  _K  to  each  count (default 1).  The
       option hhaavvee__ccoouunnttss==11 indicates that the input consists  of
       N-gram  counts and that replacement should be performed on
       them.  Note this will not  merge  counts  that  have  been
       mapped  to identical N-grams, since this is done automati-
       cally when nnggrraamm--ccoouunntt(1) reads count  data.   The  option
       ppaarrttiiaall==11  prevents multi-word class expansions from being
       replaced when more than one space character  occurs  inbe-
       tween the words.

       uunniiffoorrmm--ccllaasssseess takes a file in ccllaasssseess--ffoorrmmaatt(5) and adds
       uniform probabilities to  expansions  that  don't  have  a
       probability explicitly stated.

       mmaakkee--ddiiaaccrriittiicc--mmaapp  constructs  a  map  file that pairs an
       ASCII-fied version of the words  in  _v_o_c_a_b  with  all  the
       occurring  non-ASCII word forms.  Such a map file can then
       be used with ddiissaammbbiigg(1) and a language  model  to  recon-
       struct  the  non-ASCII  word  form with diacritics from an
       ASCII text.

       vvpp22tteexxtt is a reimplementation of the filter  used  in  the
       DARPA Hub-3 and Hub-4 CSR evaluations to convert ``verbal-
       ized punctuation'' texts to language model training  data.

       ccoommppuuttee--oooovv--rraattee  determines the out-of-vocabulary rate of
       a corpus from its unigram _c_o_u_n_t_s and a  target  vocabulary
       list in _v_o_c_a_b.

SSEEEE AALLSSOO
       ngram-count(1),  ngram(1), classes-format(5), disambig(1),
       select-vocab(1).

BBUUGGSS
       Some of the tools could be generalized  and/or  made  more
       robust to misuse.

AAUUTTHHOORR
       Andreas Stolcke <[e-mail]>.
       Copyright 1995-2006 SRI International



SRILM Tools        $Date: 2006/08/11 22:35:11 $training-scripts(1)