lm-scripts.1

这是一款很好用的工具包

lm-scripts(1)                                       lm-scripts(1)



NNAAMMEE
       lm-scripts,  add-dummy-bows,  change-lm-vocab,  empty-sen-
       tence-lm, get-unigram-probs, make-hiddens-lm, make-lm-sub-
       set, make-sub-lm, remove-lowprob-ngrams, reverse-lm, sort-
       lm - manipulate N-gram language models

SSYYNNOOPPSSIISS
       aadddd--dduummmmyy--bboowwss [_l_m_-_f_i_l_e] >>_n_e_w_-_l_m_-_f_i_l_e
       cchhaannggee--llmm--vvooccaabb --vvooccaabb _v_o_c_a_b --llmm _l_m_-_f_i_l_e --wwrriittee--llmm _n_e_w_-_l_m_-
       _f_i_l_e [--ttoolloowweerr] [--ssuubbsseett] [_n_g_r_a_m_-_o_p_t_i_o_n_s...]
       eemmppttyy--sseenntteennccee--llmm  --pprroobb  _p  --llmm _l_m_-_f_i_l_e --wwrriittee--llmm _n_e_w_-_l_m_-
       _f_i_l_e [_n_g_r_a_m_-_o_p_t_i_o_n_s...]
       ggeett--uunniiggrraamm--pprroobbss [lliinneeaarr==11]
       mmaakkee--hhiiddddeennss--llmm [_l_m_-_f_i_l_e] >>_h_i_d_d_e_n_s_-_l_m_-_f_i_l_e
       mmaakkee--llmm--ssuubbsseett _c_o_u_n_t_-_f_i_l_e|-- [_l_m_-_f_i_l_e|--]
       mmaakkee--ssuubb--llmm [mmaaxxoorrddeerr==_N] [_l_m_-_f_i_l_e] >>_n_e_w_-_l_m_-_f_i_l_e
       rreemmoovvee--lloowwpprroobb--nnggrraammss [_l_m_-_f_i_l_e] >>_n_e_w_-_l_m_-_f_i_l_e
       rreevveerrssee--llmm [_l_m_-_f_i_l_e] >>_n_e_w_-_l_m_-_f_i_l_e
       ssoorrtt--llmm [_l_m_-_f_i_l_e] >>_s_o_r_t_e_d_-_l_m_-_f_i_l_e

DDEESSCCRRIIPPTTIIOONN
       These scripts perform various useful manipulations  on  N-
       gram models in their textual representation.  Most operate
       on backoff N-grams in ARPA nnggrraamm--ffoorrmmaatt(5).

       Since these tools are implemented as  scripts  they  don't
       automatically  input or output compressed model files cor-
       rectly, unlike the main SRILM tools.  However, since  most
       scripts  work with data from standard input or to standard
       output (by leaving out the file argument, or specifying it
       as  ``-'')  it  is  easy to combine them with gguunnzziipp(1) or
       ggzziipp(1) on the command line.

       Also note that many of the scripts take their options with
       the ggaawwkk(1) syntax _o_p_t_i_o_n==_v_a_l_u_e instead of the more common
       --_o_p_t_i_o_n _v_a_l_u_e.

       aadddd--dduummmmyy--bboowwss adds dummy backoff weights to N-grams, even
       where  they are not required, to satisfy some broken soft-
       ware that expects backoff weights on all  N-grams  (except
       those of highest order).

       cchhaannggee--llmm--vvooccaabb  modifies  the  vocabulary  of an LM to be
       that in _v_o_c_a_b.  Any N-grams  containing  out-of-vocabulary
       words  are  removed, new words receive a unigram probabil-
       ity, and the model is renormalized.  The  --ttoolloowweerr  option
       causes  case  distinctions  to  be  ignored.  --ssuubbsseett only
       removes words from the LM vocabulary, without adding  any.
       Any  remaining  _n_g_r_a_m_-_o_p_t_i_o_n_s  are passes to nnggrraamm(1), and
       can be used to set debugging level, N-gram order, etc.

       eemmppttyy--sseenntteennccee--llmm modifies an LM so  that  it  allows  the
       empty sentence with probability _p.  This is useful to mod-
       ify existing LMs that are trained on  non-empty  sentences
       only.   _n_g_r_a_m_-_o_p_t_i_o_n_s  are  passes to nnggrraamm(1), and can be
       used to set debugging level, N-gram order, etc.

       mmaakkee--hhiiddddeennss--llmm constructs an N-gram  model  that  can  be
       used  with  the nnggrraamm --hhiiddddeennss option.  The new model con-
       tains intra-utterance sentence boundary tags ``<#s>'' with
       the  same probability as the original model had final sen-
       tence tags </s>.  Also, utterance-initial  words  are  not
       conditioned on <s> and there is no penalty associated with
       utterance-final </s>.  Such as model might work better  it
       the  test  corpus is segmented at places other than proper
       <s> boundaries.

       mmaakkee--llmm--ssuubbsseett forms a new LM containing only the  N-grams
       found  in  the  _c_o_u_n_t_-_f_i_l_e, in nnggrraamm--ccoouunntt(1) format.  The
       result still needs to be renormalized with  nnggrraamm  --rreennoorrmm
       (which  will also adjust the N-gram counts in the header).

       mmaakkee--ssuubb--llmm removes N-grams of order  exceeding  _N.   This
       function  is  now  redundant, since all SRILM tools can do
       this implicitly (without using extra memory and very small
       time  overhead) when reading N-gram models with the appro-
       priate --oorrddeerr parameter.

       rreemmoovvee--lloowwpprroobb--nnggrraammss eliminates N-grams whose probability
       is  lower than that which they would receive through back-
       off.  This is useful when building  finite-state  networks
       for  N-gram  models.   However,  this function is now per-
       formed much faster by nnggrraamm(1)  with  the  --pprruunnee--lloowwpprroobbss
       option.

       rreevveerrssee--llmm produces a new LM that generates sentences with
       probabilities equal to the reversed sentences in the input
       model.

       ssoorrtt--llmm  sorts the n-grams in an LM in lexicographic order
       (left-most words being the most significant).  This is not
       a  requirement  for SRILM, but might be necessary for some
       other LM software.  (The LMs output by  SRILM  are  sorted
       somewhat  differently, reflecting the internal data struc-
       tures used; that is also the order that should  give  best
       cache utilization when using SRILM to read models.)

       ggeett--uunniiggrraamm--pprroobbss  extracts the unigram probabilities in a
       simple table format from a backoff  language  model.   The
       lliinneeaarr==11  option  causes  probabilities  to be output on a
       linear (instead of log) scale.

SSEEEE AALLSSOO
       ngram-format(5), ngram(1).

BBUUGGSS
       These are quick-and-dirty scripts, what do you expect?
       rreevveerrssee--llmm supports  only  bigram  LMs,  and  can  produce
       improper probability estimates as a result of inconsistent
       marginals in the input model.

AAUUTTHHOORR
       Andreas Stolcke <[e-mail]>.
       Copyright 1995-2006 SRI International



SRILM Tools        $Date: 2006/11/18 22:32:45 $     lm-scripts(1)