revisions.txt

基于稀疏网络的精选机器学习模型

SNoW revision history

This report is intended to be useful to users as well as SNoW's developers.
For questions or bug reports, contact Nick Rizzolo ([e-mail])

10/6/99 
  - initial SNoW release (v2.0)

11/10/99 (v2.0.1)
  - fixed bug in train/test mode that miscounted the number of examples and
    misreported accuracy as a result.  this only affected the -T option.

2/10/00 (v2.0.2)
  - fixed bugs with online learning mode:
    - when an eligibility threshold of 1 was used, a garbage example was
      presented to the network for training due to a file i/o bug
    - when an eligibility threshold of 2 or greater was used, no new features
      were added to the network because the active feature counts were not being
      updated properly 

3/23/00 (v2.0.3)
  - added "base activation" output to allactivations and allboth output modes.
    in addition to the normalized activation, the "true" activation of the
    target concepts are output.

9/18/00 (v2.0.4)
  - fixed bugs in the counts of positive examples and active features in the
    network file output
  - fixed a bug in the activation output where not enough space was given to
    output extreme activations

1/22/01 (v2.1.0)
  - added "percentage eligibility" feature pruning (use '-e percent:x')
  - count-based eligibility threshold is now specified with '-e count:x' rather
    than '-e x' - some old scripts may break as a result
  - added conjunction generation option (see -g flag)
  - added option for Winnow and Perceptron to calculate the default weight
    parameter using information from the training data.  to use, omit the
    default weight from the algorithm definition.
  - fixed a bug where the fixed feature was added to targets even if no positive
    examples were seen for that target, which caused the target to have higher
    activation than it should
  - the Makefile now specifies the -O2 flag for optimizations, which gives a
    substantial increase in performance in some cases.  if this causes problems,
    simply remove the flag.  
  - fixed a bug that caused segfaults when NaiveBayes networks had targets which
    never saw positive examples during training.  this caused segfaults when
    that network was used for testing.
  - fixed a bug in the 'winners' output mode where an endline was missing  

7/5/02 (v3.0.0)
  - thoroughly revamped Usage.h, the file in which SNoW's "manpage" resides.  it
    is dumped to STDOUT when SNoW is executed with no command line parameters.

  - bug fixes:
    - fixed a bug in the example parsing function ReadLabeled() in which the
      first feature in the example was not dealt with appropriately.
    - fixed a bug involving target IDs in full (as opposed to sparse) network
      mode.  target IDs were treated as features by other targets.
    - SNoW's internal representation of a target used to store the number of
      features linked to that target in a member variable.  this member variable
      had no discernable function, and was written to the network anyway.
      meanwhile, another important internal variable, the non-active count of
      the target itself, was not being written to the network, and this caused
      incorrect behavior in incremental learning ('-i +').  SNoW now writes the
      non-active count where the useless active feature count used to be.  this
      new network specification is completely compatible with the old one.

  - performance improvements:
    - SNoW's internal representation of an example now stores a simple array of
      features instead of the STL <map> it used to.  this way, examples take up
      less memory, and SNoW runs much faster in training and testing.
    - during testing, time was taken to find any targets that used Naive Bayes
      as their learning algorithm.  for any such targets, PrepareToRank() was
      called.  as it turns out, PrepareToRank() does nothing in the Naive Bayes
      code.  so, the code that finds Naive Bayes targets and calls
      PrepareToRank() was removed.
    - SNoW no longer checks for duplicated features as examples are read in.
    - check out the new '-M' command line parameter for storing examples in
      memory.

  - added the '-server' mode to which clients can connect to get their examples
    tested.  the main advantage to running a server is that the network doesn't
    have to be reloaded every time another example (or file full of examples)
    needs to be tested.

  - new algorithms:
    - the thick separater training algorithm has been added.  the network is now
      updated when the activation of a positive example is less than <threshold>
      + <thickness>, and when the activation of a negative example is greater
      than <threshold> - <thickness>.  to set <thickness>, specify
      '-S <thickness>' on the command line.  the default is 0.
    - the threshold relative algorithm has been added.  when in use with Winnow,
      either alpha or beta (depending on if a mistake was made on a positive or
      negative example) is multiplied by <threshold> / <activation>.  when in
      use with Perceptron, <learning_rate> is multiplied by
      (<threshold> - <activation>).  specify '-t +' to enable it.  the default
      is '-t -'.
    - the constraint classification training algorithm has been added.  in the
      code, it is referred to as "ordered targets mode."  to enable it, specify
      '-O +' on the command line.  the default is '-O -'.
    - added sequential model functionality.  there is no command line parameter
      associated with it; just a new example format, fully compatible with the
      old one.  a normal example lists features separated by commas (with
      optional strengths in parenthases) and ends with a colon.  now, you can
      put a semi-colon after the last feature and list target IDs thereafter,
      all before the terminating colon.  the example (before the semi-colon)
      will be presented to only those target IDs listed after the semi-colon.
      if there is no semi-colon, or if there are no targets listed thereafter,
      the example is presented to all targets, as usual.

  - algorithmic modifications:
    - changed the Perceptron update rule.  it used to be w_i += learning_rate.
      now it is w_i += learning_rate * strength_i.
    - changed the Winnow update rule.  it used to be w_i *= alpha. now it is
      w_i *= pow(alpha, strength_i).

  - new command line parameters:
    - added the '-a' command line parameter.  previously, pending features were
      not written to the network.  now, specifying '-a +' in train mode will
      cause all non-discarded features that have been encountered at least once
      by a target to be written to the network.  the default is '-a -'.
    - modified the functionality of the '-e' parameter so that features' active
      counts are checked against the eligibility theshold as they are read in
      from the network file in testing mode, and their discard states are set to
      pending or active as indicated.
    - added the fixed feature command line parameter.  to suppress creation of
      the fixed feature in each example, specify '-f -' on the command line.
      the default is '-f +'.
    - added the '-M' command line parameter for training mode.  previously, the
      internal representation of an example was discarded after that example had
      been trained on.  when a new cycle began, the input stream was rewound,
      and examples were parsed all over again.  now, specifying '-M +' forces
      all examples to be stored internally, which can greatly reduce execution
      time when running many cycles (> 10).  the default is '-M -'.
    - added the '-z' command line parameter which enables "conventional" or
      "raw" mode.  this mode is intended to adhere to the most traditional
      definitions of the learning algorithms used.  the network is full instead
      of sparse, features are immediately linked to all targets, activations are
      not normalized, and several alternate defaults are set for other command
      line parameters.  the default is '-'.

  - output related modifications:
    - redirected all the output created by output modes > VERBOSE_MIN to the
      file specified in the '-R' parameter, if one is specified.
    - redirected the output in evaluation mode to the file specified by the '-R'
      parameter, if one is specified.
    - SNoW now reports the number of cycles run after training is complete if
      the network got 100% of the training examples correct in one cycle, thus
      ending the training process early.
    - changed the '-c <interval>' learning curve parameter.  it used to write a
      new network file after every <interval> examples of training.  now, it
      must be specified in conjunction with the '-T <test file>' parameter.
      after every <interval> training examples, it tests the network with
      <test file> and outputs results.  the network is only written at the end
      of training.  it does not test during the first cycle if '-u -' is also
      specified.

11/23/02 (v3.0.1)
  - updated the booleanexample/ and tutorial/ directories.
  - made small change to the makefile.
  - added "percentage eligibility" feature pruning to the Perceptron algorithm.
    (use '-e percent:x')

1/15/03 (v3.0.2)
  - the '-m' flag default setting has been changed from '-' to '+'.  this will
    not affect any aspect of SNoW's performance on training data with no more
    than one target ID per example.
  - the setting of the '-m' flag can now affect testing and evaluation when used
    in conjunction with a non-zero smoothing parameter.  when set to '-', a
    given target will treat all other target IDs as features, and previously
    unseen features are smoothed during testing and evaluation.  users who have
    never changed this flag's setting in the past will not see any differences
    as a result of this modification if the flag is left set to its new default
    ('+').
  - SNoW is now more robust towards target IDs in examples.  during testing, an
    example's label will now always be recognized as a label no matter where in
    the example it appears.  previously, it had to be the first feature in the
    example.  during testing and training, SNoW performs more efficiently,