history.prme

speech signal process tools

.\"	pic history.prme | refer -n -p esps.refs -e | gpstt | iroff -me -D'prerasterization on' -S /tmp/history.impress
.lo
.de SM          \"macro to shrink left margin
.br
.nr $i \\n($i+\\$1i
..
.de RM          \"macro to restore original left margin
.br
.nr $i 0
..
.de MS		\"macro for main section (sans serif bold)
.ft SB
.nf
\\$1
.sp -.8
.fi
..
.de SS		\"macro for sub-section (sans serif italics)
.ft SI
.nf
\\$1
.sp -.8
.fi
..
.sc
.he 'RECORD KEEPING IN ESPS''page %'
.fo '3.9'\ERL'1/22/93'
.nf
.ce 
\s+4\fBENTROPIC RESEARCH LABORATORY, INC.\fR\s-4
.sp .5i
.ce 100
\s+1\fB

ESPS APPLICATIONS NOTE: File Headers and Record Keeping in ESPS
\fP\s-1
.sp .3i
\fIJohn Shore \fP
.sp .3i
Entropic Research Laboratory, Inc.
600 Pennsylvania Ave. SE, Suite 202
Washington, D.C. 20003
[phone]
.sp .5i
.fi
.ft LR
.RM
.sh 1 "INTRODUCTION"
.sp 1
.pp
The Entropic Signal Processing System (ESPS) contains record-keeping
mechanisms that result in ESPS file headers containing a complete history
of the origin of the data in the file and of all intermediate processing
steps.  This Applications Note explains ESPS record-keeping in terms of 
two spectrum analysis examples.  For a general introduction to 
ESPS, see.
.[
ESPS introduction
.]
.(f
\(co Copyright 1988,1989,1990 Entropic Speech, Inc.; Copyright 1993 Entropic Research Lab, Inc.; All rights reserved.
.)f

.sh 2 "User Level Programs"
.pp
ESPS includes the following user level programs that support record 
keeping:
.nf
.ta 1.5i

\fIcomment\fP (1\-\s-1ESPS\s+1)	- displays comments from or adds them to ESPS file headers
\fIpsps\fP (1\-\s-1ESPS\s+1)	- print header and record information from an ESPS file
\fIhd_shrink\fP (1\-\s-1ESPS\s+1)	- remove recursive headers from an ESPS file
\fIaddgen\fP (1\-\s-1ESPS\s+1)	- adds a generic header item to an existing or new ESPS file header

.fi
The use of these programs is described in this Applications Note.  
.sh 2 "Library Functions"
.pp
ESPS programs implement record keeping using calls to the following
ESPS library functions:
.nf
.ta 2i

\fIadd_source_file\fP (3\-\s-1ESPS\s+1)	- add source file name and header to ESPS header
\fIadd_comment\fP (3\-\s-1ESPS\s+1)	- append a string to comment field of an ESPS file header

.fi
These functions are not discussed further in this document.  Users should
refer to the relevant Section 3 manual pages and to the ESPS programming
guidelines.
.[
ESPS programming guide
.]
.bp
.nf

.GP fig1.gps 4 6 0 0
.GE
.sp 1
.ce
.b "Figure 1: Portions of four SD files generated by demo script."

.fi
.sh 1 "EXAMPLE SCRIPTS"
.lp
ESPS record keeping will be illustrated in this Applications Note 
in terms of two demonstration scripts.  Both do essentially the 
same things, but one relies heavily on \s-1UNIX\s+1 pipes.  The 
first script, usually located in /usr/esps/demo/specdemo, 
adds two sine waves to Gaussian noise, computes reflection 
coefficients, and then computes a power spectrum.  Here is a 
summary of the processing (this is the actual output seen when
the demo script is run): 
.nf

.ft CW
    make 1 second of Gaussian noise
       %testsd -r20000 -Tgauss gauss.sd
    make 1 second of a sine wave at frequency 600 Hz and one at 1100 Hz
       %testsd -Tsine -f600 -r20000 sine1.sd
       %testsd -Tsine -f1100 -r20000 sine2.sd
    add the two sine waves
       %addsd sine1.sd sine2.sd sines.sd
    add the Gaussian noise to the sines
       %addsd gauss.sd sines.sd signal.sd
    compute reflection coefficients to 100th order
       %refcof -o100 -r1:2500 -l0 signal.sd signal.rc
    compute spectrum
       %me_spec signal.rc signal.spec

.ft LR
.fi
Figure 1 shows portions of the two sine waves (sine1.sd, sine2.sd), 
the sum of the two sine waves (sines.sd), and the sum of the 
sine waves plus noise (signal.sd).  The plot was generated 
using \fImlplot\fP (1\-\s-1ESPS\s+1).  \(dg
.(f
\(dg The plots were included in this Applications Note by generating
\s-1MASSCOMP\s+1 gps output from the ESPS plotting programs and then using
\fIgpstt\fP (1) to convert the results to device-independent \fItroff\fP.
.)f
Note that \fImlplot\fP (1\-\s-1ESPS\s+1) can also plot sampled data files 
in "vertical" format, so that the actual data is apparent.  This 
is shown in Fig. 2. 
.nf
.GP fig2.gps 4 6 0 0
.GE
.sp 1
.ce
.b "Figure 2: Portions of four SD files in ""vertical"" format."

.fi
.lp
Figure 3 shows the power spectrum of the signal shown in the 
last line of Fig. 1 (signal.sd).  This spectrum is contained in
the file signal.spec.  
.(z
.nf

.GP fig3.gps 3 5 0 .5
.GE
.sp 1
.ce
.b "Figure 3: Power spectrum generated by demo script."

.fi
.)z
.lp
The second example script, usually located in /usr/esps/demo/pipedemo, 
does the same thing as the \fIspecdemo\fP script, but it uses pipes.  Here
is a summary of the processing (note that the last command 
ties four ESPS commands together with pipes): 
.nf

.ft CW
    make 1 second of a sine wave at frequency 600 Hz
       %testsd -Tsine -r20000 -f600 psine1.sd

    make 1 second of a sine wave at frequency 1100 Hz and add to other one
       %testsd -Tsine -f1100 -r20000 - | addsd - psine1.sd  psines.sd

    make 1 second of Gaussian noise, then add it to the two sines;
    then compute reflection coefficients to 100th order ;
    and then compute the spectrum 
       %testsd -Tgauss -r20000 - | addsd - psines.sd - 
                     | refcof -o100 -r1:2500 -l0 - - | me_spec - psignal.spec

.ft LR
.fi
.sh 1 "OVERVIEW OF RECORD KEEPING"
.pp
ESPS files have a common structure consisting of a variable-length
header followed by data records.  Within a single ESPS file, the
records have a fixed-length that depends on information in the
header.  Record-lengths vary, however, even among files of the same
basic type.  
.pp
The ESPS header is the primary means by which record-keeping is performed
within ESPS.  The header of an ESPS file contains standard "bookkeeping"
information, a copy of the command line used to generate the file, informal
ASCII comments, and values of all important parameters to the program that
created the ESPS file.  When an ESPS program produces an output ESPS file,
the output header includes the names of any source files as well the
headers of these source files.  Since ESPS headers are recursive
structures, these source file headers themselves contain the names and
headers of files that were their sources.  Thus, ESPS file headers contain
the headers of all source files in the processing chain.  It follows that
most ESPS files contain a complete history of the origin of the data in the
file and all the intermediate processing steps.  
.pp
As an example of this, Figure 4 shows a diagram of the complete ESPS header
for signal.spec, the file produced by the program \fIme_spec\fP at the end
of the \fIspecdemo\fP script.  Each box represents the basic ESPS header of
the file named in the box, i.e., the contents of the header except for the
names and headers of source files.  Thus the top box in Fig.  4 is the
basic header of signal.spec.  The full header of signal.spec includes the
header of the input file signal.rc, which is indicated in Fig.  4 by the
arrow pointing to the basic header of signal.rc.  The header of signal.rc
in turn includes the header of signal.sd (the input file to \fIrefcof\fP,
the program that produced signal.rc), and so on.
.bp
.po 2.5i
.nf
.(b
.PS
boxwid = 0.35i; boxht = 0.15i
lineht = 0.20i
BA:  box "signal.spec"
     arrow down from BA.s
BB:  box "signal.rc"
     arrow down from BB.s
BC:  box "signal.sd"
box "gauss.sd" at BC.s - (0.25i, 0.25i)
move right 0.30i
box "sines.sd" at BC.s + (0.25i, -0.25i)
arrow from BC.s to 4th box.n
arrow from BC.s to 5th box.n
box "sine1.sd" at 5th box.s - (0.25i, 0.25i)
move right 0.30i
box "sine2.sd" at 5th box.s + (0.25i, -0.25i)
arrow from 5th box.s to 6th box.n
arrow from 5th box.s to 7th box.n
.PE
.sp 1
.fi
.)b
.po
.ce 1
.b "Figure 4: Embedded source file headers in the header of signal.spec."
.sp 1
.pp
The information in ESPS headers is available to programs that process ESPS
files, so that the behavior of these programs can depend on the such
information.  Here, however, we are concerned with the value of this
information to the human user.
.pp
The ASCII comment field in an ESPS file can be viewed using 
.i comment
(1\-\s-1ESPS\s+1), which also provides options for adding arbitrary text to
the comment field and for viewing recursively the comments in all of the
embedded headers.  Full file headers (and data records) are viewed in ASCII
form by means of the program
.i psps
(1\-\s-1ESPS\s+1), which includes options for viewing recursively all of the 
embedded headers.  The output from 
.i psps
identifies values from header and data-records fields by name. Within
programs, the fields are also referred to by name.  
.sh 1 "ESPS COMMENTS"
.pp
As I have already mentioned, one of the fields in the header is a
variable-length field containing ASCII comments that can be viewed using
.i comment
(\s-1ESPS\s+1).  By convention, ESPS programs put their command line (as
typed by the user) into the comment field of output files.  For example,
the comment field in the first sine-wave SD file from \fIspecdemo\fP can be
output as follows:
.nf
.ft CW

    %comment sine1.sd
    testsd -Tsine -f600 -r20000 sine1.sd 

.ft LR
.fi
Some programs (e.g., \fIaddsd\fP (1\-\s-1ESPS\s+1)) put in additional comments
besides the command line.  For example, the comment field in the 
file produced by adding the two sine waves sine1.sd and sine2.sd 
is output as follows:
.nf
.ft CW

    %comment sines.sd
    addsd sine1.sd sine2.sd sines.sd 
      Added samples from sine2.sd to samples 1 - 8000 of sine1.sd.

.ft LR
.fi
In the case of \fIaddsd\fP, the extra comment is included because the range
of points might not be stated on the command line (it could come from an
ESPS Parameter file or Common file, for example; see
.[[
ESPS introduction
.]
.[
ESPS parameters
.]).
.pp
A quick summary of the processing history of a file can be obtained by
looking at all the comment fields of a header and all embedded headers.
This follows because the combined fields contain, not only the command line
of the program that produced the file, but the command lines of all of the
predecessor programs.  This capability is provided by running \fIcomment\fP
with the \fB\-a\fP option.  Here is the result for signal.spec, the final
file produced by \fIspecdemo\fP.  
.nf
.ft CW

    %comment -a signal.spec
    me_spec signal.rc signal.spec 
     refcof -o100 -p1:2500 -l0 signal.sd signal.rc 
      addsd gauss.sd sines.sd signal.sd 
        Added samples from sines.sd to samples 1 - 8000 of gauss.sd.
       testsd -r20000 -Tgauss gauss.sd 
       addsd sine1.sd sine2.sd sines.sd 
         Added samples from sine2.sd to samples 1 - 8000 of sine1.sd.
        testsd -Tsine -f600 -r20000 sine1.sd 
        testsd -Tsine -f1100 -r20000 sine2.sd 
    
.ft LR
.fi
.lp
The indentation of the output increases with each level of recursion (i.e.,
with each horizontal layer in Fig.  4).  If any given file in the
processing chain was written by a program that had more than one input
file, the comments from the input files come out at the same indentation
level.
.sh 2 "The Comment Field and Pipes"
.pp
Because most ESPS programs are designed to work on \s-1UNIX\s+1 pipes, the
history mechanism was also designed so that it does not depend on the
existence of intermediate disk files.  For example, here are the comments
from the final output file produced by \fIpipedemo\fP:
.nf
.ft CW

    %comment -a psignal.spec
    me_spec - psignal.spec 
     refcof -o100 -p1:2500 -l0 - - 
      addsd - psines.sd - 
        Added samples from psines.sd to samples 1 - 2147483647 of <stdin>.
       testsd -Tgauss -r20000 - 
       addsd - psine1.sd psines.sd 
         Added samples from psine1.sd to samples 1 - 2147483647 of <stdin>.
        testsd -Tsine -f1100 -r20000 - 
        testsd -Tsine -r20000 -f600 psine1.sd 

.ft LR
.fi
Since an input to \fIaddsd\fP came from a pipe, the program 
could not tell how many samples were coming (for disk files, it
can) \- hence the huge number, which happens to be the largest 
possible value of a long.  
.sh 2 "Adding Informal Comments to the Comment Field"
.pp
Users can add their own informal comments to ESPS files by using
\fIcomment\fP with the \fB\-c\fP (command-line comment) or \fB\-C\fP
(comment file) options.  If "\-" is specified for a comment file, 
the user is prompted to give a comment on standard input.  All
user-added comments are preceded by a date stamp.
Here's an example:
.nf
.ft CW

    %comment -C - signal.spec
    comment: enter comment, end with blank line
    This file was produced by a demo script; it contains the 
    power spectrum of two sine waves plus Gaussian noise.

    %comment signal.spec
    me_spec signal.rc signal.spec 
    comment added: Fri Oct 23 10:28:53 1987
    This file was produced by a demo script; it contains the
    power spectrum of two sine waves plus Gaussian noise.

    %comment -a signal.spec
    me_spec signal.rc signal.spec 
    comment added: Fri Oct 23 10:28:53 1987
    This file was produced by a demo script; it contains the 
    power spectrum of two sine waves plus Gaussian noise.
     refcof -o100 -p1:2500 -l0 signal.sd signal.rc 
      addsd gauss.sd sines.sd signal.sd 
        Added samples from sines.sd to samples 1 - 8000 of gauss.sd.
       testsd -r20000 -Tgauss gauss.sd 
       addsd sine1.sd sine2.sd sines.sd 
         Added samples from sine2.sd to samples 1 - 8000 of sine1.sd.
        testsd -Tsine -f600 -r20000 sine1.sd 
        testsd -Tsine -f1100 -r20000 sine2.sd 
    
.ft LR
.fi
Like other comments, user-added comments are propagated
by the ESPS record-keeping mechanism \- thus, they appear 
in the comment fields of subsequent files.  
.sh 1 "DETAILED RECORD KEEPING IN ESPS FILE HEADERS"
.pp
ESPS file headers comprise three parts:  the universal section, the
type-specific section, and generic header items.  When headers are viewed
with \fIpsps\fP, all header items are identified by name.  The header items
are also referred to by these same names within ESPS programs, which makes
it easier to write, read, and maintain programs.
.sh 3 "The Universal Section of the Header"
.pp
As the name implies, the universal section of the ESPS header has the same
definition for all ESPS file types.  The comment field discussed above is
part of the universal section, which also contains information about the
type of data in the file, the program that created the file, the user who
created the file, the input source files, etc.  
.pp
When \fIpsps\fP is called without any options, it outputs the universal
section of the header followed by all of the data records.  The universal
header alone, can be viewed with the \fB\-D\fP (no data) option, and the
universal header plus a restricted range of data records can be viewed with
the \fB\-r\fP (range) option.  Here, for example, is the universal header
and first 15 data records of a sampled data file from \fIspecdemo\fP:
.ft CW
.nf

    %psps -r1:15 sines.sd
    File: sines.sd

    ---Universal Part of Header---
    File type: FEA (Feature File)
    File header date: Tue Mar 20 14:37:01 1990 , header version: 1.73
    Produced by: addsd, version: 3.12, of: 2/19/90, user: burton
    Hostname:path: epiwrl:/u2/burton/esps/doc
    Machine type: Masscomp; file is in NATIVE format.
    Number of data records: 8000
    Data is not tagged.
    comment text:
     addsd sine1.sd sine2.sd sines.sd 
       Added samples from sine2.sd to samples 1 - 8000 of sine1.sd.
    Source files: sine1.sd
                  sine2.sd
    There is no reference header.
    
         1:      0.0000   263.0596   502.7742   698.2844   833.4172
         6:    898.3527   890.5669   814.9475   683.0757   511.7628
        11:    321.0198   131.6999   -36.9073  -169.2464  -255.1805

.ft LR
.fi
The meaning of most of the header items is obvious; for the 
others see ESPS (5\-\s-1ESPS\s+1) in Section 5 of the User's Manual.  
.sh 2 "Type-Specific Section of the Header"
.pp
As the name implies, this section of the header is used to record
information relevant to the specific file type.  The items defined for