history.doc

speech signal process tools

                         ENTROPIC RESEARCH LABORATORY, INC.





          ESPS APPLICATIONS NOTE: File Headers and Record Keeping in ESPS



                                    John Shore


                         Entropic Research Laboratory, Inc.
                        600 Pennsylvania Ave. SE, Suite 202
                               Washington, D.C. 20003
                                   [phone]




       1 .  INTRODUCTION


            The Entropic Signal Processing  System  (ESPS)  contains  record-
       keeping  mechanisms that result in ESPS file headers containing a com-
       plete history of the origin of the data in the file and of all  inter-
       mediate  processing  steps.   This  Applications  Note  explains  ESPS
       record-keeping in terms of two spectrum analysis examples.  For a gen-
       eral introduction to ESPS, see. [1]


       1 .1 .  User Level Programs

            ESPS includes the following  user  level  programs  that  support
       record keeping:

       comment (1-ESPS)- displays comments from or adds them to ESPS file headers
       psps (1-ESPS)  - print header and record information from an ESPS file
       hd_shrink (1-ESPS)- remove recursive headers from an ESPS file
       addgen (1-ESPS)- adds a generic header item to an existing or new ESPS file header

       The use of these programs is described in this Applications Note.




       ____________________
          c Copyright 1988,1989,1990 Entropic Speech,  Inc.;  Copyright  1993
       Entropic Research Lab, Inc.; All rights reserved.












       RECORD KEEPING IN ESPS                                          page 2


       1 .2 .  Library Functions

            ESPS programs implement record keeping using calls to the follow-
       ing ESPS library functions:

       add_source_file (3-ESPS)- add source file name and header to ESPS header
       add_comment (3-ESPS)- append a string to comment field of an ESPS file header

       These functions are not discussed further  in  this  document.   Users
       should  refer  to  the relevant Section 3 manual pages and to the ESPS
       programming guidelines. [2]












































       3.9                              ERL                           1/22/93







       RECORD KEEPING IN ESPS                                          page 3


           Figure 1: Portions of four SD files generated by demo script.


       2 .  EXAMPLE SCRIPTS

       ESPS record keeping will be illustrated in this Applications  Note  in
       terms  of  two  demonstration  scripts.   Both do essentially the same
       things, but one relies heavily on UNIX pipes.  The first script,  usu-
       ally  located in /usr/esps/demo/specdemo, adds two sine waves to Gaus-
       sian noise, computes reflection  coefficients,  and  then  computes  a
       power  spectrum.   Here  is  a  summary of the processing (this is the
       actual output seen when the demo script is run):

           make 1 second of Gaussian noise
              %testsd -r20000 -Tgauss gauss.sd
           make 1 second of a sine wave at frequency 600 Hz and one at 1100 Hz
              %testsd -Tsine -f600 -r20000 sine1.sd
              %testsd -Tsine -f1100 -r20000 sine2.sd
           add the two sine waves
              %addsd sine1.sd sine2.sd sines.sd
           add the Gaussian noise to the sines
              %addsd gauss.sd sines.sd signal.sd
           compute reflection coefficients to 100th order
              %refcof -o100 -r1:2500 -l0 signal.sd signal.rc
           compute spectrum
              %me_spec signal.rc signal.spec

       Figure 1 shows portions of the two sine  waves  (sine1.sd,  sine2.sd),
       the  sum  of  the  two  sine waves (sines.sd), and the sum of the sine
       waves plus noise (signal.sd).  The plot  was  generated  using  mlplot
       (1-ESPS).  | Note that mlplot (1-ESPS)  can  also  plot  sampled  data
       files in "vertical" format, so that the actual data is apparent.  This
       is shown in Fig. 2.

             Figure 2: Portions of four SD files in "vertical" format.


       Figure 3 shows the power spectrum of the signal shown in the last line
       of  Fig.  1  (signal.sd).   This  spectrum  is  contained  in the file
       signal.spec.



                 Figure 3: Power spectrum generated by demo script.


       ____________________

          | The plots were included in this Applications Note  by  generating
       MASSCOMP  gps  output  from  the ESPS plotting programs and then using
       gpstt (1) to convert the results to device-independent troff.




       3.9                              ERL                           1/22/93







       RECORD KEEPING IN ESPS                                          page 4


       The second example script, usually located in /usr/esps/demo/pipedemo,
       does  the  same thing as the specdemo script, but it uses pipes.  Here
       is a summary of the processing (note that the last command  ties  four
       ESPS commands together with pipes):

           make 1 second of a sine wave at frequency 600 Hz
              %testsd -Tsine -r20000 -f600 psine1.sd

           make 1 second of a sine wave at frequency 1100 Hz and add to other one
              %testsd -Tsine -f1100 -r20000 - | addsd - psine1.sd  psines.sd

           make 1 second of Gaussian noise, then add it to the two sines;
           then compute reflection coefficients to 100th order ;
           and then compute the spectrum
              %testsd -Tgauss -r20000 - | addsd - psines.sd -
                            | refcof -o100 -r1:2500 -l0 - - | me_spec - psignal.spec


       3 .  OVERVIEW OF RECORD KEEPING

            ESPS files have a common  structure  consisting  of  a  variable-
       length  header  followed  by data records.  Within a single ESPS file,
       the records have a fixed-length that depends  on  information  in  the
       header.   Record-lengths  vary,  however, even among files of the same
       basic type.

            The ESPS header is the primary means by which  record-keeping  is
       performed  within  ESPS.  The header of an ESPS file contains standard
       "bookkeeping" information, a copy of the command line used to generate
       the file, informal ASCII comments, and values of all important parame-
       ters to the program that created the ESPS file.  When an ESPS  program
       produces  an output ESPS file, the output header includes the names of
       any source files as well the headers of  these  source  files.   Since
       ESPS headers are recursive structures, these source file headers them-
       selves contain the names and headers of files that were their sources.
       Thus, ESPS file headers contain the headers of all source files in the
       processing chain.  It follows that most ESPS files contain a  complete
       history of the origin of the data in the file and all the intermediate
       processing steps.

            As an example of this, Figure 4 shows a diagram of  the  complete
       ESPS  header for signal.spec, the file produced by the program me_spec
       at the end of the specdemo script.  Each box represents the basic ESPS
       header  of the file named in the box, i.e., the contents of the header
       except for the names and headers of source files.  Thus the top box in
       Fig.   4  is  the  basic  header  of  signal.spec.  The full header of
       signal.spec includes the header of the input file signal.rc, which  is
       indicated  in  Fig.   4  by  the arrow pointing to the basic header of
       signal.rc.  The header of signal.rc in turn  includes  the  header  of
       signal.sd  (the  input  file  to  refcof,  the  program  that produced
       signal.rc), and so on.




       3.9                              ERL                           1/22/93







       RECORD KEEPING IN ESPS                                          page 5


		FIGURE NOT REPRODUCED IN ON-LINE VERSION


        Figure 4: Embedded source file headers in the header of signal.spec.


            The information in ESPS headers is  available  to  programs  that
       process  ESPS files, so that the behavior of these programs can depend
       on the such information.  Here, however, we  are  concerned  with  the
       value of this information to the human user.

            The ASCII comment field in an ESPS file can be viewed using  com-
       ment  (1-ESPS),  which also provides options for adding arbitrary text
       to the comment field and for viewing recursively the comments  in  all
       of  the  embedded  headers.   Full file headers (and data records) are
       viewed in ASCII form by means of  the  program  psps  (1-ESPS),  which
       includes  options for viewing recursively all of the embedded headers.
       The output from psps identifies values from  header  and  data-records
       fields  by  name.  Within programs, the fields are also referred to by
       name.

       4 .  ESPS COMMENTS

            As I have already mentioned, one of the fields in the header is a
       variable-length  field  containing  ASCII  comments that can be viewed
       using comment (ESPS).  By convention, ESPS programs put their  command
       line  (as  typed  by the user) into the comment field of output files.
       For example, the comment field in the first  sine-wave  SD  file  from
       specdemo can be output as follows:

           %comment sine1.sd
           testsd -Tsine -f600 -r20000 sine1.sd

       Some programs  (e.g.,  addsd  (1-ESPS))  put  in  additional  comments
       besides  the command line.  For example, the comment field in the file
       produced by adding the two sine waves sine1.sd and sine2.sd is  output
       as follows:


       3.9                              ERL                           1/22/93







       RECORD KEEPING IN ESPS                                          page 6


           %comment sines.sd
           addsd sine1.sd sine2.sd sines.sd
             Added samples from sine2.sd to samples 1 - 8000 of sine1.sd.

       In the case of addsd, the extra comment is included because the  range
       of  points might not be stated on the command line (it could come from
       an ESPS Parameter file or Common file, for example; see[1,3).

            A quick summary of the  processing  history  of  a  file  can  be
       obtained  by  looking  at  all  the comment fields of a header and all
       embedded headers.  This follows because the combined  fields  contain,
       not  only  the command line of the program that produced the file, but
       the command lines of all of the predecessor programs.  This capability
       is provided by running comment with the -a option.  Here is the result
       for signal.spec, the final file produced by specdemo.

           %comment -a signal.spec
           me_spec signal.rc signal.spec
            refcof -o100 -p1:2500 -l0 signal.sd signal.rc
             addsd gauss.sd sines.sd signal.sd
               Added samples from sines.sd to samples 1 - 8000 of gauss.sd.
              testsd -r20000 -Tgauss gauss.sd
              addsd sine1.sd sine2.sd sines.sd
                Added samples from sine2.sd to samples 1 - 8000 of sine1.sd.
               testsd -Tsine -f600 -r20000 sine1.sd
               testsd -Tsine -f1100 -r20000 sine2.sd


       The indentation of the output increases with each level  of  recursion
       (i.e.,  with  each horizontal layer in Fig.  4).  If any given file in
       the processing chain was written by a program that had more  than  one
       input  file,  the  comments  from the input files come out at the same
       indentation level.

       4 .1 .  The Comment Field and Pipes

            Because most ESPS programs are designed to work  on  UNIX  pipes,
       the  history mechanism was also designed so that it does not depend on
       the existence of intermediate disk files.  For example, here  are  the
       comments from the final output file produced by pipedemo:

           %comment -a psignal.spec
           me_spec - psignal.spec
            refcof -o100 -p1:2500 -l0 - -
             addsd - psines.sd -
               Added samples from psines.sd to samples 1 - 2147483647 of <stdin>.
              testsd -Tgauss -r20000 -
              addsd - psine1.sd psines.sd
                Added samples from psine1.sd to samples 1 - 2147483647 of <stdin>.
               testsd -Tsine -f1100 -r20000 -
               testsd -Tsine -r20000 -f600 psine1.sd

       Since an input to addsd came from a pipe, the program could  not  tell


       3.9                              ERL                           1/22/93







       RECORD KEEPING IN ESPS                                          page 7


       how many samples were coming (for disk files, it can) - hence the huge
       number, which happens to be the largest possible value of a long.

       4 .2 .  Adding Informal Comments to the Comment Field

            Users can add their own informal comments to ESPS files by  using
       comment  with  the  -c  (command-line  comment)  or  -C (comment file)
       options.  If "-" is specified for a comment file, the user is prompted
       to give a comment on standard input.  All user-added comments are pre-
       ceded by a date stamp.  Here's an example:

           %comment -C - signal.spec
           comment: enter comment, end with blank line
           This file was produced by a demo script; it contains the
           power spectrum of two sine waves plus Gaussian noise.

           %comment signal.spec
           me_spec signal.rc signal.spec
           comment added: Fri Oct 23 10:28:53 1987
           This file was produced by a demo script; it contains the
           power spectrum of two sine waves plus Gaussian noise.

           %comment -a signal.spec
           me_spec signal.rc signal.spec
           comment added: Fri Oct 23 10:28:53 1987
           This file was produced by a demo script; it contains the
           power spectrum of two sine waves plus Gaussian noise.
            refcof -o100 -p1:2500 -l0 signal.sd signal.rc
             addsd gauss.sd sines.sd signal.sd
               Added samples from sines.sd to samples 1 - 8000 of gauss.sd.
              testsd -r20000 -Tgauss gauss.sd
              addsd sine1.sd sine2.sd sines.sd
                Added samples from sine2.sd to samples 1 - 8000 of sine1.sd.
               testsd -Tsine -f600 -r20000 sine1.sd
               testsd -Tsine -f1100 -r20000 sine2.sd

       Like other comments, user-added comments are propagated  by  the  ESPS
       record-keeping  mechanism - thus, they appear in the comment fields of
       subsequent files.