files.doc

speech signal process tools

                         ENTROPIC RESEARCH LABORATORY, INC.


       TECHNICAL MEMORANDUM

       SUBJECT: Data Files in the Entropic Signal Processing System (ESPS)

       ID: ETM-S-86-13:rap/jtb

       VERSION: 3.5

       DATE: 1/22/93

       AUTHOR: Alan Parker and Joe Buck

       LAST REVISED BY: John Shore

       REFERENCES:

        [1] Entropic Signal Processing System, Programmers Reference Manual

        [2] ETM-S-86-12, Parameter and Common Files in  the  Entropic  Signal
            Processing System (ESPS)

        [3] ETM-S-86-14, Entropic Signal Processing System Programming Guide-
            lines

        [4] ETM-S-86-21, Introduction to the Entropic Signal Processing  Sys-
            tem (ESPS)

        [5] ETM-S-86-25, Guidelines for Implementing ESPS Feature  File  Sub-
            types

        [6] ESPS APPLICATIONS NOTE: File Headers and Record Keeping in ESPS

        [7] ESPS APPLICATIONS NOTE: Converting Data  to  and  from  ESPS  FEA
            Files

        [8] ETM-S-88--3, Machine Independent File I/O (MIIO) Module for ESPS

       1 .  Introduction

       This document describes the Entropic Signal Processing  System  (ESPS)
       data  files  and presents information required to understand and write
       programs using the ESPS data file header structures.

       The reader must also read and understand  the  ESPS  manual  pages  in
       reference  [1].    These manual pages are in the standard UNIX format;
       ____________________
          c
            Copyright 1987-1990 Entropic  Speech,  Inc.  Copyright  1991-1993
       Entropic Research Laboratory, Inc.,  All rights reserved.











       ETM-S-86-13:rap/jtb                                             page 3


       Section 1 describes user commands, Section 3  describes  library  rou-
       tines,  and  Section 5 describes the format of ESPS files.  This docu-
       ment together with reference [1] is the complete  description  of  the
       ESPS file structures.

       For additional information on ESPS programming guidelines and a sample
       program  see  reference  [3].   For a general introduction to ESPS see
       [4].

       This document also describes support for reading files with  NIST  and
       other foreign headers, and how non-ESPS programs can read ESPS files.

       2 .  Design Goals

       The design goals for ESPS files are:

         +  Store all relevant information about the file, including analysis
            conditions,  source  files, history, and the type of data that is
            in the file.

         +  Flexibility.  We will have sampled data  files,  files  of  fixed
            length  records,  and record files in which a tag marking a posi-
            tion is associated with each record.  We may want to store 8, 16,
            or 32-bit integers, or 32 or 64-bit floating point values.

         +  The design should allow programs that only understand  a  few  of
            the  header  items to run correctly.  For example, one could ima-
            gine a general plotting program that uses certain header items to
            determine where the data starts, whether the data type is integer
            or floating, and to put appropriate labels on the plot.   A  pro-
            gram  should  not need special knowledge about how a certain type
            of data is stored; it should all be in the header.

         +  The header should provide enough information to repeat an experi-
            ment or to check files for compatibility before processing.

         +  The design should allow new file types to be designed  and  added
            without affecting existing data files of another type.

         +  The design should  allow  users  to  add  additional  ("generic")
            header items to existing ESPS file types without affecting exist-
            ing data files of that type or existing programs that  deal  with
            files of that type.

         +  The header in a specific data file type should only contain items
            that are used by that specific file type.

         +  It should be possible to create user-defined ESPS file types.

         +  All fields in a data record should not have to be the same  type;
            e.g. some fields are integer while others are floating.




       Version 3.5                      ERL                           1/22/93







       ETM-S-86-13:rap/jtb                                             page 4


         +  Provide support for non-ESPS (foreign) headers.

         +  Provide reasonable support for non-ESPS programs to  access  ESPS
            files.

         +  Provide the ability for ESPS files to be moved  across  different
            computer architectures (by dealing with issues such as byte swap-
            ping, etc.).

       For a general introduction to ESPS files and file headers, see [6].

       3 .  ESPS File Types

       All ESPS files conform to a common structure, consisting of a standard
       header  followed  by data records.  The header is the primary means by
       which record-keeping is performed within ESPS.  An ESPS header include
       the values of all important parameters to the program that created the
       ESPS file, and the header is a recursive structure  that  permits  the
       inclusion  of  the headers of all source files that were inputs to the
       program.  It follows that most ESPS files contain a  complete  history
       of  the origin of the data in the file and all intermediate processing
       steps.  ESPS file headers and data records can be viewed by  means  of
       the program psps (1-ESPS).

       Current ESPS file types in the general release include the following:

            FILT (obsolete) digital filters

            SCBK            scalar quantization codebooks

            SD (obsolete)   sampled

            SPEC (obsolete) spectra

            FEA             feature file

       For  a  complete  description  of  the  ESPS  data  files   refer   to
       ESPS(5-ESPS),  SD(5-ESPS), FILT(5-ESPS), SCBK(5-ESPS), and FEA(5-ESPS)
       in reference [1].

       The FEA type deserves special mention.  It is a general  purpose  file
       type  that  can be used to store arbitrary information in fixed-length
       records; optionally, the records can point  to  positions  in  various
       source  files.   Unlike  the  case for the other ESPS file types - the
       names, sizes, and data types of the fields  in  FEA  records  are  not
       pre-defined.   This  means  that FEA files can be used to create user-
       defined ESPS files that have all of the  advantages  of  the  built-in
       file  types.   For  details,  see  [5].   We  are replacing all of the
       built-in file types with FEA subtypes.  Currently, the  following  are
       available:

         FEA_SPEC
              used for storing spectral information, such as  power  spectra.


       Version 3.5                      ERL                           1/22/93







       ETM-S-86-13:rap/jtb                                             page 5


              Each  record  contains  a  vector of spectral values (powers or
              power densities in the case of power  spectra),  together  with
              certain  optional  fields - total power, an imaginary-part vec-
              tor, number of frequencies, and a vector of frequencies -  that
              are present or absent according to the values of certain header
              items;

         FEA_ANA
              suitable for speech processing;  contains  spectral  parameters
              (various  representations),  pulse lengths, powers, and voicing
              information;

         FEA_VQ
              contains vector quantization codebooks;

         FEA_STAT
              contains statistical  information  (means,  covariances,  etc.)
              suitable for pattern-classification;

         FEA_SD
              contains sampled data; supports multi-channel and complex data

         FEA_FILT
              contains digital filter coefficients

       4 .  ESPS File Headers

       ESPS data file headers consist of two primary parts; the  common  sec-
       tion  and  the type specific section.  All ESPS data file headers have
       the same common section.  This section contains  information  relating
       to  the type of the data in the file, creation date and version infor-
       mation, housekeeping information required by the  header  access  rou-
       tines, and comments.  The type-specific section of the header contains
       data relevant to the specific file type.  Data fields  in  the  header
       are called header items.

       Internally, a file header is stored as a C  structure.   In  the  file
       itself,  there  are  two types of header items.  Most header items are
       assigned a fixed location  and  are  always  present.   The  remaining
       header  items  may  or may not be present and may be variable in size.
       These are stored in the file after the fixed location items, but  pro-
       grammers  need  not  be  concerned about the actual data format in the
       file.

       Analysis conditions stored in file headers reflect the  history  of  a
       file, not commands to programs that operate on the file.  In ESPS, the
       parameter file, the common file, and in some cases  the  command  line
       specify  analysis  conditions and options that affect the operation of
       many ESPS programs.  ESPS programs consult  the  parameter  file,  the
       common  file,  and  the  command line for analysis conditions, and for
       most analysis conditions store these in the header of the output  data
       file.  For information about the parameter and common files, see [2].



       Version 3.5                      ERL                           1/22/93







       ETM-S-86-13:rap/jtb                                             page 6