files.vtme

speech signal process tools

.lo
.de SM          \"macro to shrink left margin
.br
.nr $i \\n($i+\\$1i
..
.de RM          \"macro to restore original left margin
.br
.nr $i 0
..
.SJ "Data Files in the Entropic Signal Processing System (ESPS)"
.AU "Alan Parker and Joe Buck"
.RB "John Shore"
.TM ETM-S-86-13:rap/jtb  3.5 1/22/93
.*R
Entropic Signal Processing System, Programmers Reference Manual
.*R
ETM-S-86-12, Parameter and Common Files in the Entropic Signal Processing System (ESPS)
.*R
ETM-S-86-14, Entropic Signal Processing System Programming Guidelines
.*R
ETM-S-86-21, Introduction to the Entropic Signal Processing System (ESPS)
.*R
ETM-S-86-25, Guidelines for Implementing ESPS Feature File Subtypes
.*R
ESPS APPLICATIONS NOTE: File Headers and Record Keeping in ESPS
.*R
ESPS APPLICATIONS NOTE: Converting Data to and from ESPS FEA Files
.*R
ETM-S-88--3, Machine Independent File I/O (MIIO) Module for ESPS
.sh 1 Introduction
.lp
This document describes the Entropic Signal Processing System
(\s-1ESPS\s+1) data files and presents information
required to understand and write programs using the \s-1ESPS\s+1 data file
header structures.    
.(f
\(co Copyright 1987-1990 Entropic Speech, Inc. Copyright 1991-1993 Entropic
Research Laboratory, Inc.,  All rights reserved.  
.)f
.lp
The reader must also read and understand the \s-1ESPS\s+1
manual pages in reference [1].   These manual pages are in the
standard \s-1UNIX\s+1 format; Section 1 describes user commands, Section 3
describes library routines, and Section 5 describes the format of \s-1ESPS\s+1
files.
This document together with reference [1] is the complete description
of the \s-1ESPS\s+1 file structures.
.lp
For additional information on \s-1ESPS\s+1 programming guidelines and a
sample program see reference [3].  For a general introduction to 
ESPS see [4].  
.lp
This document also describes support for reading files with NIST
and other foreign headers, and how non-ESPS programs can read ESPS
files.
.sh 1 "Design Goals"
.lp
The design goals for ESPS files are:
.ip "\ \ \(bu"
Store all relevant information about the file, including analysis
conditions, source files, history, and the type of data that is in the
file.
.ip "\ \ \(bu"
Flexibility.  We will have sampled data files, files of fixed length
records, and record files in which a tag marking a position is
associated with each record.  We may want to store 8, 16, or 32-bit
integers, or 32 or 64-bit floating point values.
.ip "\ \ \(bu"
The design should allow programs that only understand a few of the
header items to run correctly.  For example, one could imagine a
general plotting program that uses certain header items to determine
where the data starts, whether the data type is integer or floating,
and to put appropriate labels on the plot.  A program should not need
special knowledge about how a certain type of data is stored; it
should all be in the header.
.ip "\ \ \(bu"
The header should provide enough information to repeat an experiment
or to check files for compatibility before processing.   
.ip "\ \ \(bu"
The design should allow new file types to be designed and added without 
affecting existing data files of another type.
.ip "\ \ \(bu"
The design should allow users to add additional ("generic") header 
items to existing ESPS file types without affecting existing data files
of that type or existing programs that deal with files of that type.  
.ip "\ \ \(bu"
The header in a specific data file type should only contain items that are
used by that specific file type.   
.ip "\ \ \(bu"
It should be possible to create user-defined ESPS file types.  
.ip "\ \ \(bu"
All fields in a data record should not have to be the same type;
\fIe.g.\fR some fields are integer while others are floating.
.ip "\ \ \(bu"
Provide support for non-ESPS (foreign) headers.
.ip "\ \ \(bu"
Provide reasonable support for non-ESPS programs to access ESPS files.
.ip "\ \ \(bu"
Provide the ability for ESPS files to be moved across different computer
architectures (by dealing with issues such as byte swapping, etc.).
.lp
For a general introduction to ESPS files and file headers, see [6].  
.sh 1 "ESPS File Types"
.lp
All ESPS files conform to a common structure, consisting of a standard
header followed by data records.  The header is the primary means by
which record-keeping is performed within ESPS.  An ESPS header include the
values of all important parameters to the program that created the ESPS
file, and the header is a recursive structure that permits the inclusion
of the headers of all source files that were inputs to the program.  It
follows that most ESPS files contain a complete history of the origin of
the data in the file and all intermediate processing steps.  ESPS file 
headers and data records can be viewed by means of the program 
.i psps
(1-ESPS).  
.lp
Current ESPS file types in the general release include the following:
.SM .5i
.ip "FILT (obsolete)" 16
digital filters
.ip "SCBK" 16
scalar quantization codebooks
.ip "SD (obsolete)" 16
sampled 
.ip "SPEC (obsolete)" 16
spectra 
.ip "FEA" 16
feature file
.RM
.lp
.lp
For a complete description of the \s-1ESPS\s+1 data files refer to
\fIESPS\fR(5\-\s-1ESPS\s+1), \fISD\fR(5\-\s-1ESPS\s+1),
\fIFILT\fR(5\-\s-1ESPS\s+1), \fISCBK\fR(5\-\s-1ESPS\s+1), and
\fIFEA\fR(5\-\s-1ESPS\s+1) in reference [1].
.lp
The FEA type deserves special mention.  It is a general purpose file
type that can be used to store arbitrary information in fixed-length
records; optionally, the records can point to positions in various
source files.  Unlike the case for the other ESPS file types \- the
names, sizes, and data types of the fields in FEA records are not
pre-defined.  This means that FEA files can be used to create
user-defined ESPS files that have all of the advantages of the
built-in file types.  For details, see [5].  We are replacing all of
the built-in file types with FEA subtypes.  Currently, the following
are available:
.SM .25i
.ip "FEA_SPEC"
used for storing spectral information, such as power spectra.  Each
record contains a vector of spectral values (powers or power densities
in the case of power spectra), together with certain optional fields
\- total power, an imaginary-part vector, number of frequencies, and a
vector of frequencies \- that are present or absent according to the
values of certain header items;
.ip "FEA_ANA"
suitable for speech processing; contains spectral parameters (various
representations), pulse lengths, powers, and voicing information;
.ip "FEA_VQ"
contains vector quantization codebooks;
.ip "FEA_STAT"
contains statistical information (means, covariances, etc.) suitable
for pattern-classification;
.ip "FEA_SD"
contains sampled data; supports multi-channel and complex data
.ip "FEA_FILT"
contains digital filter coefficients
.RM
.sh 1 "ESPS File Headers"
.lp
\s-1ESPS\s+1 data file headers consist of two primary parts; the common
section and the type specific section.  All \s-1ESPS\s+1 data file
headers have the same common section.  This section contains information
relating to the type of the data in the file, creation date and version
information, housekeeping information required by the header access
routines, and comments.  The type-specific section of the header
contains data relevant to the specific file type.  Data fields in the
header are called \fIheader items\fR.  
.lp
Internally, a file header is stored as a C structure.  In the file
itself, there are two types of header items.  Most header items are
assigned a fixed location and are always present.  The remaining header
items may or may not be present and may be variable in size.  These are
stored in the file after the fixed location items, but programmers need
not be concerned about the actual data format in the file.
.lp
Analysis conditions stored in file headers reflect the history of a
file, not commands to programs that operate on the file.  In
\s-1ESPS\s+1, the parameter file, the common file, and in some cases the
command line specify analysis conditions and options that affect the
operation of many \s-1ESPS\s+1 programs.  \s-1ESPS\s+1 programs consult
the parameter file, the common file, and the command line for analysis
conditions, and for most analysis conditions store these in the header
of the output data file.  For information about the parameter and common
files, see [2].  
.lp
There will be cases where there are not header items to save particular
analysis conditions or options.  This is particularly likely during
experimentation or testing of changes to programs.  If it is determined
that a new analysis condition or option is significant it will likely be
added to the file header in a future release.  In the meantime,
experimental versions are free to use the spare locations in file
headers (documented in [2, Section 5] for each file type), with the
understanding that official ESPS releases may invalidate such use.  To
further support record keeping, all \s-1ESPS\s+1 programs will record
the command line that invoked them in the comment field of any generated
output file headers, and in the case of conditions or options without
corresponding header entries recording the command line will be the
record of their use.  This convention facilitates record keeping when 
files are produced by unofficial version of programs with special 
analysis options.  The ESPS library routines
.i get_cmd_line 
and
.i add_comment
(3\-\s-1ESPS\s+1) support this convention.  
.lp
To allow a single file to contain its entire history, headers are recursive
structures. This means that an entire file header may be included as an
optional header item. The C structure representing the header in memory
has fields that can contain pointers to other header structures.
.sh 1 "Programming with the \s-1ESPS\s+1 Header Structures"
.lp
This section presents information needed to write programs using the \s-1ESPS\s+1
header structures.
.sh 2 "Header Structure"
.lp
Refer to the listing of \fIheader.h\fR in reference [1] to help
understand this section.  Also, see \fIESPS\fR(5\-\s-1ESPS\s+1).  In the
following, we assume that the variable "head" is a pointer to an ESPS
file header structure.  As mentioned above, the header structure
contains two portions \- the common portion and the type-specific
portion.  The common portion of the header itself contains two
structures, 
.i common
and
.i variable
.lp
The structure \fIcommon\fR contains items present in every \s-1ESPS\s+1
data file, regardless of type.  This section contains the type code of
the header, a check value, identification of the program and its version
that created the header, the name of the person who ran that program,
the number of data records in the file, whether the data is tagged
(meaning each record has a tag that refers to a point in a source file),
and the type of the records in the data file.  An example (file creation
date) is 
.nf

	head\->common.date.  

.fi
.lp
The structure \fIvariable\fR contains pointers to source file names,
embedded source file headers, two text fields (one is the comment field
mentioned earlier), the name of a reference file, and two variables that
tell the number of source file names and embedded headers.  An example
(reference file name) is 
.nf

	head\->variable.refer.  

.lp
The variable portion also contains so-called "generic" header items,
which are used by programs to store arbitrary information not included
in the standard header.  This facility means that the basic design of an
ESPS file type does not have to anticipate every application.  For
example, the SD file type does not make provision for storing in the
file header the average rate of zero crossings, but a program that needs
to can do so by means of a generic header item.  If such an item 
was created and called "zero_crossing", a pointer to 
the stored value is given by 
.nf

	get_genhd("zero_crossing", ih)

.fi
.lp
The type-specific portion of the header is provided by a union \fIhd\fR
that contains pointers to each possible type-specific structure.  They
are declared in a union so that additional pointers can be added in the
future without affecting existing data files (since the size of the
pointers themselves are the same).  Programmers can tell which of these
pointers to use by checking the item head\->common.type.  The pointer 
names are given in the following table:
.nf

.TS
box,center;
c c
l l.
Value of \fIcommon.type\fR	Pointer in \fIhd\fR to use
_
\s-1FT_SD\s+1	\fIhd.sd\fR
\s-1FT_SPEC\s+1	\fIhd.spec\fR
\s-1FT_FILT\s+1	\fIhd.filt\fR
\s-1FT_SCBK\s+1	\fIhd.scbk\fR
\s-1FT_FEA\s+1	\fIhd.fea\fR
.TE

.fi
.lp
For example, the sampling frequency from the header of an (old-style) SD file 
is head\->hd.sd\->sf.
.sh 3 "NIST (Sphere) Headers"
.lp
ESPS programs can read sample data files containing the NIST Shpere