foo1

speech signal process tools

			 ENTROPIC RESEARCH LABORATORY, INC.


       TECHNICAL MEMORANDUM

       SUBJECT:	Data Files in the Entropic Signal Processing System (ESPS)

       ID: ETM-S-86-13:rap/jtb

       VERSION:	3.4

       DATE: 9/11/91

       AUTHOR: Alan Parker and Joe Buck

       LAST REVISED BY:	John Shore

       REFERENCES:

	[1] Entropic Signal Processing System, Programmers Reference Manual

	[2] ETM-S-86-12, Parameter and Common Files in	the  Entropic  Signal
	    Processing System (ESPS)

	[3] ETM-S-86-14, Entropic Signal Processing System Programming Guide-
	    lines

	[4] ETM-S-86-21, Introduction to the Entropic Signal Processing	 Sys-
	    tem	(ESPS)

	[5] ETM-S-86-25, Guidelines for	Implementing ESPS Feature  File	 Sub-
	    types

	[6] ESPS APPLICATIONS NOTE: File Headers and Record Keeping in ESPS

	[7] ESPS APPLICATIONS NOTE: Converting Data  to	 and  from  ESPS  FEA
	    Files

	[8] ETM-S-88--3, Machine Independent File I/O (MIIO) Module for	ESPS

       1 .  Introduction

       This document describes the Entropic Signal Processing  System  (ESPS)
       data  files  and	presents information required to understand and	write
       programs	using the ESPS data file header	structures.

       The reader must also read and understand	 the  ESPS  manual  pages  in
       reference  [1].	  These	manual pages are in the	standard UNIX format;
       ____________________
	  c
	    Copyright 1987-1990	Entropic Speech, Inc. Copyright	1991 Entropic
       Research	Laboratory, Inc.,  All rights reserved.











       ETM-S-86-13:rap/jtb					       page 2


       Section 1 describes user	commands, Section 3  describes	library	 rou-
       tines,  and  Section 5 describes	the format of ESPS files.  This	docu-
       ment together with reference [1]	is the complete	 description  of  the
       ESPS file structures.

       For additional information on ESPS programming guidelines and a sample
       program	see  reference	[3].   For a general introduction to ESPS see
       [4].

       This document also describes support for	reading	files with  NIST  and
       other foreign headers, and how non-ESPS programs	can read ESPS files.

       2 .  Design Goals

       The design goals	for ESPS files are:

	 +o  Store all relevant information about the file, including analysis
	    conditions,	 source	 files,	history, and the type of data that is
	    in the file.

	 +o  Flexibility.  We will have sampled data  files,  files  of	fixed
	    length  records,  and record files in which	a tag marking a	posi-
	    tion is associated with each record.  We may want to store 8, 16,
	    or 32-bit integers,	or 32 or 64-bit	floating point values.

	 +o  The	design should allow programs that only understand  a  few  of
	    the	 header	 items to run correctly.  For example, one could ima-
	    gine a general plotting program that uses certain header items to
	    determine where the	data starts, whether the data type is integer
	    or floating, and to	put appropriate	labels on the plot.   A	 pro-
	    gram  should  not need special knowledge about how a certain type
	    of data is stored; it should all be	in the header.

	 +o  The	header should provide enough information to repeat an experi-
	    ment or to check files for compatibility before processing.

	 +o  The	design should allow new	file types to be designed  and	added
	    without affecting existing data files of another type.

	 +o  The	design should  allow  users  to	 add  additional  ("generic")
	    header items to existing ESPS file types without affecting exist-
	    ing	data files of that type	or existing programs that  deal	 with
	    files of that type.

	 +o  The	header in a specific data file type should only	contain	items
	    that are used by that specific file	type.

	 +o  It should be possible to create user-defined ESPS file types.

	 +o  All	fields in a data record	should not have	to be the same	type;
	    _e._g. some fields are integer while others are floating.




       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 3


	 +o  Provide support for	non-ESPS (foreign) headers.

	 +o  Provide reasonable support for non-ESPS programs to	 access	 ESPS
	    files.

	 +o  Provide the	ability	for ESPS files to be moved  across  different
	    computer architectures (by dealing with issues such	as byte	swap-
	    ping, etc.).

       For a general introduction to ESPS files	and file headers, see [6].

       3 .  ESPS File Types

       All ESPS	files conform to a common structure, consisting	of a standard
       header  followed	 by data records.  The header is the primary means by
       which record-keeping is performed within	ESPS.  An ESPS header include
       the values of all important parameters to the program that created the
       ESPS file, and the header is a recursive	structure  that	 permits  the
       inclusion  of  the headers of all source	files that were	inputs to the
       program.	 It follows that most ESPS files contain a  complete  history
       of  the origin of the data in the file and all intermediate processing
       steps.  ESPS file headers and data records can be viewed	by  means  of
       the program _p_s_p_s	(1-ESPS).

       Current ESPS file types in the general release include the following:

	    FILT (obsolete) digital filters

	    SCBK	    scalar quantization	codebooks

	    SD (obsolete)   sampled

	    SPEC (obsolete) spectra

	    FEA		    feature file

       For  a  complete	 description  of  the  ESPS  data  files   refer   to
       _E_S_P_S(5-ESPS),  _S_D(5-ESPS), _F_I_L_T(5-ESPS),	_S_C_B_K(5-ESPS), and _F_E_A(5-ESPS)
       in reference [1].

       The FEA type deserves special mention.  It is a general	purpose	 file
       type  that  can be used to store	arbitrary information in fixed-length
       records;	optionally, the	records	can point  to  positions  in  various
       source  files.	Unlike	the  case for the other	ESPS file types	- the
       names, sizes, and data types of the fields  in  FEA  records  are  not
       pre-defined.   This  means  that	FEA files can be used to create	user-
       defined ESPS files that have all	of the	advantages  of	the  built-in
       file  types.   For  details,  see  [5].	 We  are replacing all of the
       built-in	file types with	FEA subtypes.  Currently, the  following  are
       available:

	 FEA_SPEC
	      used for storing spectral	information, such as  power  spectra.


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 4


	      Each  record  contains  a	 vector	of spectral values (powers or
	      power densities in the case of power  spectra),  together	 with
	      certain  optional	 fields	- total	power, an imaginary-part vec-
	      tor, number of frequencies, and a	vector of frequencies -	 that
	      are present or absent according to the values of certain header
	      items;

	 FEA_ANA
	      suitable for speech processing;  contains	 spectral  parameters
	      (various	representations),  pulse lengths, powers, and voicing
	      information;

	 FEA_VQ
	      contains vector quantization codebooks;

	 FEA_STAT
	      contains statistical  information	 (means,  covariances,	etc.)
	      suitable for pattern-classification;

	 FEA_SD
	      contains sampled data; supports multi-channel and	complex	data

	 FEA_FILT
	      contains digital filter coefficients

       4 .  ESPS File Headers

       ESPS data file headers consist of two primary parts; the	 common	 sec-
       tion  and  the type specific section.  All ESPS data file headers have
       the same	common section.	 This section contains	information  relating
       to  the type of the data	in the file, creation date and version infor-
       mation, housekeeping information	required by the	 header	 access	 rou-
       tines, and comments.  The type-specific section of the header contains
       data relevant to	the specific file type.	 Data fields  in  the  header
       are called _h_e_a_d_e_r _i_t_e_m_s.

       Internally, a file header is stored as a	C  structure.	In  the	 file
       itself,	there  are  two	types of header	items.	Most header items are
       assigned	a fixed	location  and  are  always  present.   The  remaining
       header  items  may  or may not be present and may be variable in	size.
       These are stored	in the file after the fixed location items, but	 pro-
       grammers	 need  not  be	concerned about	the actual data	format in the
       file.

       Analysis	conditions stored in file headers reflect the  history	of  a
       file, not commands to programs that operate on the file.	 In ESPS, the
       parameter file, the common file,	and in some cases  the	command	 line
       specify	analysis  conditions and options that affect the operation of
       many ESPS programs.  ESPS programs consult  the	parameter  file,  the
       common  file,  and  the	command	line for analysis conditions, and for
       most analysis conditions	store these in the header of the output	 data
       file.  For information about the	parameter and common files, see	[2].



       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 5


       There will be cases where there are not header items to save  particu-
       lar  analysis conditions	or options.  This is particularly likely dur-
       ing experimentation or testing of  changes  to  programs.   If  it  is
       determined  that	 a new analysis	condition or option is significant it
       will likely be added to the file	header in a future release.   In  the
       meantime, experimental versions are free	to use the spare locations in
       file headers (documented	in [2, Section 5] for each file	 type),	 with
       the understanding that official ESPS releases may invalidate such use.
       To further support record keeping, all ESPS programs will  record  the
       command	line  that invoked them	in the comment field of	any generated
       output file headers, and	in the case of conditions or options  without
       corresponding  header  entries  recording the command line will be the
       record of their use.  This convention facilitates record	keeping	 when
       files  are  produced  by	 unofficial  version of	programs with special
       analysis	 options.   The	 ESPS  library	routines   _g_e_t__c_m_d__l_i_n_e	  and
       _a_d_d__c_o_m_m_e_n_t (3-ESPS) support this convention.

       To allow	a single file to contain  its  entire  history,	 headers  are
       recursive  structures.  This  means  that an entire file	header may be
       included	as an optional header item. The	C structure representing  the
       header  in memory has fields that can contain pointers to other header
       structures.

       5 .  Programming	with the ESPS Header Structures

       This section presents information needed	to write programs  using  the
       ESPS header structures.

       5 .1 .  Header Structure

       Refer to	the listing of _h_e_a_d_e_r._h	in reference [1] to  help  understand
       this  section.	Also,  see _E_S_P_S(5-ESPS).  In the following, we assume
       that the	variable "head"	is a pointer to	an ESPS	 file  header  struc-
       ture.   As mentioned above, the header structure	contains two portions
       - the common portion and	the type-specific portion.  The	 common	 por-
       tion of the header itself contains two structures, _c_o_m_m_o_n and _v_a_r_i_a_b_l_e

       The structure _c_o_m_m_o_n contains items present in every ESPS  data	file,
       regardless  of  type.   This  section  contains	the  type code of the
       header, a check value, identification of	the program and	 its  version
       that  created the header, the name of the person	who ran	that program,
       the number of data records in the file, whether	the  data  is  tagged
       (meaning	 each  record  has  a  tag that	refers to a point in a source
       file), and the type of the records in the data file.  An	example	(file
       creation	date) is

	       head->common.date.


       The structure _v_a_r_i_a_b_l_e contains pointers	to source file names,  embed-
       ded  source  file  headers,  two	text fields (one is the	comment	field
       mentioned earlier), the name of a reference file,  and  two  variables
       that  tell  the	number of source file names and	embedded headers.  An


       Version 3.4			ERL			      9/11/91







       ETM-S-86-13:rap/jtb					       page 6


       example (reference file name) is

	       head->variable.refer.


       The variable portion also contains so-called "generic"  header  items,
       which are used by programs to store arbitrary information not included
       in the standard header.	This facility means that the basic design  of
       an  ESPS	file type does not have	to anticipate every application.  For
       example,	the SD file type does not make provision for storing  in  the
       file  header  the  average  rate	of zero	crossings, but a program that
       needs to	can do so by means of a	generic	header item.  If such an item
       was  created and	called "zero_crossing",	a pointer to the stored	value
       is given	by

	       get_genhd("zero_crossing", ih)