idl_stsdas.tex

basic median filter simulation

\documentstyle[12pt]{article}
\begin{document}
% This file is taken from Chapter 10 of the GHRS users manual, written by
% J. Blackwell, S.N. Shore, R.D. Robinson, K. Feggans, D. Lindler,
% E. Malamuth, J. Sandoval, and T.B. Ake.    It is also available via 
% anonymous FTP from ftp://hrssun.gsfc.nasa.gov/pub/ghrsdaf/docs as the
% file swq-v2_1-sec10.ps.
% Last Edit 04/15/94
% removed change bars 
%
\Large
\section{\bf Manipulating STSDAS Files and Tables}
\label{fits}
\label{sdas tables and files}
\normalsize

\index{SDAS tables}
\index{SDAS images}
\index{FITS images}
\index{GEIS files}
Most of 
the data delivered by the STScI are in the form of {\it STSDAS} 
files and tables.  STSDAS files conform to a FITS format catered to 
HST data and can contain 1-dimensional (e.g., spectra), or multi-dimensional 
(e.g., images) data.  STSDAS tables are in a format that is unique to STSDAS. 
Throughout this section, we will refer to STSDAS files as {\it image} files 
and STSDAS tables as {\it table} files (see also the STScI 
{\it STSDAS} users' manual for additional details).\\

Table files are designed for storing tabular information. Unlike image 
files, which exist as a separate {\it header} and {\it data} file, the data 
are all contained in a single file with {\bf .tab} as the default 
extension.  Header information is stored {\it with the data} and is 
accessible through various IDL procedures described in this section. The data 
itself is arranged by row and column. Each column can have its own data type 
and can be accessed by {\it either} a column number or a column name. The 
rows are {\it always} accessed by number, beginning with zero for the 
first row.\\

Some examples of image and table files are the GEIS files (image) , 
GHRS calibration files (image and table; found in the {\bf ZCAL} directory), 
and data files (primarily image) from other HST instruments such as FOS and 
WF/PC.  These are the ones you are most likely to use.\\

This section 
begins with a brief introduction to the FITS format concentrating
on the STSDAS {\it Extended} FITS format used by the STScI.  Next, we 
discuss the IDL routines used to manipulate STSDAS image files (the {\bf SX} 
procedures).  Following this we provide an explanation of STSDAS table files 
and finally we outline the IDL routines used to create and access tables.\\

One final note about terminology.  This section uses the terms {\it STSDAS}, 
{\it FITS}, and {\it SDAS} somewhat interchangeably.  The first simply 
refers to the newer name for {\it SDAS} and the second {\it can} refer to 
to either STSDAS-FITS, or the FITS format in general; the former being most
often the case.

\subsection{\bf FITS Primer}

FITS is an acronym for {\bf Flexible Image Transport System}.  It was 
originally developed as a standard means to transfer astronomical data from
one place to another via 9-track magnetic tape using fixed physical record
sizes.  Though this is still its most basic use, the original
format has undergone numerous changes over the years.  This primer is intended
to provide only the briefest of introductions to the FITS format and how it
relates 
to HST data.\\

In its most basic form, a FITS file contains an ASCII text header
which may or may not be followed by a {\it binary} data array (the
presence of which is dictated by the FITS keywords {\bf NAXIS} or {\bf
NAXISn}).  If accompanied by binary data, both text and data are
contained in a {\it single} file.  However, in most applications
(including HST data), the header file and the data file are separate,
but linked, entities. 

\subsubsection{\bf The FITS Header}
\index{FITS header}
FITS header records contain parameters which describe the structure of the 
binary data.   These parameters are structured into a {\it keyword}, an 
optional {\it value}, and an optional {\it comment}.  Each of these records 
follows a strict formatting convention.  The header file consists of a number 
of these so-called {\it card images} (i.e., each record is fixed-length, 
80-character ASCII text).  The file extension for FITS headers follows the 
convention {\bf .xxh} where {\bf xx} can be any alphanumeric characters 
(e.g., {\bf .d0h}).  {\bf Note:} IDL routines described in this section 
default to the extension {\bf .hhh}.\\

One immediate consequence of fixed-length files is that one must take
care when attempting to edit them using a text editor (such as EDT). 
For instance, anything which is added to the file must be 80 bytes
(characters) in length.  However, the fixed-length format should not
cause problems when reviewing the FITS header files on a terminal
or when sending them to a printer (IDL software utilities discussed in
subsequent sections are designed to allow for the manipulation of FITS
header files). 

\subsubsection{\bf FITS and GEIS}
\index{GEIS format}
Any HST observation consists of a series of both binary data and ASCII
header files that are collectively referred to as a {\it Generic
Edited Information Set} or as ``GEIS files''.  GEIS files have as
their basis an {\it Extended} FITS format.\\

Each ASCII header file consists of a number of 80-byte card images 
that contain parameter information for the data (i.e., describes the format 
of the binary data file).  The header files also contain a number of 
additional records which describe the details of the observation (i.e., 
target, observer, position, instrument configuration and condition, etc...).  
The file names follow conventions discussed in Section~\ref{geis}.\\

Each binary data file consists of one or more {\it fixed size blocks} of data
called {\bf groups}.  Each group has associated with it a fixed size
block of data called a {\bf parameter block} (a.k.a {\it a group parameter}),
which contains zero or more parameters.  The parameter block is physically 
stored at the 
end of each group.  A minimum structure consists of a 
{\it single} group and {\it no} group parameters.

\subsubsection{\bf Mandatory FITS Header Keywords}
\index{FITS keywords}
As with all FITS format files, the headers must contain a certain number of 
keywords (usually as the first few records) that define the structure of the 
binary data files.  Those which are common to components of a given set of 
GEIS files are listed below. {\bf Note:} most of these keywords are common 
to all FITS headers.

\begin{itemize}

\item {\bf SIMPLE}---This keyword specifies whether the data file conforms to 
the {\it basic} FITS standard, that is, a file consisting of a primary header 
followed by a single data array.  The value of {\bf SIMPLE} is either 
{\bf T} (``True'') or {\bf F} (``False'').  Since GEIS files are in 
{\it Extended} FITS, this keyword is normally set to {\bf F}.
  
\item {\bf BITPIX}---This keyword is assigned an integer value, the absolute
value of which is used in computing the sizes of data structures.  It 
specifies the number of bits that represent a data value. The only 
allowed values for {\bf BITPIX} are given in Table 10.1.

\begin{scriptsize}
\begin{center}
{\bf Table 10.1}\\
Valid BITPIX Keyword Values\\
\end{center}
\begin{center}
\begin{tabular}{rl} \hline \hline
Value  & Description \\ \hline \\
8 & Character or unsigned binary integer\\
16 & 16-bit twos complement binary integer\\ 
32 & 32-bit twos complement binary integer\\ 
-32 & IEEE single precision floating point\\
-64 & IEEE double precision floating point\\ \hline
\end{tabular}
\end{center}
\end{scriptsize}
  
\item {\bf DATATYPE}---Specifies the data type of the data items.  All data
items are in FORTRAN-format unless otherwise specified.  Supported data types 
are listed in Table 10.2.

\begin{scriptsize}
\begin{center}
{\bf Table 10.2}\\
Supported Values of DATATYPE Keyword\\
\end{center}
\begin{center}
\begin{tabular}{ll} \hline \hline
DATATYPE & Comment \\ \hline \\
INTEGER$*$1 & \\
UNSIGNED$*$1 & (non-FORTRAN)\\
INTEGER$*$2 & \\
UNSIGNED$*$2 & (non-FORTRAN)\\
INTEGER$*$4 & \\
UNSIGNED$*$4 & (non-FORTRAN)\\
REAL$*$4 & \\
REAL$*$8 & \\
COMPLEX$*$8 & \\
COMPLEX$*$16 & \\ \hline
\end{tabular}
\end{center}
\end{scriptsize}
\newpage

\item {\bf NAXIS}---
This keyword is assigned a positive integer value which
gives the number of dimensions (or coordinate axes) in each data group.

\item {\bf NAXISn}---This {\it indexed} keyword (hence the {\bf n}) is 
a positive integer designating the size of each component
of a group {i.e., n = 1,...,NAXIS}.  The ordering of each component
goes in order of decreasing size.  For example, {\bf NAXIS =}{\it 2} means 
that the data are contained in a 2 dimensional array (recall that in IDL a 
1-D array is usually called a {\it vector}).  Suppose that the data array 
contains an image which is 100$\times$200 
(i.e., {\bf data=}{\it data(100,200)}).  In this case, 
{\bf NAXIS1} would be {\it 200}, and {\bf NAXIS2} would be {\it 100}.  
{\bf Note:} if {\bf NAXIS=}{\it 0}, there should not be any {\bf NAXISn} 
keywords.

\item {\bf GROUPS}---A logical keyword that, if {\bf T}, specifies that the 
binary data has a {\it group format} with parameter blocks.

\item {\bf GCOUNT}---An integer keyword specifying the number of groups 
contained in the binary data file.

\item {\bf PCOUNT}---An integer keyword specifying the number of group 
parameters in each parameter block associated with each group.

\item {\bf PSIZE}---An integer keyword specifying the sum total of bits 
in all the parameter blocks.

\item {\bf PSIZEn}---An {\it indexed} integer keyword specifying the number 
of {\it bits} in each group parameter (n=1,$\cdots$,PCOUNT).

\item {\bf PTYPEn}---An {\it indexed} character string keyword which 
specifies the name of each group parameter (n=1,$\cdots$,PCOUNT).  {\bf Names 
identified with these GROUP keywords are the group parameter names or 
``the group parameters'' of the data file}.

\item {\bf PDTYPEn}---An {\it indexed} character string keyword which 
specifies the data type of each group parameter (n=1,$\cdots$,PCOUNT).
  
\item {\bf END}---A keyword used to designate the end of the FITS header.  It
has no value.
\newpage

\end{itemize}

As an example, 
consider the following header segment (from the {\bf .d0h} file)
for the FOS observation whose rootname is {\it y0ya0201t}:

\begin{scriptsize}
\begin{center}
\begin{verbatim}
SIMPLE  =                    F /                                                
BITPIX  =                   32 /                                                
DATATYPE= 'REAL*4'             /                                                
NAXIS   =                    2 /                                                
NAXIS1  =                   64 /                                                
NAXIS2  =                    1 /                                                
GROUPS  =                    T /                                                
GCOUNT  =                    8 /                                                
PCOUNT  =                   25 /                                                
PSIZE   =                 1024 /                                                
PTYPE1  = 'CRVAL1  '           / right ascension of reference pixel (deg)       
PDTYPE1 = 'REAL*8'             /                                                
PSIZE1  =                   64 /                                                
PTYPE2  = 'CRVAL2  '           / declination of reference pixel (deg)           
PDTYPE2 = 'REAL*8'             /                                                
PSIZE2  =                   64 /                                                
PTYPE3  = 'CRPIX1  '           / x-coordinate of reference pixel                
PDTYPE3 = 'REAL*4'             /                                                
PSIZE3  =                   32 /                                                
PTYPE4  = 'CRPIX2  '           / y-coordinate of reference pixel                
PDTYPE4 = 'REAL*4'             /                                                
.
.
.
PTYPE25 = 'Y_OFFSET'           / y_offset for GIMP correction (defl.units)      
PDTYPE25= 'REAL*4'             /                                                
PSIZE25 =                   32 /                                                
.
.
.
END                                                                             
\end{verbatim}
\end{center}
\end{scriptsize}

{\bf Note:} this particular file is online in the 
{\bf hrs8:[hrsdata.fos.0ya002]} directory.\\

{\bf Translation}: The binary data file associated with this 
header (the {\bf .d0d} file) contains {\it 8} {\bf groups} or arrays of type
{\it REAL$*$4} (32 bit) that are two dimensional (64$\times$1, a vector in IDL).
Each of the groups has 25 parameters following it of different sizes {\bf PSIZEn},
names {\bf PTYPEn}, and formats {\bf PDTYPEn}.  {\bf Note:} each of the
{\bf PTYPEn} gives the name of the parameter and a description of what that
parameter is.

\subsubsection{\bf Other Header Keywords}

\begin{itemize}

\item {\bf HISTORY}---This is one of the valid {\it Bibliographic} keywords 
that contains a log of the steps taken in 
processing the data.  Any number of {\bf HISTORY} card images may appear in a
header.  One example of the use of this keyword can be found in the 
GHRS calibration files (the {\bf ZCAL} directory path).

\end{itemize}
\newpage

\subsection{\bf Manipulating Image Files: SX Routines}
\label{sdas images}
\index{SDAS image}

The 
following sections describe the IDL procedures needed to create,
read and manipulate STSDAS image files.  Some of these routines are very
basic and can be used with most FITS format files, while others are
specific to the STSDAS Extended FITS image files (e.g., GEIS).
The basic routines for reading and creating STSDAS image files are listed 
below:\\

\begin{description}
\item [SXOPEN :] A procedure used to open an STSDAS image file.
\item [SXMAKE :] A procedure to create new STSDAS image files (header and data).
\item [SXREAD :] A function used to read data from an STSDAS image data file.
\item [SXHREAD :] A procedure used to read the STSDAS image header file.
\item [SXWRITE :] A procedure used to write data to an STSDAS image file.
\item [SXHWRITE :] A procedure that writes header information to a STSDAS image 
header file.
\item [SXGPAR :] A function used to extract keyword values from the group
parameters in the STSDAS FITS image header file.
\item [SXPAR :] A function that extracts values of STSDAS FITS header keywords.
\item [SXADDPAR :] A procedure used to add or modify STSDAS FITS header keywords.
\item [SXADDHIST :] A procedure used to add {\it history} lines to an STSDAS 
FITS header file.
\item [SXHEDIT :] A procedure used for editing STSDAS FITS header files.
\end{description}

\subsubsection{\bf Opening the Image File}
\index{SXOPEN}
\index{SDAS images - opening}
\label{sxopen}
\begin{description}
\item [SXOPEN, {\it unit, filename $[$, HEADER, HISTORY, access$]$} : ]

This procedure opens an image file for processing. It first opens the 
header file, reads it, closes it, and then opens the binary data 
file.  The binary file is then left open.

\begin{itemize}

\item {\bf unit}---
An IDL logical unit number from 1 to 9.

\item {\bf filename}---The name of the header file.  If another set of
file extensions are used, they must have the form {\bf .xxh} and {\bf .xxd} 
where {\bf xx} can be any alphanumeric characters.  {\bf Note:} VMS file 
version numbers cannot be used in {\bf filename} and that {\bf SXOPEN} will use 
the most recent (highest version number) files.
\newpage
\item {\bf HEADER}---
An optional FITS format string vector to which is 
returned the names, values and comments from the FITS header 
{\bf Note:} use the function {\bf SXPAR} to obtain individual parameter values.

\item {\bf HISTORY}---An optional string vector to which is returned 
the {\bf HISTORY} records from the header, if they exist.