database.tex

basic median filter simulation

\documentstyle[12pt,titlepage]{article}
\topmargin -0.2in
\oddsidemargin-0.15in
\evensidemargin-0.25in
\textheight 8.5in     
\textwidth 6.5in
\newcommand{\de}{$^{\rm o}$ } 
\newcommand{\kms}{km~s$^{-1}$ } 
\newcommand{\exbegin}{\par\medskip}
\newcommand{\exend}{\medskip\noindent}
\newcommand{\exs}[2]{
\hbox to \hsize{\small\hskip .2in
\parbox[t]{2.2in}{\raggedright\setlength{\parindent}{-.2in}\tt #1}
\hspace{.2in}
\parbox[t]{3.4in}{\raggedright\setlength{\parindent}{-.2in}\rm #2}\hss}
\prevdepth=1.5pt\relax}
\newcommand{\exc}[2]{
\hbox to \hsize{\small\hskip .2in
\parbox[t]{3.0in}{\raggedright\setlength{\parindent}{-.2in}\tt #1}
\hspace{.2in}
\parbox[t]{2.6in}{\raggedright\setlength{\parindent}{-.2in}\rm #2}\hss}
\prevdepth=1.5pt\relax}
% One line example
\newcommand{\exone}[1]{\begin{center}\tt #1 \end{center}}
\newcommand{\ea}{{\em et~al.} {}}
\newcommand{\apjl}{{\em Ap.~J. (Letters)}}
\begin{document}
\title{The IDLAstro Database System \\(Version July  2001)}
\author{W.B. Landsman \\ Raytheon ITSS}
\maketitle
\section{INTRODUCTION}

This document describes the use of an IDL database system that is available in
the IDL Astronomy Library ({\tt http://idlastro.gsfc.nasa.gov/}).   The
database software was designed by Don Lindler of the GHRS (Goddard
High-Resolution Spectrograph) group, and  has been adopted by several
astronomy  groups.      The software was originally written in 1987 (though
modest improvements are  made each year) and is a ``flat'' (as opposed to a
relational or object-oriented) database system.   It is thus a ``primitive''
database system compared to modern commercial alternatives but remains useful
because  it shares the IDL programming, plotting, and image display syntax.   
This makes the database system very versatile and easy to learn for those
already familiar with IDL programming.
In addition, the database software can be run on any computer 
with IDL installed (Unix, Windows, MacOS, or VMS).

In discussing the database software, it is useful to have in mind a
book copy of, for example, the Yale Bright Star Catalogue 5th Edition 
called `YALE\_BS' in the computer database.  (A copy of YALE\_BS and other 
IDL databases is available at \\
{\tt http://idlastro.gsfc.nasa.gov/ftp/zdbase/}).  The data for a specific star
is contained in one row, while column headings are placed at the top of
a page.  Similar concepts apply to the computer database, and the following
terms will be referred to constantly.
\begin{description}
\item[entry]  a ``row'' of a catalogue.  The
YALE\_BS catalog contains 9110 entries.
\item[item]   a ``column'' of a catalogue.
The YALE\_BS catalog contains 41 items including `HD', `NAME', and
`V\_MAG'.  
\item[value]  the field corresponding to a specified entry and item. 
It can be either numeric or a character string.
For entry 1708 of the YALE\_BS catalog, item `NAME' contains
a value of `ALP AUR' and item `V\_MAG' contains a value of 0.08.
It is possible for an item to be multiple valued.  For example, the
`COPERNICUS'  database contains {\em Copernicus} spectra of 40 hot stars.
In this case, the value of the item `FLUX' for a particular entry
consists of 2250 numbers specifying the relative flux between 1000 and
1450 \AA.
\end{description}

The managers of the IDL database would greatly profit by suggestions from 
the users.  In particular, the managers would appreciate hearing about 
(1) astronomical
catalogues that should be added to the database  (2) improvements needed in the
help files of a database or items within a database, (3) catalogues that 
should be linked together via pointers (4) items that should
be indexed.  (As explained below, indexed items require more disk space, but
can be searched much more quickly than non-indexed items.)

Section 2 of this document describes the six ``core'' database procedures,
which may be all that are ever required by the typical user.  Section 3
discusses five more special-purpose procedures and the concept of
``pointers.''   Section 4 gives instructions on how to create or modify
a database, and need only be read by would-be experts.  
This document is most effectively read 
while sitting at a terminal, where the numerous examples can be worked out.

\section{DATABASE FUNDAMENTALS}
All of the database procedures begin with the letters ``DB''.  
As with all other IDL procedures,
help on a specific database procedure can be obtained by typing
{\tt man,`$<$procedure\_name$>$'}.  These help files provide more detailed information
than is given in this document.
Although there exist a total of 35 database procedures,
the following
six ``core'' procedures will often be all that are required by the user.
\begin{description}
\item[DBOPEN] Open a catalogue for subsequent processing.
\item[DBHELP] Display information on catalogues, or items within a catalogue.
\item[DBFIND] Find entries meeting specified search criteria.
\item[DBPRINT] Print catalogue information of specified entries and items.
\item[DBEXT] Extract specified values as vectors for subsequent processing
with IDL.
\item[DBCLOSE] Close a catalogue.
\end{description}
In addition to the ``core'' procedures above, there are seven other database 
procedures that may be used occasionally.
\begin{description}
\item[DBCIRCLE] Search for entries in positional catalogue within a specified 
              radius of a specified center
\item[DBGET] Use instead of DBFIND when search values are in an IDL vector.
\item[DBMATCH] Find one entry for each element of a vector of item values.
\item[DBSORT] Sort a list of catalogue entries by any item.
\item[DB\_OR] Remove duplicate values from a list of entries.
\item[IMDBASE] Find entries within an image with a specified FITS header
\item[TVDBASE] Overlay position of entries within image with a specified FITS
header
\end{description}
  
The rest of the 35 database procedures are either low-level, or used for
building databases (see section 4).

\subsection{DBOPEN and DBHELP}
The database commands discussed here should all be entered in response
to the IDL prompt.  
To list the names of the online catalogues, one simply types {\tt DBHELP}.
Alternatively, the command {\tt DBHELP,1} will print a one-line description 
of all catalogues, and the command {\tt DBHELP,$<$name$>$} will
print a description of a specified database. 
Before any further work can be done with a database, it must be opened 
with the DBOPEN
command.  If later, one decides to work with another catalogue, then 
the DBOPEN command must be used again.  DBOPEN will close the first
database and open the second.  Once a database  has been opened, 
DBHELP is used to give the name, type of data, and brief description of items
within the database.   (If an item is multiple-valued, then a number in
parentheses will appear beside the item name.) The command 
{\tt DBHELP,1} will also show
which items are indexed, and thus much quicker to search on.
The following example illustrates these ideas:
\exbegin
\exs{\$IDL}{!Get into IDL}
\exs{DBHELP}{;List the names of all databases}
\exs{DBHELP,1}{;Give a one-line description of all databases}
\exs{DBHELP,'PTL'}{;Print a brief description of the ASTRO Program Target 
List (PTL) database}
\exs{DBOPEN,'PTL'}{;Open the PTL database}
\exs{DBHELP}{;Print the names of all items in the PTL}
\exs{DBHELP,1}{;Give the datatype and a brief description of all items and show 
which ones are indexed}
\exs{DBHELP,'JOTFID'}{;Print a description of the JOTFID item}
\exs{DBOPEN,'SAO'}{;Close the PTL catalogue and open the SAO catalogue}
\exs{DBCLOSE}{;Close all catalogues}
\exend
The output device of all the database procedures is controlled by the
TEXTOUT keyword.   The default output device (TEXTOUT=1) is the user's
terminal.   Set TEXTOUT=3 to direct output to a disk file with a default
name, or set TEXTOUT = `filename', to specify the output file name.   The
non-standard system variable !TEXTOUT can also be used instead of the
TEXTOUT keyword.    The documentation for the TEXTOPEN procedure gives a 
complete description of the TEXTOUT keyword.  

For example, if no database has yet been opened, 
then the following commands would write a one-line description of all catalogues
to a disk file.
\exbegin
\exs{DBHELP,1,TEXT = 3}{;Write a one line description of all catalogues to a
disk file DBHELP.PRT}
\exs{DBHELP,1,T='db.txt'}{;Write a one line description to a file db.txt}
\subsection{DBFIND}

The function DBFIND is used to select the entries of interest in a catalogue.
The general format of the DBFIND call is
\exone{LIST = DBFIND(`SEARCH\_CRITERIA',[ SUBLIST ] )}
LIST is an IDL longword output vector containing the desired entry numbers.  It is 
subsequently used either by DBPRINT to display the desired entry values, or
by DBEXT to extract item values for plotting or analysis.   SUBLIST is
an optional input parameter that restricts the search to a subset of the
catalogue.   SEARCH\_CRITERIA is a string or string array that contains
the desired search items.
Search criteria can be selected in seven different ways.  For example, the
JOTFID item (Joint Target File ID) in the PTL database could be searched in
the following ways: \\
\vspace{0.1in}
\begin{tabular}{lll}
 & Search Format & Example     \\
  &   & \\
(1) & ITEM = value & JOTFID = 8102 \\
(2) & ITEM = [value1,value2] & JOTFID = [8104,8105] \\
(3) & ITEM = min\_value $<$ ITEM $<$ max\_value  & 7000 $<$ JOTFID $<$ 7999 \\
(4) & ITEM $>$ min\_value    & JOTFID $>$ 3000 \\
(5) & ITEM $<$ max\_value    & JOTFID $<$ 2999 \\
(6) & ITEM = value(tolerance) & JOTFID = 5000(2000) \\
(7) & ITEM  ;non-zero value & JOTFID   \\
\end{tabular}
\vspace{0.1in}

The $>$ and $<$ signs in (3) -- (5) are interpreted as less than or equal to
(i.e.\ example (5) would include all jotfid numbers up to and including 2999).

You can use two or more search criteria at the same time by separating the
individual criteria with commas.  For example, to find the UIT observations
of normal galaxies (6000 $\leq$ JOTFID $\leq$ 6999) in the ASTRO target list,
\exbegin
\exc{dbopen,'PTL'}{;Open the Program Target List}  
\exc{list = dbfind('U=U,6000 < jotfid < 6999')}{;Specify UIT target, JOTFID
range}
\exend
Suppose, one now one wants to further restrict the list found above to targets in
the northern hemisphere.  The DBFIND search {\em could} be repeated adding
the additional search criterion {\tt 'dec $>$ 0'}.  However, it would be quicker
to restrict the search for positive declinations to the entries that have
already been selected and stored in the vector LIST:
\exone{newlist = dbfind('dec>0',list)}

When using DBHELP to display the contents of a catalogue, certain items are
identified as being ``indexed.''  Indexed items can be searched {\em much} 
faster than non-indexed items. 
{\em Use indexed items whenever possible in your search criteria.}   
(The reason that not every item is indexed is that such items require more disk
space.)
For example, one could find
the star $\alpha$ Aur in the SKYMAP star catalog as follows:
\exbegin
\exc{dbopen,'skymap'}{;Open the SKYMAP catalog}
\exc{list = dbfind(`name = alp aur')}{;Slow non-indexed search for 
the star name}
\exend
but the search will take a couple of  minutes since NAME is not an indexed
item.  On the other hand, a search on the HD number will be almost
instantaneous.
\exbegin
\exc{list = dbfind(`hd=34029')}{;Fast indexed search on the HD number}
\exend 

The most commonly used items with DBFIND are probably those 
pertaining to position.
All of the positional databases will have an indexed item named RA in decimal
hours, and an indexed item named DEC in decimal degrees.  In addition, another
set of non-indexed items may exist (e.g.\ RA\_1950, DEC\_1950) which store
the position as character strings.  These latter items are used for pretty
output with DBPRINT, {\em and should not be used with} DBFIND.  A nice
feature of DBFIND is that it recognizes numbers separated by colons as being
in sexigesimal format (e.g.\  45:30 = 45.5).
For example, to determine if 3C273 is an IRAS source, we could search on the
known position with a $15'' (=1^s)$ tolerance:
\exbegin
\exc{dbopen,`iras\_psc'}{;Open the IRAS point source catalog}
\exc{list = dbfind('ra = 12:26:33.3(0:0:1), dec=2:19:43(0:0:15)')}{;Search
on known position of 3C273}
\exc{dbprint,list,`*'}{;Print all items in table format}
\exend
The colons can be used with any item, although, of course, the use of 
sexigesimal format is most common with RA and Dec.   The help file for
DBFIND can be read to learn how to use slashes and dashes to encode the
date and time.  However, be aware that arithmetic operations {\em cannot}
be done within the search\_criteria string of DBFIND; if the right ascension
is 30 degrees, then conversion to hours must be done before the DBFIND call
\exbegin
\exc{list = dbfind('ra=30/15.')}{;ILLEGAL statement  - do not try this!}
\exend

By default, string searches are matched whenever the supplied string appears anywhere in
an item.   Thus, in the YALE\_BS catalog
\exone{list = dbfind('name = tau')}
will find all stars with the three characters `tau' appearing anywhere in
their name.    The search is case insensitive and leading and trailing blanks 
are ignored.   The /FULLSTRING keyword to DBFIND can be used if you want all
characters in the database value to match the search string (i.e. no substring
matches).  One can also use the same item twice in a search.   Thus
\exone{list = dbfind(`name = tau,name=eri')}
will find all stars (nine of them!) with the substrings `tau' and `eri' in
their name.