database.tex

basic median filter simulation

\subsection{DBPRINT}

DBPRINT will display selected fields of database on the device specified
by the TEXTOUT keyword.  The basic calling sequence is:

\exone{DBPRINT, LIST, ITEMS, TEXTOUT = , FORMS = }
where LIST is a vector of entry numbers (e.g. as found with DBFIND), and 
ITEMS is a list of the desired items to print.   One line of output will
be generated for each entry printed, and fields will be printed with
appropiate headings.  (Page overflow will occur if the item list doesn't
fit on a single line -- 80 characters for a typical terminal and 132 
characters for a line printer).  The list of items to be used can
be specified in six different ways:
\exbegin
\exc{dbprint,list}{Display default items}
\exc{dbprint,list,''}{;Interactively select items via menu}
\exc{dbprint,list,'jotfid,id1,id2'}{;Items are in a single string}
\exc{dbprint,list,['jotfid','   id1','   id2']}{;Items are in a string array}
\exc{dbprint,list,'\$FILENAME'}{;Items are in a disk file named FILENAME,
(one item per line)}
\exc{dbprint,list,indgen(3)+1}{;Use items 1-3, (Item 0 is always the entry
number)}
\exc{dbprint,list,'*'}{;Select all items and print in table format}
\exend
An useful variant of the first form of DBPRINT is to pass
either an undefined variable or an empty string as the ITEMS parameter
to DBPRINT.   A full screen menu will appear, and the items to be printed
can be selected with the mouse.  The items selected will be returned
in the ITEMS variables, so that one can skip the menu on subsequent calls
to DBPRINT.
\exbegin
\exc{f = ''}{;Define an empty string}
\exc{dbprint,list,f}{;Interactively select items via menu, print, and return
items list in f}
\exc{dbprint,list,f,text=3}{;Print to a disk file with items previously selected}
\exend
The list of entry numbers can be either the output of DBFIND, or a scalar
or vector directly specified by the user.  Set LIST $= -1$ to print all 
entries.  Entry numbers begin with 1, so that supplying an entry number
of zero may give nonsensical results.
\exbegin
\exc{dbopen,'ptl'}{;Open the PTL database}
\exc{items = indgen(9)+1} {;Select first 9 items for printing}
\exc{dbprint,132,items}{;Print selected items of entry 132}
\exc{dbprint,indgen(50)+1,items}{;Print selected items of first 50 entries}
\exc{dbprint,-1,items}{;Print selected item, all entries}
\exend
\subsection{DBEXT}
DBEXT allows one to extract item vectors from a database for plotting
or subsequent analysis.  Its basic calling sequence is 
\exone{DBEXT,LIST,ITEMS,V1,V2.,\ldots V12}
The parameters LIST and ITEMS have the same meanings as in the DBPRINT
command.  The outputs V1,V2,\ldots are the IDL variable names to be
filled with the values of the specified items.  For example, to produce
a scatter plot of the right ascension and declination of all targets
in the PTL:
\exbegin
\exc{dbopen,'ptl'}{;Open the Program Target List}
\exc{list = dbfind('ra<24.')}{;Solar system objects have RA=99.9}
\exc{dbext,list,'ra,dec',r,d}{;Extract RA and dec}
\exc{plot,r,d,psym=3}{;Plot RA vs. Dec for all Astro targets}
\exend 
Thanks to WUPPE and BBXRT, the observed distibution of targets shows some
concentration toward the galactic plane.

For another example, de Lapparent \ea (\apjl, {\bf 302}, L1) have used the
CFA redshift survey to display a ``slice of the universe''.  Figure 1
in their paper (the so-called ``dancing man'') was obtained by plotting
galaxy velocity (distance) versus right ascension.  The galaxies were
restricted to the declination wedge 26.5\de $\leq \delta \leq$ 32.5\de,
and also $m_{B} \leq 15.5$, and $V \leq 15000$ \kms.   The following
IDL statements will quickly create a rough version of this plot.
\exbegin
\exc{dbopen,'REDSHIFT'}{;Open the CFA Redshift Catalogue}
\exc{list = dbfind('$26.5<$dec$<32.5$, b\_mag$<15.5$, $1<$vhelio$<15000$')}
{;Select galaxies meeting search criteria}
\exc{dbext,list,'ra,vhelio',ra,vhelio}{;Extract RA and velocity vectors}
\exc{plot, ra, vhelio, psym=3}{;Plot projection of galaxies on the RA-Vhelio plane}
\exend
In this example, galaxies for which a redshift has not been determined
were assigned a velocity of zero.  Therefore, it was essential that the search
on the VHELIO item had a lower limit of 1 and not 0.

It is possible for an item to contain more than one value for a particular
entry.  This is often true for databases containing spectra,   
where the wavelength and flux items will be multiple-valued.
For example, to plot a spectrum of $\eta$ Uma (HR 5191) from the 
TD-1 spectrophotometric catalog:
\exbegin
\exc{dbopen,'TD1\_SPEC'}{;Open spectrophotometric catalog}
\exc{dbhelp,'FLUX'}{;Read how to create wavelength array}
\exc{w = [1360.\ + findgen(60)*20., 2740.]}{;Wavelength array is 
1360 \AA ~-- 2540 \AA ~at 20 \AA ~resolution plus 2740 \AA ~photometer}
\exc{list = dbfind('bs\_no = 5191')}{;Find entry number for $\eta$ Uma}
\exc{dbext,list,'flux',f}{;Extract 61 element flux vector}
\exc{plot,w,f}{;Plot flux vs. wavelength}
\exend
The TD1\_SPEC catalog does not include an item for the wavelength array
because it is the same for each entry; instead the help file for the
item FLUX tells how to construct the wavelength array in a single IDL
statement.  Note that if more than one entry number were supplied to DBEXT,
then the output flux vector would be 2 dimensional, with the first dimension
containing the 61 flux values for a particular entry. 

The DBEXT command can be combined with the WHERE function of IDL to 
perform searches on fields not directly present in the catalogue.
As an example, we will use the IRAS point source catalog to search
for infrared selected high luminosity galaxy and quasar candidates (AGNs).  

Following Low \ea (1988)
(\apjl, {\bf 327}, L41) we set our search criteria to be (1) a 25 to 60
micron flux ratio of $0.25 <$ F25/F60 $< 3$,
(2) a galactic latitude greater than 30\de, and (3) no previous
identification from a stellar catalog.  (As a prerequisite for criterion
(1), the sources must have measurable fluxes at 25 and 60 microns.)
Search criterion (1) cannot be performed using DBFIND since F25/F60 is
not an item in the database.  Instead one must extract the F25 and F60
vectors with DBEXT, form the ratio, and use the WHERE function to 
select the desired
range.  Similarly, since the catalogue does not include galactic coordinates,
one must DBEXT the RA and DEC, and then convert to galactic coordinates. 
Criterion (3) presents a minor problem because DBFIND does not allow
a ``not equal to" search criterion; however, one can explicitly search
on the catalog identifications (item IDTYPE) that are not stellar
(IDTYPE = 2).

\exbegin
\exc{dbopen,`iras\_psc'}{;Open IRAS point source catalog}
\exc{list = dbfind(`idtype =[0,1,3,4], 25\_fqual>2, 60\_fqual>2')}
{;Not in stellar catalogs, detected at 25 and 60 microns}
\exc{dbext,list,`60\_flux,25\_flux',f60,f25}{;Extract 25 and 
60 micron flux vectors}
\exc{ratio = f25/f60}{;Form 25 to 60 micron flux ratio}
\exc{list = list(where ( (ratio gt 0.25) and (ratio lt 3.0) ))}{;Ratio
to select for AGNs}
\exc{dbext,list,`ra,dec',ra,dec}{;Extract RA and DEC vectors}
\exc{euler,ra*15.,dec,l,b,1}{;Convert to galactic coordinates}
\exc{list = list(where ( (b gt 30) or (b lt -30)) )}{;Select high galactic
latitude objects}
\exc{dbprint,list,`name,b\_mag, 12\_flux,25\_flux,60\_flux,100\_flux}{;Print 
name and IRAS fluxes of selected sources}
\exend

The above sequence of commands above will run slowly
because the 25\_fqual, 60\_fqual items are not indexed,
and because not all the desired search items (e.g.\ galactic coordinates)
are in the database.   Should similar searches of the IRAS catalog be required
often, 
then the database manager should include and index the desired items.
\section{Advanced Databasing}
\subsection{DBCIRCLE}
DBCIRCLE can be used to search a catalog for all sources within a specified
radius of a given position.  For example, suppose one wants to determine
if any quasars are within the $20'$ UIT field of the star AE Aqr (JOTFID number 3226).
\exbegin
\exc{dbopen,'PTL'}{;Open the ASTRO Program Target List}
\exc{l = dbfind('jotfid=3226')}{;Get entry number of JOTFID 3226}
\exc{dbext,l,'ra,dec',ra,dec}{;Extract the Ra and Dec of this star}
\exc{dbopen,'quasars'}{;Open Hewitt and Burbidge (1989) Quasar catalog}
\exc{list = dbcircle(ra,dec,20,dis)}{;Find sources within $20'$ of given RA and
Dec}
\exend
DBCIRCLE will display that 3 entries were found in the quasar catalog, and
place the entry values in the vector {\tt list}.
The vector {\tt dis} contains the distance (in arc minutes) of each quasar 
found to the specified field center.   

\subsection{DB\_OR}
The DB\_OR function concatenates the entries found in two different
lists, while removing duplicates.
For example, suppose one wants to identify the Astro targets that are 
{\em either} WUPPE or HUT targets. The command
\exone{list = dbfind('H=H,W=W')}
will identify targets that belong to  {\em both}  HUT and WUPPE.
To find targets that belong to either instruments, one must perform
two searches and concatenate the results.
\exbegin
\exc{list1 = dbfind(`H=H')}{;Get entry numbers of HUT targets}
\exc{list2 = dbfind(`W=W')}{;Get entry numbers of WUPPE targets}
\exc{list = db\_or(list1,list2)}{;Combine entry vectors and remove
duplicates}
\exend
Of course, IDL allows one to combine the three steps above into a single step:
\exone{list = db\_or( dbfind(`H=H'), dbfind(`W=W') )}
\subsection{Sorting}
Up to this point all results were printed in the order stored in the database
(by entry number).  The procedure DBSORT will sort an entry list on up to nine
sort items.  Its calling sequence is  
\exone{SORTLIST = DBSORT(LIST,`item1,item2 \ldots')}
where LIST is the input list of entry numbers and SORTLIST is the sorted
list.   Item1 is the primary sort item, item2 the secondary, and so on.
For example, the following statements will produce a printout of all 
IUE high-dispersion observations of the nuclei of of planetary nebulae,
(object class 70)
sorted by right ascension:
\exbegin
\exc{dbopen,`IUE'}{;Open the IUE catalogue}
\exc{list = dbfind(`obj\_class = 70,disp=h')}{;Specify object class, dispersion
mode}
\exc{sortlist = dbsort(list,`ra,image')}{;Primary sort is by RA, secondary by
image number}
\exc{dbprint,sortlist, `object,ra\_1950,dec\_1950,cam\_no,image'}{;Print
selected items}
\exend
\subsection{DBGET}
Suppose one has a list of five IUE SWP images, and wishes to obtain information
about the observational parameters.  DBFIND can be used to find the entry
numbers
\exone{list = dbfind(`cam\_no=3,image=[3427,15191,20227,29992,30022]')}
The SWP camera is camera number 3, and the individual images are identified.
For a larger number of images, however, this use of DBFIND breaks down.
It is awkward to write each image value in a string, and, in fact, DBFIND
can only parse 10 individual values.   What is needed is a function that
can search on values in an IDL vector, and this is why DBGET was created.
\exbegin
\exc{images = [3427,15191,20227,29992,30022]}{;Values are in an IDL vector}
\exc{list = dbfind(`cam\_no=3')}{;Restrict search to SWP camera}
\exc{list = dbget(`IMAGE',images,list)}{;Search on ``images'' vector}
\exend
One limitation of DBGET is that it can only be used with one item at
a time.  Be aware that the number of entries returned by DBGET might
not equal the number of values in the search vector; if, for example,
an image number is missing, or appears twice (e.g.\ as both large 
and small aperture).   The function DBMATCH should be used if a one-to-one 
correspondence is desired between the elements of the search item vector
and the found entries.
\subsection{DBMATCH} 
    Suppose one wants to find the Gliese catalog number of every star in the
Yale Bright Star catalog.  Both these catalogs contain an HD item, so one
can extract the HD numbers from the Yale Bright Star Catalog, and then use
this vector to search for entries in the Gliese catalog.      
\exbegin
\exc{dbopen,`YALE\_BS'}{;Open the Yale Bright Star Catalog}
\exc{dbext,-1,`HD\_NO',hd}{;Extract the HD number for all stars}
\exc{dbopen,`GLIESE'}{;Open the Gliese Catalog of Nearby Stars}
\exc{gl = dbmatch(`HD\_NO',hd)}{;Find Gliese numbers of specified HD numbers}
\exend
The output vector {\tt gl} will contain 9110 elements -- one for each entry
in the Yale Bright Star catalog.  Stars not in the Gliese catalog will contain
a value of 0 in the {\tt gl} vector.   DBGET could be used 
to find all the Gliese numbers of stars in the Yale Bright Star catalog,
but it would not keep track of which Gliese number went with which star.
DBMATCH is slower than either DBFIND or DBGET because it must loop over
each element of the item search vector.  However, DBMATCH is very useful
for building a ``pointer'' from one catalog to another.
\subsection{Pointers}
It often happens that the entries in two different catalogues can
refer to the same object.   It is then possible to open both catalogues
simultaneously, and for the entry in catalogue 1 to ``point'' to the
entry in catalogue 2 corresponding to the same object.  The user can
then print, or search on, items from either catalogue.
For example, suppose one wants a printout of comments that have been
written about the ASTRO targets, along with the names of the instrument(s)
associated with each target.  The comments are given in the PTLCOM database,
while the instruments are given in the PTL database.
\exbegin
\exc{dbopen,'PTL,PTLCOM'}{;Open both the PTL and the PTLCOM databases}
\exc{list = dbfind('flag\_com')} {;Item is non-zero when comments exist}
\exc{dbprint,list,'jotfid,id1,h,w,u,b,lcomm'}{;Print instrument and
comments}
\exend
Use DBHELP to learn if one catalogue points to any others.  
You {\em cannot} simultaneously open databases which do not have pointers
already built in by the database manager.
\section{Creating and Modifying a Database}
\subsection{Introduction}
A database actually consists of four disk files, each identified
with a unique 3 letter extension.  For example, the PTL database
consists of the four files, ptl.dbd, ptl.dbh, ptl.dbf, and ptl.dbx.
The .dbd file is an ASCII file that contains all the item definitions,
print formats, pointers etc.  The .dbh file contains a list of all items
and item titles stored in binary format for quick access.  The .dbf
file contains all the data stored in binary in entry order.  Finally,
the .dbx file contain the values of all the indexed items stored in
binary in item order.   The table below summarizes the four database