database.tex

basic median filter simulation

files.       \\

\vspace{0.1in}
\begin{tabular}{|l|l|l|l|}  \hline
.dbf & DataBase File & Binary & Row-ordered data \\ \hline
.dbh & DataBase Contents & Binary & Title and item descriptions \\ \hline
.dbx & DataBase indeX & Binary & Indexed and sorted data  \\  \hline
.dbd & DataBase Definition & ASCII & User-supplied item characteristics \\
\hline
\end{tabular}
\vspace{0.2in}

In addition to these four files, the user can create ASCII help files as
necessary.  For example, the file YALE\_BS.HLP will be printed when no database
has been opened and the user types {\tt dbhelp,'yale\_bs'}.  The file 
YALE\_BS\_DBLE\_NAME.HLP will be printed if the YALE\_BS database has been 
opened, and the user types {\tt DBHELP,`DBLE\_NAME'}.     A final optional
file for a database is an ASCII file with the extension .ITEMS.     This
file lists the default print items (one per row) to  be used when the
user types {\tt DBPRINT,LIST}.


The steps a user must follow to create a database are as follows:
\begin{itemize}
\item Define the logical name (VMS) or environment variable (UNIX) 
ZDBASE to point to the directories containing
the catalogues.   To identify where existing database files are located, 
type {\tt \$sho logical zdbase} (VMS) or {\tt printenv ZDBASE} (Unix).
\item Create a database definition .dbd file using a text editor
\item Use DBCREATE to create the .dbh file, and empty versions of the .dbx
and .dbf files
\item Fill the .dbx and .dbf files with data usually using the DBBUILD
procedure.  Alternatively, entries can be written one at a time into the
.dbf file with DBWRT, and the .dbx file can filled using DBINDEX.
\end{itemize}
\subsection{The .dbd file}
The critical step in creating a database is making the database description
(.dbd) file.  Reproduced in full below is a .dbd file for a data base that will
be called EXAMPLE.
\begin{center}
{\bf example.dbd} \\
************************************
\end{center}
\begin{tabbing}
FLUX(61)XX \= I*2XXX \= Root   \kill
\#title     \\
Example of a Database catalogue \\
\>  \>  \\ 
\#maxentries   \\
1356            \\
\>  \>  \\ 
\#items       \\
CAT\_NO    \>   I*2  \>   Catalogue Number  \\
BS\_NO     \>   I*4  \>   Bright Star Number \\
RA\_2000   \>   C*10 \>   RA (2000)... Use RA for search  \\
DEC\_2000  \>   C*9  \>   Dec (2000)... Use Dec for search  \\
FLUX(61)   \>   R*4  \>   Flux (x 10[-10]), 1380A - 2740A) \\
RA         \>   R*4  \>   RA (J2000, hours)...Use for Search          \\
DEC        \>   R*4  \>   Dec (J2000, degrees)...Use for Search       \\
 \> \> \\
\#formats       \\
ENTRY       \>   I6   \>   Entry               \\
CAT\_NO     \>   I4   \>   TD1,No             \\
BS\_NO      \>   I6   \>   Bright,Star,No    \\
RA\_2000   \>   A10  \>   RA,(2000)          \\
DEC\_2000   \>   A10  \>   Dec,(2000)         \\
 \> \> \\
\#index  \\
CAT\_NO    \>     index  \\
BS\_NO     \>     sort/index       \\
RA         \>     sorted      \\
DEC        \>     sort        \\
 \> \> \\
\#pointers                  \\
BS\_NO     \>      yale\_bs    \\ 
\end{tabbing}
\begin{center}
************************************
\end{center}

A .dbd file contains several ``block'' headers identifiable by a 
preceding pound sign ``\#''.  The \#title and \#items blocks are
required, the \#maxentries and \#formats blocks are strongly recommended,
while the \#index and \#pointers blocks are optional.
\begin{description}
\item [\#title] Underneath the \#title header should be a one line description
(50 characters or less) of the database that will be displayed with
DBHELP.  The actual name of the database (to be used with DBOPEN) is the
same as the name of the .dbd file.   
\item [\#maxentries] Underneath the \#maxentries header should be a single
number giving the maximum number of entries one expects to be in the database.
The only cost in making the value of \#maxentries too large is that extra
disk space will have to be allocated for the index files.   On the other
hand, if the value of \#maxentries is less than the actual number of entries,   
then you will not be able to create the index file.
\item [\#items]  This required block contains three columns of information.
The first column contains the name of every item.  Multiple valued items
should have the number of values per entry put in parentheses next to the
item name.
The second column
gives the datatype of every item.  Acceptable values of datatype include
R*4, I*2, I*4, R*8, L*1 or B*1.   Since IDL V5.2 64 bit integers (I*8) and
unsigned integers (U*2, U*4, U*8) are also allowed datatypes. The datatype 
of a string item should be
written as C*[n] where [n] is the string length.  The last column
gives a brief description of the item that will be used with DBHELP.
\item [\#formats] This block lists the item name, print format, and print
heading.  If an item is not listed here, then it is given the default
IDL print format for its datatype (e.g.\ I7 for I*2 data), and the item
name is used for the print heading.  Each print heading consist of three
rows, so that a heading can consist of up to three words separated by
commas.   However, each word must fit into the space allocated by the
print format; e.g. a heading for the item BS\_NO (format I6) will be
truncated after 6 characters.
\item [\#index]  
This block lists the indexed items and their index types.    
There are four acceptable index types; ``index'', ``sorted'',
``sort'', and ``sort/index''.   
The values of an ``index'' item are copied in 
entry order to the index file; this allows the values to be extracted
quickly.   The values of a ``sort'' item are copied to the index file
in numeric order (i.e sorted), along with a lookup table relating 
the numeric order to the entry order.  ``Sort'' items can therefore be
searched very quickly.  `Sorted' items are assumed to have entry order
already coincident with numeric order, and the ``sorted'' index should be
used whenever it is allowed. For example, some catalogs are
listed by increasing right ascension, which would allow RA to be a
``sorted'' item.  Neither multiple-valued items nor string items are allowed 
an index type of ``sort'' or
``sorted''.  Finally, the values of a ``sort/index'' item are copied in both
entry order and numeric order to the index file.  This type is mainly used
for items which are used to ``point'' at another database (see below).
The following table summarizes the relative disk space and search
speed of the different index types. 
\vspace{0.1in}
\begin{tabular}{|l|l|l|l|l|} \hline
Index  & Disk  & Search & Extraction & Comment \\ 
Type & Space  & Speed & Speed & \\ \hline\hline
None & 1 &  Slow & Slow & \\ \hline
Index & 2 & Moderate & Fast & \\ \hline
Sorted & 3 & Very Fast & Fast & Item must be in entry order \\ \hline 
Sort & 4 & Fast & Slow & Multiple and String items not allowed \\   \hline
Sort/Index & 5 & Fast & Fast & Multiple and String items not allowed \\ \hline
\end{tabular}
\vspace{0.1in}
\item [\#pointers] The \#pointers block contains the names of items
that point to the entry numbers of another catalogue.  In our example,
BS\_NO points to the entry number in the Yale Bright Star catalogue.
Item that serve as pointers must be either index type ``index'',
``sort/index'' or the item ENTRY.

The database software has no features specific to astronomy.    However,
in order that general position-search software can be written, it is useful
to have an established convention as to the labeling of right ascension and
declination.   This convention is that right ascension should be
stored in hours in the (searchable) item RA, and declination stored in
degrees in the item DEC.     The item description should include the 
equinox of the coordinates (e.g.\ J2000 or B1950).
 
\end{description}
\subsection{Adding or Modifying Data}

Once a database description (.dbd) file has been created, adding and 
modifying data is relatively easy.   The procedure DBCREATE is used
to create the contents (.dbh) file, and optionally, new copies of the 
data (.dbf) and index (.dbx) file.  Its basic calling sequence is
\exone{DBCREATE,`$<$DATABASE\_NAME$>$',[NEWINDEX,NEWDB]}
where NEWINDEX is non-zero to create a new .dbx file, and NEWDB is non-zero
to create a new .dbf file.  DBCREATE requires that the user set the system
variable !PRIV=2; this is to prevent novice users from accidently corrupting
the database.

It assumed that the user has been able to read the data into IDL vectors.
(The procedures READCOL, RDFLOAT, and READFMT are extremely useful for 
reading raw data from ASCII files into IDL vectors.)   For example, to create 
the EXAMPLE database from the previous section, the user should have vectors
named, say, CAT,BS,RA\_2000,DEC\_2000,FLUX,RA, and DEC corresponding to 
each of the items.   The database must then be opened for update by
adding a second parameter to the DBOPEN command.  It is also necessary
for the user to have sole access to the database; an error message will
result if one tries to update a database while another user is reading it.
Finally, the database is loaded with the procedure
DBBUILD as follows:
\exbegin
\exc{!PRIV=2}{;Set !PRIV to create or modify database files}
\exc{dbcreate,`EXAMPLE',1,1}{;Need new index and data files}
\exc{dbopen,`EXAMPLE',1}{;Open the database for update}
\exc{dbbuild,cat,bs,ra\_2000,dec\_2000,flux, ra,dec}{;Load IDL vectors into
database}
\exend
If {\tt DBCREATE} is supplied with an /EXTERNAL keyword then the data
and index files are written in IEEE format, and the database can be
shared by machines with different architectures (e.g.\ Suns and Vaxes).
However, there is a substantial overhead for the use /EXTERNAL format 
on machines which do not internally use the IEEE format.

To modify the item titles or print formats in
an existing database, one simply edits the .dbd file with the new information,
and then types {\tt DBCREATE}.   There is no need to create new .dbx or .dbf
files.  If, however, one wants to change the index type of an item or 
set of items, then a new .dbx file must be created and built with DBINDEX.
\exbegin
\exc{!PRIV=2}{;As always}
\exc{dbcreate,'EXAMPLE',1}{;Create a new index file}
\exc{dbopen,'EXAMPLE',1}{;Open database for update} 
\exc{dbindex}{;Make the index file}
\exend
There are several ways to append or modify the actual data in a database.
To append new entries, the DBBUILD command can again be used, but without
first calling DBCREATE, since the data in the existing .dbx and .dbf files
must remain.  The procedure DBUPDATE can be used to load new item values into
a database.  DBUPDATE can be viewed as the inverse of DBEXT -- instead of
extracting item vectors, it will insert them.  For example, suppose the RA and
DEC items are in 1950 equinox, and one wants to convert them to 2000 equinox.
\exbegin
\exc{!PRIV=2 \& dbopen,'EXAMPLE',1}{;Ultimately, will update database}
\exc{dbext,-1,'RA,DEC',ra,dec}{;Extract RA and Dec vectors}
\exc{ra = ra*15.}{;Convert to degrees}
\exc{precess,ra,dec,1950,2000}{;Convert to 2000 equinox}
\exc{dbupdate,-1,'RA,DEC',ra/15.,dec}{;Load new values of RA and Dec} 

Finally, the procedure DBEDIT is useful for editing individual item
values.  For example, suppose a database has values of V\_MAG set to 
99.9 whenever the visual magnitude was unknown.  Once these values become
known they can be inserted by hand into the database:
\exbegin
\exc{!PRIV=2 \& dbopen,'EXAMPLE',1}{;Open for update}
\exc{list = dbfind('V\_MAG=99.9')}{;Get entry numbers with bad V\_MAG}
\exc{dbedit,list,'V\_MAG'}{Interactive editing of selected entries}
\exend
DBEDIT will display the existing value of an item, and prompt the user
whether to keep or replace it.

\newpage
\section{APPENDIX:  ONLINE CATALOGS}
This appendix lists the IDL databases available at 
{\tt http://idlastro.gsfc.nasa.gov/ftp/zdbase/}.  Use
DBHELP to obtain a more complete and up-to-date listing.at your local
installation.   

\vspace{0.15in}

\begin{tabular}{lll}
 General Catalogues and Databases & \# of Entries & Name       \\
         &          &        \\

 Abell Rich Clusters of Galaxies & 5250 & ABELL  \\
 EUVE 2nd Source List (1996) & 514 & EUVE \\
 Gliese Catalog of Nearby Stars & 3,802 & GLIESE \\
 ~~Preliminary 3rd Edition (1991) & & \\
  Hipparcos Catalog (1997) & 118218 & HIPPARCOS \\
 Hipparcos Input Catalog (1992) & 118218 & HIC \\
 Catalogue of HST observations (2000) & 234,069 & HST\_CATALOG \\
 {\em IRAS} Point Source Catalogue & 255,578 & IRAS\_PSC \\
 {\em IUE} Merged Log Catalogue (1997) & 116,736 & IUE           \\ 
 Nearby Galaxies Tully (1988) & 2367 & NEARBY\_GAL  \\
 New General Catalog (NGC) objects (1988) & 13,226 & NGC2000 \\
 Position and Proper Motion (PPM) catalog & 468,907 & PPM \\
 Catalogue of Principal Galaxies (Paturel et al. 1989) & 73,197 & PRIN\_GAL \\
 Quasars and AGN $10^{th}$ ed. & 30,119 & QUASAR\_AGN \\
 ~~Veron-Cetty and Veron (2001) & & \\
 Catalog of Quasars and BL Lac Objects & 7,315 & QUASARS \\
 ~~Hewitt and Burbidge (1993) \\
 3rd Reference Catalog of Galaxies  & 23,022 & RC3 \\
  CFA Redshift Survey (1995) & 57,536 & REDSHIFT            \\
  Catalogue of White Dwarfs 4th ed   & 4,401 & WDWARF \\
 ~~McCook and Sion (1998) & & \\
 Yale Bright Star Catalogue & 9,110 & YALE\_BS        \\  
 ~~Preliminary 5th Edition (1991) & & \\

\end{tabular}
                                                      
\end{document}