ch11.htm

CGI programming is the hottest stuff to look out for in this book

<H2><A NAME="CGIWorkintheBackground"><FONT SIZE=5 COLOR=#FF0000>CGI
Work in the Background</FONT></A></H2>
<P>
A search engine, in fact, is made of lots of different programs,
each one accomplishing a different task:
<UL>
<LI><FONT COLOR=#000000>An information gatherer (either a robot
or a Web interface to receive URLs given by the users)</FONT>
<LI><FONT COLOR=#000000>An index creator or information organizer
to catalog information</FONT>
<LI><FONT COLOR=#000000>A Web interface to permit information
retrieval</FONT>
</UL>
<P>
The robot and the index creator or organizer can be independent
programs that either speak the HTTP protocol with Web servers
around the world and/or catalog information on local disks. On
the other hand, Web interfaces are coupled with CGI applications
that process users' input. A Web interface for URL additions must
get data about the URL submitted by a user and pass it to a program
that will either insert it immediately in the database or put
it in a queue for later processing by a human or a URL-verifier
robot. A Web search interface passes its input to a CGI application
that searches the database and sends results back to the user.
It is the application parameters-and, at the origin, fields on
the HTML form-that define which information will appear on-screen.
A URL containing the application call with the different parameters
is usually found in the top of a results page:
<BLOCKQUOTE>
<TT><FONT FACE="Courier"><A HREF="http://www.lycos.com/cgi-bin/pursuit?query=sams&amp;ab=the_catalog">http://www.lycos.com/cgi-bin/pursuit?query=sams&amp;ab=the_catalog</A></FONT></TT>
</BLOCKQUOTE>
<P>
This URL is a CGI program call and indicates Lycos to search for
&quot;sams&quot; in &quot;the catalog,&quot; one of Lycos' databases.
On Infoseek, the CGI call is quite similar (just ignore the parameters
you don't understand):
<BLOCKQUOTE>
<TT><FONT FACE="Courier"><A HREF="http://guide-p.infoseek.com/Titles?qt=sams&amp;col=WW&amp;sv=IS&amp;lk=frames">http://guide-p.infoseek.com/Titles?qt=sams&amp;col=WW&amp;sv=IS&amp;lk=frames</A></FONT></TT>
</BLOCKQUOTE>
<P>
When you click on the Submit button on a search form, you are
actually sending your query to the CGI script, either by using
the <TT><FONT FACE="Courier">POST</FONT></TT> or the <TT><FONT FACE="Courier">GET</FONT></TT>
method. Because no updating to the database of URLs will happen
when you submit your query, the <TT><FONT FACE="Courier">GET</FONT></TT>
method is generally recommended. <TT><FONT FACE="Courier">POST</FONT></TT>
method submissions are generally reserved for long submissions
(with many fields) or for submissions that may alter data on the
server. Both methods, however, can be used.
<P>
On Lycos, the method used is <TT><FONT FACE="Courier">GET</FONT></TT>:
<BLOCKQUOTE>
<TT><FONT FACE="Courier">&lt;form action=&quot;http://www.lycos.com/cgi-bin/pursuit&quot;
method=GET&gt;</FONT></TT>
</BLOCKQUOTE>
<P>
On Excite, the method used is <TT><FONT FACE="Courier">POST</FONT></TT>:
<BLOCKQUOTE>
<TT><FONT FACE="Courier">&lt;FORM ACTION=&quot;http://www.excite.com/search.gw&quot;
METHOD=POST&gt;</FONT></TT>
</BLOCKQUOTE>
<P>
Your query is received by a CGI application that is responsible
for either finding the relevant information or passing the arguments
to another custom application on the server that will do this
task. This application could be, for example, a relational database
gateway or query application. Finally, the result is sent back
and displayed on your browser's window. The simplicity of this
process hides the power associated with a search engine. Behind
the scenes, powerful hardware and software work to find and classify
information on an index database, related to the words you submitted
(your query). In Altavista, for example, a set of three Alpha
Servers with 6 GB RAM and 210 GB hard disk are able to search
the current 40 GB database in less than a second! And all this
power is available to you from a simple Web page.
<P>
Special care should be taken on the search algorithm if you plan
to develop your own algorithm on a custom index database and specially
if you plan to make it available for everyone on the Internet.
Your server could get many hits per day, and the resources used
by one invocation of the application are multiplied by the number
of users submitting queries. This can rapidly bring your actual
server to its knees.
<H2><A NAME="DevelopingaSimpleCGIforaWhitePage"><FONT SIZE=5 COLOR=#FF0000>Developing
a Simple CGI for a White Pages Database</FONT></A></H2>
<P>
An electronic White Pages database is an organized list containing
e-mail addresses. There is no list containing all the e-mail addresses
valid in the Internet, but there are already some lists that contain
many e-mail addresses. We will present here a CGI application
in Perl that offers users a search interface on the e-mail addresses
list. See Listing 11.1 for the source of this script.
<HR>
<BLOCKQUOTE>
<B>Listing 11.1. The White Pages application (a CGI Perl script).
<BR>
</B>
</BLOCKQUOTE>
<BLOCKQUOTE>
<TT><FONT FACE="Courier">#!/usr/bin/perl<BR>
<BR>
###########################################################################
<BR>
# wp.pl 1.0 - White Pages search script&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#<BR>
#&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#<BR>
# Antonio 
Ferreira&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#
<BR>
# 
[e-mail]&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp
;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#<BR>
#&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#<BR>
# April 
1996&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#
<BR>
###########################################################################
<BR>
<BR>
require '/usr/lib/cgi-lib.pl';<BR>
<BR>
######################### Variables ########################<BR>
$url = 'http://www.your_domain.com/cgi-bin/wp.pl';&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# White Pages URL<BR>
$pathBackground = '/bg.gif';<BR>
$cat = '/usr/bin/cat';<BR>
$tr = '/usr/bin/tr';<BR>
$grep = '/usr/bin/grep';<BR>
$email_list = '/usr/local/WWW/Docs/WP/email.list';<BR>
<BR>
########################## Start of main program ##########################
<BR>
<BR>
&amp;ReadParse(*input);&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# field=value<BR>
print &amp;PrintHeader();&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# Content-type: text/html\n\n<BR>
<BR>
if (&amp;MethGet() || defined $input{'goback.x'}) {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#
GET<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&amp;InitialForm();&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
# ... initial form<BR>
} else {&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#
POST ... other options<BR>
&nbsp;&nbsp;&nbsp;&nbsp;if (defined $input{'addForm.x'}) {<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&amp;AddForm();
<BR>
&nbsp;&nbsp;&nbsp;&nbsp;} elsif (defined $input{'addEmail.x'})
{<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&amp;AddEmail();
<BR>
&nbsp;&nbsp;&nbsp;&nbsp;} elsif (defined $input{'help.x'}) {<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&amp;Help();<BR>
&nbsp;&nbsp;&nbsp;&nbsp;} else {<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&amp;Search();
<BR>
&nbsp;&nbsp;&nbsp;&nbsp;}<BR>
}<BR>
<BR>
exit(0);<BR>
<BR>
########################## End of main program ##########################
<BR>
<BR>
#################### Subroutines ###################<BR>
<BR>
##### Initial search form #####<BR>
sub InitialForm {<BR>
&nbsp;&nbsp;&nbsp;&nbsp;print &lt;&lt;EOM;<BR>
&lt;HTML&gt;<BR>
&lt;HEAD&gt;<BR>
&lt;TITLE&gt;White Pages&lt;/TITLE&gt;<BR>
&lt;!-- (c) Esoterica 1996, [e-mail] --<BR>
&lt;/HEAD&gt;<BR>
&lt;BODY BACKGROUND= &quot;$pathBackground&quot;&gt;<BR>
&lt;P ALIGN=center&gt;&lt;IMG SRC=&quot;/Images/WP/wp.gif&quot;
ALT=&quot;WHITE PAGES&quot; BORDER=0 WIDTH=319<BR>
&Acirc;HEIGHT=123&gt;&lt;/P&gt;<BR>
&lt;H3 ALIGN=center&gt;&lt;I&gt;The email directory!&lt;/I&gt;&lt;/H3&gt;
<BR>
&lt;P&gt;<BR>
&lt;FORM ACTION=&quot;$url&quot; METHOD=post&gt;<BR>
&lt;CENTER&gt;<BR>
&lt;B&gt;Search for:&lt;/B&gt; &lt;INPUT NAME=&quot;key&quot;
SIZE=30&gt; &lt;INPUT TYPE=submit NAME=search &Acirc;VALUE=&quot;Get
it&quot; ALIGN=top&gt;<BR>
&lt;P&gt;<BR>
&lt;UL&gt;<BR>
Please enter the name (or part of it) of the person you want to
find.<BR>
&lt;/UL&gt;<BR>
&lt;INPUT TYPE=image SRC=&quot;/Images/WP/addwp.gif&quot; NAME=addForm
BORDER=0&gt;<BR>
&lt;INPUT TYPE=image SRC=&quot;/Images/WP/helpwp.gif&quot; NAME=help
BORDER=0&gt;<BR>
&lt;IMG SRC=&quot;/Images/c_esot.gif&quot; ALIGN=right ALT=&quot;&quot;&gt;
<BR>
&lt;/CENTER&gt;<BR>
&lt;/FORM&gt;<BR>
&lt;/BODY&gt;<BR>
&lt;/HTML&gt;<BR>
EOM<BR>
}<BR>
<BR>
##### Form for email address addition&nbsp;&nbsp;#####<BR>
sub AddForm {<BR>
&nbsp;&nbsp;&nbsp;&nbsp;print &lt;&lt;EOM;<BR>
&lt;HTML&gt;<BR>
&lt;HEAD&gt;<BR>
&lt;TITLE&gt;Add an email address to the White Pages database&lt;/TITLE&gt;
<BR>
&lt;!-- (c) Esoterica 1996, [e-mail] --&gt;<BR>
&lt;/HEAD&gt;<BR>
<BR>
&lt;BODY BACKGROUND=$pathBackground&gt;<BR>
&lt;H1 ALIGN=center&gt;Add an email address to the White Pages
database&lt;/H1&gt;<BR>
&lt;P&gt;<BR>
&lt;FORM ACTION=&quot;$url&quot; METHOD=post&gt;<BR>
&lt;CENTER&gt;<BR>
&lt;PRE&gt;<BR>
&lt;B&gt;&nbsp;&nbsp;&nbsp;Name:&lt;/B&gt; &lt;INPUT NAME=&quot;name&quot;
SIZE=40&gt;<BR>
&lt;B&gt;Company:&lt;/B&gt; &lt;INPUT NAME=&quot;company&quot;
SIZE=40&gt;<BR>
&lt;B&gt;&nbsp;&nbsp;Email:&lt;/B&gt; &lt;INPUT NAME=&quot;email&quot;
SIZE=40&gt;<BR>
&lt;/PRE&gt;<BR>
&lt;P&gt;<BR>
&lt;INPUT TYPE=image SRC=&quot;/Images/WP/additwp.gif&quot; NAME=addEmail
BORDER=0&gt;<BR>
&lt;INPUT TYPE=image SRC=&quot;/Images/WP/retwp.gif&quot; NAME=goback
BORDER=0&gt;<BR>
&lt;/CENTER&gt;<BR>
&lt;/FORM&gt;<BR>
&lt;/BODY&gt;<BR>
&lt;/HTML&gt;<BR>
EOM<BR>
}<BR>
<BR>
##### Add email address to the list #####<BR>
sub AddEmail {<BR>
&nbsp;&nbsp;&nbsp;&nbsp;if ( index($input{'email'},'@') &gt;=
0 ) {<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ($input{'company'}
eq '') {<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$coment
= &quot;&gt;&quot;;<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} else {<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$coment
= &quot; - &quot;.$input{'company'}.&quot;&gt;&quot;;<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$line = $input{'email'}.&quot;
&lt;&quot;.$input{'nome'}.$coment;<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;open (LIST,&quot;&gt;&gt;$email_list&quot;);
<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print LIST (&quot;\n$line&quot;);
<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;close(LIST);<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print &lt;&lt;EOM;
<BR>
&lt;HTML&gt;<BR>
&lt;HEAD&gt;<BR>
&lt;TITLE&gt;Email address added&lt;/TITLE&gt;<BR>
&lt;!-- (c) Esoterica 1996, [e-mail] --&gt;<BR>
&lt;/HEAD&gt;<BR>
&lt;BODY BACKGROUND=&quot;$pathBackground&quot;&gt;<BR>
&lt;H1 ALIGN=center&gt;Email address added&lt;/H1&gt;<BR>
&lt;P&gt;<BR>
&lt;FORM ACTION=&quot;$url&quot; METHOD=post&gt;<BR>
Your email address was included in the White Pages database.<BR>
&lt;P&gt;<BR>