complextype.html

一个开源的网页爬虫一个开源的网页爬虫一个开源的网页爬虫一个开源的网页爬虫一个开源的网页爬虫一个开源的网页爬虫

</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/deciderules/TooManyPathSegmentsDecideRule.html" title="class in org.archive.crawler.deciderules">TooManyPathSegmentsDecideRule</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Rule REJECTs any CrawlURIs whose total number of path-segments (as
 indicated by the count of '/' characters not including the first '//')
 is over a given threshold.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/deciderules/TransclusionDecideRule.html" title="class in org.archive.crawler.deciderules">TransclusionDecideRule</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Rule ACCEPTs any CrawlURIs whose path-from-seed ('hopsPath' -- see
 <A HREF="../../../../../org/archive/crawler/datamodel/CandidateURI.html#getPathFromSeed()"><CODE>CandidateURI.getPathFromSeed()</CODE></A>) ends 
 with at least one, but not more than, the given number of 
 non-navlink ('L') hops.</TD>
</TR>
</TABLE>
&nbsp;
<P>
<A NAME="org.archive.crawler.extractor"><!-- --></A>
<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#CCCCFF" CLASS="TableHeadingColor">
<TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2">
Uses of <A HREF="../../../../../org/archive/crawler/settings/ComplexType.html" title="class in org.archive.crawler.settings">ComplexType</A> in <A HREF="../../../../../org/archive/crawler/extractor/package-summary.html">org.archive.crawler.extractor</A></FONT></TH>
</TR>
</TABLE>
&nbsp;
<P>

<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#CCCCFF" CLASS="TableSubHeadingColor">
<TH ALIGN="left" COLSPAN="2">Subclasses of <A HREF="../../../../../org/archive/crawler/settings/ComplexType.html" title="class in org.archive.crawler.settings">ComplexType</A> in <A HREF="../../../../../org/archive/crawler/extractor/package-summary.html">org.archive.crawler.extractor</A></FONT></TH>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/AggressiveExtractorHTML.html" title="class in org.archive.crawler.extractor">AggressiveExtractorHTML</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Extended version of ExtractorHTML with more aggressive javascript link
 extraction where javascript code is parsed first with general HTML tags
 regexp, and than by javascript speculative link regexp.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ChangeEvaluator.html" title="class in org.archive.crawler.extractor">ChangeEvaluator</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This processor compares the CrawlURI's current 
 <A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html#getContentDigest()"><CODE>content digest</CODE></A>
 with the one from a previous crawl.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/Extractor.html" title="class in org.archive.crawler.extractor">Extractor</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Convenience shared superclass for Extractor Processors.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorCSS.html" title="class in org.archive.crawler.extractor">ExtractorCSS</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This extractor is parsing URIs from CSS type files.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorDOC.html" title="class in org.archive.crawler.extractor">ExtractorDOC</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;This class allows the caller to extract href style links from word97-format word documents.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html" title="class in org.archive.crawler.extractor">ExtractorHTML</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Basic link-extraction, from an HTML content-body,
 using regular expressions.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTTP.html" title="class in org.archive.crawler.extractor">ExtractorHTTP</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Extracts URIs from HTTP response headers.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorImpliedURI.html" title="class in org.archive.crawler.extractor">ExtractorImpliedURI</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;An extractor for finding 'implied' URIs inside other URIs.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorJS.html" title="class in org.archive.crawler.extractor">ExtractorJS</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Processes Javascript files for strings that are likely to be
 crawlable URIs.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorPDF.html" title="class in org.archive.crawler.extractor">ExtractorPDF</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Allows the caller to process a CrawlURI representing a PDF
  for the purpose of extracting URIs</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorSWF.html" title="class in org.archive.crawler.extractor">ExtractorSWF</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Extracts URIs from SWF (flash/shockwave) files.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorUniversal.html" title="class in org.archive.crawler.extractor">ExtractorUniversal</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;A last ditch extractor that will look at the raw byte code and try to extract
 anything that <i>looks</i> like a link.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorURI.html" title="class in org.archive.crawler.extractor">ExtractorURI</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;An extractor for finding URIs inside other URIs.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorXML.html" title="class in org.archive.crawler.extractor">ExtractorXML</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;A simple extractor which finds HTTP URIs inside XML/RSS files,
 inside attribute values and simple elements (those with only
 whitespace + HTTP URI + whitespace as contents)</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/extractor/HTTPContentDigest.html" title="class in org.archive.crawler.extractor">HTTPContentDigest</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;A processor for calculating custum HTTP content digests in place of the 
 default (if any) computed by the HTTP fetcher processors.</TD>
</TR>
</TABLE>
&nbsp;
<P>
<A NAME="org.archive.crawler.fetcher"><!-- --></A>
<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#CCCCFF" CLASS="TableHeadingColor">
<TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2">
Uses of <A HREF="../../../../../org/archive/crawler/settings/ComplexType.html" title="class in org.archive.crawler.settings">ComplexType</A> in <A HREF="../../../../../org/archive/crawler/fetcher/package-summary.html">org.archive.crawler.fetcher</A></FONT></TH>
</TR>
</TABLE>
&nbsp;
<P>

<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#CCCCFF" CLASS="TableSubHeadingColor">
<TH ALIGN="left" COLSPAN="2">Subclasses of <A HREF="../../../../../org/archive/crawler/settings/ComplexType.html" title="class in org.archive.crawler.settings">ComplexType</A> in <A HREF="../../../../../org/archive/crawler/fetcher/package-summary.html">org.archive.crawler.fetcher</A></FONT></TH>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/fetcher/FetchDNS.html" title="class in org.archive.crawler.fetcher">FetchDNS</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Processor to resolve 'dns:' URIs.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/fetcher/FetchFTP.html" title="class in org.archive.crawler.fetcher">FetchFTP</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Fetches documents and directory listings using FTP.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;class</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../../org/archive/crawler/fetcher/FetchHTTP.html" title="class in org.archive.crawler.fetcher">FetchHTTP</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HTTP fetcher that uses <a
 href="http://jakarta.apache.org/commons/httpclient/">Apache Jakarta Commons
 HttpClient</a> library.</TD>
</TR>
</TABLE>
&nbsp;
<P>
<A NAME="org.archive.crawler.filter"><!-- --></A>
<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#CCCCFF" CLASS="TableHeadingColor">
<TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2">
Uses of <A HREF="../../../../../org/archive/crawler/settings/ComplexType.html" title="class in org.archive.crawler.settings">ComplexType</A> in <A HREF="../../../../../org/archive/crawler/filter/package-summary.html">org.archive.crawler.filter</A></FONT></TH>
</TR>
</TABLE>
&nbsp;
<P>

<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#CCCCFF" CLASS="TableSubHeadingColor">
<TH ALIGN="left" COLSPAN="2">Subclasses of <A HREF="../../../../../org/archive/crawler/settings/ComplexType.html" title="class in org.archive.crawler.settings">ComplexType</A> in <A HREF="../../../../../org/archive/crawler/filter/package-summary.html">org.archive.crawler.filter</A></FONT></TH>