crawluri.html

用JAVA编写的,在做实验的时候留下来的,本来想删的,但是传上来,大家分享吧

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Perform usual extraction on a CrawlURI</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorDOC.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorDOC.html#extract(org.archive.crawler.datamodel.CrawlURI)">extract</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Processes a word document and extracts any hyperlinks from it.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorCSS.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorCSS.html#extract(org.archive.crawler.datamodel.CrawlURI)">extract</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected abstract &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>Extractor.</B><B><A HREF="../../../../../org/archive/crawler/extractor/Extractor.html#extract(org.archive.crawler.datamodel.CrawlURI)">extract</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#extract(org.archive.crawler.datamodel.CrawlURI)">extract</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>(package private) &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#extract(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence)">extract</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
        java.lang.CharSequence&nbsp;cs)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Run extractor.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorURI.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorURI.html#extractLink(org.archive.crawler.datamodel.CrawlURI, org.archive.crawler.extractor.Link)">extractLink</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
            <A HREF="../../../../../org/archive/crawler/extractor/Link.html" title="class in org.archive.crawler.extractor">Link</A>&nbsp;wref)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Consider a single Link for internal URIs</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>HTTPContentDigest.</B><B><A HREF="../../../../../org/archive/crawler/extractor/HTTPContentDigest.html#innerProcess(org.archive.crawler.datamodel.CrawlURI)">innerProcess</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTTP.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTTP.html#innerProcess(org.archive.crawler.datamodel.CrawlURI)">innerProcess</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ChangeEvaluator.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ChangeEvaluator.html#innerProcess(org.archive.crawler.datamodel.CrawlURI)">innerProcess</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;void</CODE></FONT></TD>
<TD><CODE><B>Extractor.</B><B><A HREF="../../../../../org/archive/crawler/extractor/Extractor.html#innerProcess(org.archive.crawler.datamodel.CrawlURI)">innerProcess</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;boolean</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#isHtmlExpectedHere(org.archive.crawler.datamodel.CrawlURI)">isHtmlExpectedHere</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Test whether this HTML is so unexpected (eg in place of a GIF URI)
 that it shouldn't be scanned for links.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorTool.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorTool.html#outlinks(org.archive.crawler.datamodel.CrawlURI)">outlinks</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#processEmbed(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence, java.lang.CharSequence)">processEmbed</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
             java.lang.CharSequence&nbsp;value,
             java.lang.CharSequence&nbsp;context)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#processEmbed(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence, java.lang.CharSequence, char)">processEmbed</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
             java.lang.CharSequence&nbsp;value,
             java.lang.CharSequence&nbsp;context,
             char&nbsp;hopType)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#processGeneralTag(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence, java.lang.CharSequence)">processGeneralTag</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
                  java.lang.CharSequence&nbsp;element,
                  java.lang.CharSequence&nbsp;cs)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#processLink(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence, java.lang.CharSequence)">processLink</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
            java.lang.CharSequence&nbsp;value,
            java.lang.CharSequence&nbsp;context)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Handle generic HREF cases.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;boolean</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#processMeta(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence)">processMeta</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
            java.lang.CharSequence&nbsp;cs)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Process metadata tags.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#processScript(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence, int)">processScript</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
              java.lang.CharSequence&nbsp;sequence,
              int&nbsp;endOfOpenTag)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>AggressiveExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/AggressiveExtractorHTML.html#processScript(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence, int)">processScript</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
              java.lang.CharSequence&nbsp;sequence,
              int&nbsp;endOfOpenTag)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#processScriptCode(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence)">processScriptCode</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
                  java.lang.CharSequence&nbsp;cs)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B>ExtractorHTML.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorHTML.html#processStyle(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence, int)">processStyle</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
             java.lang.CharSequence&nbsp;sequence,
             int&nbsp;endOfOpenTag)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Process style text.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>static&nbsp;long</CODE></FONT></TD>
<TD><CODE><B>ExtractorCSS.</B><B><A HREF="../../../../../org/archive/crawler/extractor/ExtractorCSS.html#processStyleCode(org.archive.crawler.datamodel.CrawlURI, java.lang.CharSequence, org.archive.crawler.framework.CrawlController)">processStyleCode</A></B>(<A HREF="../../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A>&nbsp;curi,
                 java.lang.CharSequence&nbsp;cs,
                 <A HREF="../../../../../org/archive/crawler/framework/CrawlController.html" title="class in org.archive.crawler.framework">Crawl