adaptiverevisitfrontier.html

用JAVA编写的,在做实验的时候留下来的,本来想删的,但是传上来,大家分享吧

<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#ATTR_FORCE_QUEUE">ATTR_FORCE_QUEUE</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Queue assignment to force on CrawlURIs.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>static&nbsp;java.lang.String</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#ATTR_HOST_VALENCE">ATTR_HOST_VALENCE</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Maximum simultaneous requests in process to a host (queue)</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>static&nbsp;java.lang.String</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#ATTR_MAX_DELAY">ATTR_MAX_DELAY</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Never wait more than this long, regardless of multiple</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>static&nbsp;java.lang.String</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#ATTR_MAX_RETRIES">ATTR_MAX_RETRIES</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Maximum times to emit a CrawlURI without final disposition</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>static&nbsp;java.lang.String</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#ATTR_MIN_DELAY">ATTR_MIN_DELAY</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Always wait this long after one completion before recontacting
 same server, regardless of multiple</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>static&nbsp;java.lang.String</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#ATTR_PREFERENCE_EMBED_HOPS">ATTR_PREFERENCE_EMBED_HOPS</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Number of hops of embeds (ERX) to bump to front of host queue</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>static&nbsp;java.lang.String</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#ATTR_QUEUE_IGNORE_WWW">ATTR_QUEUE_IGNORE_WWW</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Should the queue assignment ignore www in hostnames, effectively 
  stripping them away.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>static&nbsp;java.lang.String</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#ATTR_RETRY_DELAY">ATTR_RETRY_DELAY</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;For retryable problems, seconds to wait before a retry</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>static&nbsp;java.lang.String</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#ATTR_USE_URI_UNIQ_FILTER">ATTR_USE_URI_UNIQ_FILTER</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Should the Frontier use a seperate 'already included' datastructure
  or rely on the queues'.</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected static&nbsp;java.lang.String</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#DEFAULT_FORCE_QUEUE">DEFAULT_FORCE_QUEUE</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected static&nbsp;java.lang.Boolean</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#DEFAULT_QUEUE_IGNORE_WWW">DEFAULT_QUEUE_IGNORE_WWW</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected static&nbsp;java.lang.Boolean</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#DEFAULT_USE_URI_UNIQ_FILTER">DEFAULT_USE_URI_UNIQ_FILTER</A></B></CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
</TABLE>
&nbsp;<A NAME="fields_inherited_from_class_org.archive.crawler.settings.ComplexType"><!-- --></A>
<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#EEEEFF" CLASS="TableSubHeadingColor">
<TH ALIGN="left"><B>Fields inherited from class org.archive.crawler.settings.<A HREF="../../../../org/archive/crawler/settings/ComplexType.html" title="class in org.archive.crawler.settings">ComplexType</A></B></TH>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD><CODE><A HREF="../../../../org/archive/crawler/settings/ComplexType.html#definitionMap">definitionMap</A></CODE></TD>
</TR>
</TABLE>
&nbsp;<A NAME="fields_inherited_from_class_org.archive.crawler.framework.Frontier"><!-- --></A>
<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#EEEEFF" CLASS="TableSubHeadingColor">
<TH ALIGN="left"><B>Fields inherited from interface org.archive.crawler.framework.<A HREF="../../../../org/archive/crawler/framework/Frontier.html" title="interface in org.archive.crawler.framework">Frontier</A></B></TH>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD><CODE><A HREF="../../../../org/archive/crawler/framework/Frontier.html#ATTR_NAME">ATTR_NAME</A></CODE></TD>
</TR>
</TABLE>
&nbsp;<A NAME="fields_inherited_from_class_org.archive.crawler.datamodel.FetchStatusCodes"><!-- --></A>
<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#EEEEFF" CLASS="TableSubHeadingColor">
<TH ALIGN="left"><B>Fields inherited from interface org.archive.crawler.datamodel.<A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html" title="interface in org.archive.crawler.datamodel">FetchStatusCodes</A></B></TH>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD><CODE><A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_BLOCKED_BY_CUSTOM_PROCESSOR">S_BLOCKED_BY_CUSTOM_PROCESSOR</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_BLOCKED_BY_QUOTA">S_BLOCKED_BY_QUOTA</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_BLOCKED_BY_RUNTIME_LIMIT">S_BLOCKED_BY_RUNTIME_LIMIT</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_BLOCKED_BY_USER">S_BLOCKED_BY_USER</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_CONNECT_FAILED">S_CONNECT_FAILED</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_CONNECT_LOST">S_CONNECT_LOST</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_DEEMED_CHAFF">S_DEEMED_CHAFF</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_DEFERRED">S_DEFERRED</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_DELETED_BY_USER">S_DELETED_BY_USER</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_DNS_SUCCESS">S_DNS_SUCCESS</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_DOMAIN_PREREQUISITE_FAILURE">S_DOMAIN_PREREQUISITE_FAILURE</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_DOMAIN_UNRESOLVABLE">S_DOMAIN_UNRESOLVABLE</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_GETBYNAME_SUCCESS">S_GETBYNAME_SUCCESS</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_OTHER_PREREQUISITE_FAILURE">S_OTHER_PREREQUISITE_FAILURE</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_OUT_OF_SCOPE">S_OUT_OF_SCOPE</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_PREREQUISITE_UNSCHEDULABLE_FAILURE">S_PREREQUISITE_UNSCHEDULABLE_FAILURE</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_PROCESSING_THREAD_KILLED">S_PROCESSING_THREAD_KILLED</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_ROBOTS_PRECLUDED">S_ROBOTS_PRECLUDED</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_ROBOTS_PREREQUISITE_FAILURE">S_ROBOTS_PREREQUISITE_FAILURE</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_RUNTIME_EXCEPTION">S_RUNTIME_EXCEPTION</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_SERIOUS_ERROR">S_SERIOUS_ERROR</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_TIMEOUT">S_TIMEOUT</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_TOO_MANY_EMBED_HOPS">S_TOO_MANY_EMBED_HOPS</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_TOO_MANY_LINK_HOPS">S_TOO_MANY_LINK_HOPS</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_TOO_MANY_RETRIES">S_TOO_MANY_RETRIES</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_UNATTEMPTED">S_UNATTEMPTED</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_UNFETCHABLE_URI">S_UNFETCHABLE_URI</A>, <A HREF="../../../../org/archive/crawler/datamodel/FetchStatusCodes.html#S_UNQUEUEABLE">S_UNQUEUEABLE</A></CODE></TD>
</TR>
</TABLE>
&nbsp;<A NAME="fields_inherited_from_class_org.archive.crawler.frontier.AdaptiveRevisitAttributeConstants"><!-- --></A>
<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#EEEEFF" CLASS="TableSubHeadingColor">
<TH ALIGN="left"><B>Fields inherited from interface org.archive.crawler.frontier.<A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html" title="interface in org.archive.crawler.frontier">AdaptiveRevisitAttributeConstants</A></B></TH>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD><CODE><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_CONTENT_STATE_KEY">A_CONTENT_STATE_KEY</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_FETCH_OVERDUE">A_FETCH_OVERDUE</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_LAST_CONTENT_DIGEST">A_LAST_CONTENT_DIGEST</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_LAST_DATESTAMP">A_LAST_DATESTAMP</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_LAST_ETAG">A_LAST_ETAG</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_NUMBER_OF_VERSIONS">A_NUMBER_OF_VERSIONS</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_NUMBER_OF_VISITS">A_NUMBER_OF_VISITS</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_TIME_OF_NEXT_PROCESSING">A_TIME_OF_NEXT_PROCESSING</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_WAIT_INTERVAL">A_WAIT_INTERVAL</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#A_WAIT_REEVALUATED">A_WAIT_REEVALUATED</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#CONTENT_CHANGED">CONTENT_CHANGED</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#CONTENT_UNCHANGED">CONTENT_UNCHANGED</A>, <A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitAttributeConstants.html#CONTENT_UNKNOWN">CONTENT_UNKNOWN</A></CODE></TD>
</TR>
</TABLE>
&nbsp;
<!-- ======== CONSTRUCTOR SUMMARY ======== -->

<A NAME="constructor_summary"><!-- --></A>
<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#CCCCFF" CLASS="TableHeadingColor">
<TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2">
<B>Constructor Summary</B></FONT></TH>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#AdaptiveRevisitFrontier(java.lang.String)">AdaptiveRevisitFrontier</A></B>(java.lang.String&nbsp;name)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#AdaptiveRevisitFrontier(java.lang.String, java.lang.String)">AdaptiveRevisitFrontier</A></B>(java.lang.String&nbsp;name,
                        java.lang.String&nbsp;description)</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
</TABLE>
&nbsp;
<!-- ========== METHOD SUMMARY =========== -->

<A NAME="method_summary"><!-- --></A>
<TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY="">
<TR BGCOLOR="#CCCCFF" CLASS="TableHeadingColor">
<TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2">
<B>Method Summary</B></FONT></TH>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>&nbsp;long</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#averageDepth()">averageDepth</A></B>()</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">
<TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1">
<CODE>protected &nbsp;void</CODE></FONT></TD>
<TD><CODE><B><A HREF="../../../../org/archive/crawler/frontier/AdaptiveRevisitFrontier.html#batchFlush()">batchFlush</A></B>()</CODE>

<BR>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TD>
</TR>
<TR BGCOLOR="white" CLASS="TableRowColor">