📄 robotsexclusionpolicy.html
字号:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><!--NewPage--><HTML><HEAD><!-- Generated by javadoc (build 1.5.0_06) on Wed Sep 27 16:03:05 PDT 2006 --><TITLE>RobotsExclusionPolicy (Heritrix 1.10.1)</TITLE><META NAME="keywords" CONTENT="org.archive.crawler.datamodel.RobotsExclusionPolicy class"><LINK REL ="stylesheet" TYPE="text/css" HREF="../../../../stylesheet.css" TITLE="Style"><SCRIPT type="text/javascript">function windowTitle(){ parent.document.title="RobotsExclusionPolicy (Heritrix 1.10.1)";}</SCRIPT><NOSCRIPT></NOSCRIPT></HEAD><BODY BGCOLOR="white" onload="windowTitle();"><!-- ========= START OF TOP NAVBAR ======= --><A NAME="navbar_top"><!-- --></A><A HREF="#skip-navbar_top" title="Skip navigation links"></A><TABLE BORDER="0" WIDTH="100%" CELLPADDING="1" CELLSPACING="0" SUMMARY=""><TR><TD COLSPAN=2 BGCOLOR="#EEEEFF" CLASS="NavBarCell1"><A NAME="navbar_top_firstrow"><!-- --></A><TABLE BORDER="0" CELLPADDING="0" CELLSPACING="3" SUMMARY=""> <TR ALIGN="center" VALIGN="top"> <TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="../../../../overview-summary.html"><FONT CLASS="NavBarFont1"><B>Overview</B></FONT></A> </TD> <TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="package-summary.html"><FONT CLASS="NavBarFont1"><B>Package</B></FONT></A> </TD> <TD BGCOLOR="#FFFFFF" CLASS="NavBarCell1Rev"> <FONT CLASS="NavBarFont1Rev"><B>Class</B></FONT> </TD> <TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="class-use/RobotsExclusionPolicy.html"><FONT CLASS="NavBarFont1"><B>Use</B></FONT></A> </TD> <TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="package-tree.html"><FONT CLASS="NavBarFont1"><B>Tree</B></FONT></A> </TD> <TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="../../../../deprecated-list.html"><FONT CLASS="NavBarFont1"><B>Deprecated</B></FONT></A> </TD> <TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="../../../../index-all.html"><FONT CLASS="NavBarFont1"><B>Index</B></FONT></A> </TD> <TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="../../../../help-doc.html"><FONT CLASS="NavBarFont1"><B>Help</B></FONT></A> </TD> </TR></TABLE></TD><TD ALIGN="right" VALIGN="top" ROWSPAN=3><EM></EM></TD></TR><TR><TD BGCOLOR="white" CLASS="NavBarCell2"><FONT SIZE="-2"> <A HREF="../../../../org/archive/crawler/datamodel/LocalizedError.html" title="class in org.archive.crawler.datamodel"><B>PREV CLASS</B></A> <A HREF="../../../../org/archive/crawler/datamodel/RobotsHonoringPolicy.html" title="class in org.archive.crawler.datamodel"><B>NEXT CLASS</B></A></FONT></TD><TD BGCOLOR="white" CLASS="NavBarCell2"><FONT SIZE="-2"> <A HREF="../../../../index.html?org/archive/crawler/datamodel/RobotsExclusionPolicy.html" target="_top"><B>FRAMES</B></A> <A HREF="RobotsExclusionPolicy.html" target="_top"><B>NO FRAMES</B></A> <SCRIPT type="text/javascript"> <!-- if(window==top) { document.writeln('<A HREF="../../../../allclasses-noframe.html"><B>All Classes</B></A>'); } //--></SCRIPT><NOSCRIPT> <A HREF="../../../../allclasses-noframe.html"><B>All Classes</B></A></NOSCRIPT></FONT></TD></TR><TR><TD VALIGN="top" CLASS="NavBarCell3"><FONT SIZE="-2"> SUMMARY: NESTED | <A HREF="#field_summary">FIELD</A> | <A HREF="#constructor_summary">CONSTR</A> | <A HREF="#method_summary">METHOD</A></FONT></TD><TD VALIGN="top" CLASS="NavBarCell3"><FONT SIZE="-2">DETAIL: <A HREF="#field_detail">FIELD</A> | <A HREF="#constructor_detail">CONSTR</A> | <A HREF="#method_detail">METHOD</A></FONT></TD></TR></TABLE><A NAME="skip-navbar_top"></A><!-- ========= END OF TOP NAVBAR ========= --><HR><!-- ======== START OF CLASS DATA ======== --><H2><FONT SIZE="-1">org.archive.crawler.datamodel</FONT><BR>Class RobotsExclusionPolicy</H2><PRE>java.lang.Object <IMG SRC="../../../../resources/inherit.gif" ALT="extended by "><B>org.archive.crawler.datamodel.RobotsExclusionPolicy</B></PRE><DL><DT><B>All Implemented Interfaces:</B> <DD>java.io.Serializable</DD></DL><HR><DL><DT><PRE>public class <B>RobotsExclusionPolicy</B><DT>extends java.lang.Object<DT>implements java.io.Serializable</DL></PRE><P>RobotsExclusionPolicy represents the actual policy adopted with respect to a specific remote server, usually constructed from consulting the robots.txt, if any, the server provided. (The similarly named RobotsHonoringPolicy, on the other hand, describes the strategy used by the crawler to determine to what extent it respects exclusion rules.) The expiration of policies after a suitable amount of time has elapsed since last fetch is handled outside this class, in CrawlServer itself.<P><P><DL><DT><B>Author:</B></DT> <DD>gojomo</DD><DT><B>See Also:</B><DD><A HREF="../../../../serialized-form.html#org.archive.crawler.datamodel.RobotsExclusionPolicy">Serialized Form</A></DL><HR><P><!-- =========== FIELD SUMMARY =========== --><A NAME="field_summary"><!-- --></A><TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY=""><TR BGCOLOR="#CCCCFF" CLASS="TableHeadingColor"><TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2"><B>Field Summary</B></FONT></TH></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>static <A HREF="../../../../org/archive/crawler/datamodel/RobotsExclusionPolicy.html" title="class in org.archive.crawler.datamodel">RobotsExclusionPolicy</A></CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/datamodel/RobotsExclusionPolicy.html#ALLOWALL">ALLOWALL</A></B></CODE><BR> </TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>static <A HREF="../../../../org/archive/crawler/datamodel/RobotsExclusionPolicy.html" title="class in org.archive.crawler.datamodel">RobotsExclusionPolicy</A></CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/datamodel/RobotsExclusionPolicy.html#DENYALL">DENYALL</A></B></CODE><BR> </TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>(package private) <A HREF="../../../../org/archive/crawler/datamodel/RobotsHonoringPolicy.html" title="class in org.archive.crawler.datamodel">RobotsHonoringPolicy</A></CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/datamodel/RobotsExclusionPolicy.html#honoringPolicy">honoringPolicy</A></B></CODE><BR> </TD></TR></TABLE> <!-- ======== CONSTRUCTOR SUMMARY ======== --><A NAME="constructor_summary"><!-- --></A><TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY=""><TR BGCOLOR="#CCCCFF" CLASS="TableHeadingColor"><TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2"><B>Constructor Summary</B></FONT></TH></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD><CODE><B><A HREF="../../../../org/archive/crawler/datamodel/RobotsExclusionPolicy.html#RobotsExclusionPolicy(org.archive.crawler.settings.CrawlerSettings, java.util.LinkedList, java.util.HashMap, org.archive.crawler.datamodel.RobotsHonoringPolicy)">RobotsExclusionPolicy</A></B>(<A HREF="../../../../org/archive/crawler/settings/CrawlerSettings.html" title="class in org.archive.crawler.settings">CrawlerSettings</A> settings, java.util.LinkedList u, java.util.HashMap d, <A HREF="../../../../org/archive/crawler/datamodel/RobotsHonoringPolicy.html" title="class in org.archive.crawler.datamodel">RobotsHonoringPolicy</A> honoringPolicy)</CODE><BR> </TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD><CODE><B><A HREF="../../../../org/archive/crawler/datamodel/RobotsExclusionPolicy.html#RobotsExclusionPolicy(int)">RobotsExclusionPolicy</A></B>(int type)</CODE><BR> </TD></TR></TABLE> <!-- ========== METHOD SUMMARY =========== --><A NAME="method_summary"><!-- --></A><TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY=""><TR BGCOLOR="#CCCCFF" CLASS="TableHeadingColor"><TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2"><B>Method Summary</B></FONT></TH></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> boolean</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/datamodel/RobotsExclusionPolicy.html#disallows(org.archive.crawler.datamodel.CrawlURI, java.lang.String)">disallows</A></B>(<A HREF="../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A> curi, java.lang.String userAgent)</CODE><BR> </TD></TR>⌨️ 快捷键说明
复制代码
Ctrl + C
搜索代码
Ctrl + F
全屏模式
F11
切换主题
Ctrl + Shift + D
显示快捷键
?
增大字号
Ctrl + =
减小字号
Ctrl + -