📄 fetchhttp.html
字号:
<TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2"><B>Constructor Summary</B></FONT></TH></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#FetchHTTP(java.lang.String)">FetchHTTP</A></B>(java.lang.String name)</CODE><BR> Constructor.</TD></TR></TABLE> <!-- ========== METHOD SUMMARY =========== --><A NAME="method_summary"><!-- --></A><TABLE BORDER="1" WIDTH="100%" CELLPADDING="3" CELLSPACING="0" SUMMARY=""><TR BGCOLOR="#CCCCFF" CLASS="TableHeadingColor"><TH ALIGN="left" COLSPAN="2"><FONT SIZE="+2"><B>Method Summary</B></FONT></TH></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#addResponseContent(org.apache.commons.httpclient.HttpMethod, org.archive.crawler.datamodel.CrawlURI)">addResponseContent</A></B>(org.apache.commons.httpclient.HttpMethod method, <A HREF="../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A> curi)</CODE><BR> This method populates <code>curi</code> with response status and content type.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected boolean</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#checkMidfetchAbort(org.archive.crawler.datamodel.CrawlURI, org.archive.httpclient.HttpRecorderMethod, org.apache.commons.httpclient.HttpConnection)">checkMidfetchAbort</A></B>(<A HREF="../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A> curi, <A HREF="../../../../org/archive/httpclient/HttpRecorderMethod.html" title="class in org.archive.httpclient">HttpRecorderMethod</A> method, org.apache.commons.httpclient.HttpConnection conn)</CODE><BR> </TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#cleanupHttp()">cleanupHttp</A></B>()</CODE><BR> Perform any final cleanup related to the HttpClient instance.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#configureHttp()">configureHttp</A></B>()</CODE><BR> </TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected org.apache.commons.httpclient.HostConfiguration</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#configureMethod(org.archive.crawler.datamodel.CrawlURI, org.apache.commons.httpclient.HttpMethod)">configureMethod</A></B>(<A HREF="../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A> curi, org.apache.commons.httpclient.HttpMethod method)</CODE><BR> Configure the HttpMethod setting options and headers.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#crawlCheckpoint(java.io.File)">crawlCheckpoint</A></B>(java.io.File checkpointDir)</CODE><BR> Called by <A HREF="../../../../org/archive/crawler/framework/CrawlController.html" title="class in org.archive.crawler.framework"><CODE>CrawlController</CODE></A> when checkpointing.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#crawlEnded(java.lang.String)">crawlEnded</A></B>(java.lang.String sExitMessage)</CODE><BR> Called when a CrawlController has ended a crawl and is about to exit.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#crawlEnding(java.lang.String)">crawlEnding</A></B>(java.lang.String sExitMessage)</CODE><BR> Called when a CrawlController is ending a crawl (for any reason)</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#crawlPaused(java.lang.String)">crawlPaused</A></B>(java.lang.String statusMessage)</CODE><BR> Called when a CrawlController is actually paused (all threads are idle).</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#crawlPausing(java.lang.String)">crawlPausing</A></B>(java.lang.String statusMessage)</CODE><BR> Called when a CrawlController is going to be paused.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#crawlResuming(java.lang.String)">crawlResuming</A></B>(java.lang.String statusMessage)</CODE><BR> Called when a CrawlController is resuming a crawl that had been paused.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#crawlStarted(java.lang.String)">crawlStarted</A></B>(java.lang.String message)</CODE><BR> Called on crawl start.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#doAbort(org.archive.crawler.datamodel.CrawlURI, org.apache.commons.httpclient.HttpMethod, java.lang.String)">doAbort</A></B>(<A HREF="../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A> curi, org.apache.commons.httpclient.HttpMethod method, java.lang.String annotation)</CODE><BR> </TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#finalTasks()">finalTasks</A></B>()</CODE><BR> Classes subclassing this one should override this method to perform processor specific actions.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected java.lang.Object</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#getAttributeEither(org.archive.crawler.datamodel.CrawlURI, java.lang.String)">getAttributeEither</A></B>(<A HREF="../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A> curi, java.lang.String key)</CODE><BR> Get a value either from inside the CrawlURI instance, or from settings (module attributes).</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected org.apache.commons.httpclient.auth.AuthScheme</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#getAuthScheme(org.apache.commons.httpclient.HttpMethod, org.archive.crawler.datamodel.CrawlURI)">getAuthScheme</A></B>(org.apache.commons.httpclient.HttpMethod method, <A HREF="../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A> curi)</CODE><BR> </TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected org.apache.commons.httpclient.HttpClient</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#getHttp()">getHttp</A></B>()</CODE><BR> </TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#handle401(org.apache.commons.httpclient.HttpMethod, org.archive.crawler.datamodel.CrawlURI)">handle401</A></B>(org.apache.commons.httpclient.HttpMethod method, <A HREF="../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A> curi)</CODE><BR> Server is looking for basic/digest auth credentials (RFC2617).</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#initialTasks()">initialTasks</A></B>()</CODE><BR> Classes subclassing this one should override this method to perform processor specific actions.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#innerProcess(org.archive.crawler.datamodel.CrawlURI)">innerProcess</A></B>(<A HREF="../../../../org/archive/crawler/datamodel/CrawlURI.html" title="class in org.archive.crawler.datamodel">CrawlURI</A> curi)</CODE><BR> Classes subclassing this one should override this method to perform their custom actions on the CrawlURI.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE>protected void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#listUsedFiles(java.util.List)">listUsedFiles</A></B>(java.util.List list)</CODE><BR> Those Modules that use files on disk should list them all when this method is called.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#loadCookies()">loadCookies</A></B>()</CODE><BR> Load cookies from the file specified in the order file.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> void</CODE></FONT></TD><TD><CODE><B><A HREF="../../../../org/archive/crawler/fetcher/FetchHTTP.html#loadCookies(java.lang.String)">loadCookies</A></B>(java.lang.String cookiesFile)</CODE><BR> Load cookies from a file before the first fetch.</TD></TR><TR BGCOLOR="white" CLASS="TableRowColor"><TD ALIGN="right" VALIGN="top" WIDTH="1%"><FONT SIZE="-1"><CODE> java.lang.String</CODE></FONT></TD>⌨️ 快捷键说明
复制代码
Ctrl + C
搜索代码
Ctrl + F
全屏模式
F11
切换主题
Ctrl + Shift + D
显示快捷键
?
增大字号
Ctrl + =
减小字号
Ctrl + -