📄 module-robotparser.html
字号:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>12.18 robotparser -- Parser for robots.txt</title>
<META NAME="description" CONTENT="12.18 robotparser -- Parser for robots.txt">
<META NAME="keywords" CONTENT="lib">
<META NAME="resource-type" CONTENT="document">
<META NAME="distribution" CONTENT="global">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link rel="STYLESHEET" href="lib.css" tppabs="http://www.python.org/doc/current/lib/lib.css">
<LINK REL="previous" href="module-netrc.html" tppabs="http://www.python.org/doc/current/lib/module-netrc.html">
<LINK REL="up" href="netdata.html" tppabs="http://www.python.org/doc/current/lib/netdata.html">
<LINK REL="next" href="markup.html" tppabs="http://www.python.org/doc/current/lib/markup.html">
</head>
<body>
<DIV CLASS="navigation"><table align="center" width="100%" cellpadding="0" cellspacing="2">
<tr>
<td><A href="netrc-objects.html" tppabs="http://www.python.org/doc/current/lib/netrc-objects.html"><img src="previous.gif" tppabs="http://www.python.org/doc/current/icons/previous.gif" border="0" height="32"
alt="Previous Page" width="32"></A></td>
<td><A href="netdata.html" tppabs="http://www.python.org/doc/current/lib/netdata.html"><img src="up.gif" tppabs="http://www.python.org/doc/current/icons/up.gif" border="0" height="32"
alt="Up One Level" width="32"></A></td>
<td><A href="markup.html" tppabs="http://www.python.org/doc/current/lib/markup.html"><img src="next.gif" tppabs="http://www.python.org/doc/current/icons/next.gif" border="0" height="32"
alt="Next Page" width="32"></A></td>
<td align="center" width="100%">Python Library Reference</td>
<td><A href="contents.html" tppabs="http://www.python.org/doc/current/lib/contents.html"><img src="contents.gif" tppabs="http://www.python.org/doc/current/icons/contents.gif" border="0" height="32"
alt="Contents" width="32"></A></td>
<td><a href="modindex.html" tppabs="http://www.python.org/doc/current/lib/modindex.html" title="Module Index"><img src="modules.gif" tppabs="http://www.python.org/doc/current/icons/modules.gif" border="0" height="32"
alt="Module Index" width="32"></a></td>
<td><A href="genindex.html" tppabs="http://www.python.org/doc/current/lib/genindex.html"><img src="index.gif" tppabs="http://www.python.org/doc/current/icons/index.gif" border="0" height="32"
alt="Index" width="32"></A></td>
</tr></table>
<b class="navlabel">Previous:</b> <a class="sectref" href="netrc-objects.html" tppabs="http://www.python.org/doc/current/lib/netrc-objects.html">12.17.1 netrc Objects</A>
<b class="navlabel">Up:</b> <a class="sectref" href="netdata.html" tppabs="http://www.python.org/doc/current/lib/netdata.html">12. Internet Data Handling</A>
<b class="navlabel">Next:</b> <a class="sectref" href="markup.html" tppabs="http://www.python.org/doc/current/lib/markup.html">13. Structured Markup Processing</A>
<br><hr></DIV>
<!--End of Navigation Panel-->
<H1><A NAME="SECTION00141800000000000000000">
12.18 <tt class="module">robotparser</tt> --
Parser for robots.txt</A>
</H1>
<P>
<P>
<P>
This module provides a single class, <tt class="class">RobotFileParser</tt>, which answers
questions about whether or not a particular user agent can fetch a URL on
the web site that published the <span class="file">robots.txt</span> file. For more details on
the structure of <span class="file">robots.txt</span> files, see
<a class="url" href="javascript:if(confirm('http://info.webcrawler.com/mak/projects/robots/norobots.html \n\nThis file was not retrieved by Teleport Pro, because it is addressed on a domain or path outside the boundaries set for its Starting Address. \n\nDo you want to open it from the server?'))window.location='http://info.webcrawler.com/mak/projects/robots/norobots.html'" tppabs="http://info.webcrawler.com/mak/projects/robots/norobots.html">http://info.webcrawler.com/mak/projects/robots/norobots.html <img src="offsite.gif" tppabs="http://www.python.org/doc/current/icons/offsite.gif"
border='0' class='offsitelink' height='15' width='17' alt='[off-site link]'
></a>.
<P>
<dl><dt><b><a name='l2h-2548'><tt class='class'>RobotFileParser</tt></a></b> ()
<dd>
<P>
This class provides a set of methods to read, parse and answer questions
about a single <span class="file">robots.txt</span> file.
<P>
<dl><dt><b><a name='l2h-2549'><tt class='method'>set_url</tt></a></b> (<var>url</var>)
<dd>
Sets the URL referring to a <span class="file">robots.txt</span> file.
</dl>
<P>
<dl><dt><b><a name='l2h-2550'><tt class='method'>read</tt></a></b> ()
<dd>
Reads the <span class="file">robots.txt</span> URL and feeds it to the parser.
</dl>
<P>
<dl><dt><b><a name='l2h-2551'><tt class='method'>parse</tt></a></b> (<var>lines</var>)
<dd>
Parses the lines argument.
</dl>
<P>
<dl><dt><b><a name='l2h-2552'><tt class='method'>can_fetch</tt></a></b> (<var>useragent, url</var>)
<dd>
Returns true if the <var>useragent</var> is allowed to fetch the <var>url</var>
according to the rules contained in the parsed <span class="file">robots.txt</span> file.
</dl>
<P>
<dl><dt><b><a name='l2h-2553'><tt class='method'>mtime</tt></a></b> ()
<dd>
Returns the time the <code>robots.txt</code> file was last fetched. This is
useful for long-running web spiders that need to check for new
<code>robots.txt</code> files periodically.
</dl>
<P>
<dl><dt><b><a name='l2h-2554'><tt class='method'>modified</tt></a></b> ()
<dd>
Sets the time the <code>robots.txt</code> file was last fetched to the current
time.
</dl>
<P>
</dl>
<P>
The following example demonstrates basic use of the RobotFileParser class.
<P>
<dl><dd><pre class="verbatim">
>>> import robotparser
>>> rp = robotparser.RobotFileParser()
>>> rp.set_url("http://www.musi-cal.com/robots.txt")
>>> rp.read()
>>> rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")
0
>>> rp.can_fetch("*", "http://www.musi-cal.com/")
1
</pre></dl>
<P>
<DIV CLASS="navigation"><p><hr><table align="center" width="100%" cellpadding="0" cellspacing="2">
<tr>
<td><A href="netrc-objects.html" tppabs="http://www.python.org/doc/current/lib/netrc-objects.html"><img src="previous.gif" tppabs="http://www.python.org/doc/current/icons/previous.gif" border="0" height="32"
alt="Previous Page" width="32"></A></td>
<td><A href="netdata.html" tppabs="http://www.python.org/doc/current/lib/netdata.html"><img src="up.gif" tppabs="http://www.python.org/doc/current/icons/up.gif" border="0" height="32"
alt="Up One Level" width="32"></A></td>
<td><A href="markup.html" tppabs="http://www.python.org/doc/current/lib/markup.html"><img src="next.gif" tppabs="http://www.python.org/doc/current/icons/next.gif" border="0" height="32"
alt="Next Page" width="32"></A></td>
<td align="center" width="100%">Python Library Reference</td>
<td><A href="contents.html" tppabs="http://www.python.org/doc/current/lib/contents.html"><img src="contents.gif" tppabs="http://www.python.org/doc/current/icons/contents.gif" border="0" height="32"
alt="Contents" width="32"></A></td>
<td><a href="modindex.html" tppabs="http://www.python.org/doc/current/lib/modindex.html" title="Module Index"><img src="modules.gif" tppabs="http://www.python.org/doc/current/icons/modules.gif" border="0" height="32"
alt="Module Index" width="32"></a></td>
<td><A href="genindex.html" tppabs="http://www.python.org/doc/current/lib/genindex.html"><img src="index.gif" tppabs="http://www.python.org/doc/current/icons/index.gif" border="0" height="32"
alt="Index" width="32"></A></td>
</tr></table>
<b class="navlabel">Previous:</b> <a class="sectref" href="netrc-objects.html" tppabs="http://www.python.org/doc/current/lib/netrc-objects.html">12.17.1 netrc Objects</A>
<b class="navlabel">Up:</b> <a class="sectref" href="netdata.html" tppabs="http://www.python.org/doc/current/lib/netdata.html">12. Internet Data Handling</A>
<b class="navlabel">Next:</b> <a class="sectref" href="markup.html" tppabs="http://www.python.org/doc/current/lib/markup.html">13. Structured Markup Processing</A>
</DIV>
<!--End of Navigation Panel-->
<ADDRESS>
<hr>See <i><a href="about.html" tppabs="http://www.python.org/doc/current/lib/about.html">About this document...</a></i> for information on suggesting changes.
</ADDRESS>
</BODY>
</HTML>
⌨️ 快捷键说明
复制代码
Ctrl + C
搜索代码
Ctrl + F
全屏模式
F11
切换主题
Ctrl + Shift + D
显示快捷键
?
增大字号
Ctrl + =
减小字号
Ctrl + -