📄 module-tokenize.html
字号:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>17.5 tokenize -- Tokenizer for Python source</title>
<META NAME="description" CONTENT="17.5 tokenize -- Tokenizer for Python source">
<META NAME="keywords" CONTENT="lib">
<META NAME="resource-type" CONTENT="document">
<META NAME="distribution" CONTENT="global">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link rel="STYLESHEET" href="lib.css" tppabs="http://www.python.org/doc/current/lib/lib.css">
<LINK REL="next" href="module-tabnanny.html" tppabs="http://www.python.org/doc/current/lib/module-tabnanny.html">
<LINK REL="previous" href="module-keyword.html" tppabs="http://www.python.org/doc/current/lib/module-keyword.html">
<LINK REL="up" href="language.html" tppabs="http://www.python.org/doc/current/lib/language.html">
<LINK REL="next" href="module-tabnanny.html" tppabs="http://www.python.org/doc/current/lib/module-tabnanny.html">
</head>
<body>
<DIV CLASS="navigation"><table align="center" width="100%" cellpadding="0" cellspacing="2">
<tr>
<td><A href="module-keyword.html" tppabs="http://www.python.org/doc/current/lib/module-keyword.html"><img src="previous.gif" tppabs="http://www.python.org/doc/current/icons/previous.gif" border="0" height="32"
alt="Previous Page" width="32"></A></td>
<td><A href="language.html" tppabs="http://www.python.org/doc/current/lib/language.html"><img src="up.gif" tppabs="http://www.python.org/doc/current/icons/up.gif" border="0" height="32"
alt="Up One Level" width="32"></A></td>
<td><A href="module-tabnanny.html" tppabs="http://www.python.org/doc/current/lib/module-tabnanny.html"><img src="next.gif" tppabs="http://www.python.org/doc/current/icons/next.gif" border="0" height="32"
alt="Next Page" width="32"></A></td>
<td align="center" width="100%">Python Library Reference</td>
<td><A href="contents.html" tppabs="http://www.python.org/doc/current/lib/contents.html"><img src="contents.gif" tppabs="http://www.python.org/doc/current/icons/contents.gif" border="0" height="32"
alt="Contents" width="32"></A></td>
<td><a href="modindex.html" tppabs="http://www.python.org/doc/current/lib/modindex.html" title="Module Index"><img src="modules.gif" tppabs="http://www.python.org/doc/current/icons/modules.gif" border="0" height="32"
alt="Module Index" width="32"></a></td>
<td><A href="genindex.html" tppabs="http://www.python.org/doc/current/lib/genindex.html"><img src="index.gif" tppabs="http://www.python.org/doc/current/icons/index.gif" border="0" height="32"
alt="Index" width="32"></A></td>
</tr></table>
<b class="navlabel">Previous:</b> <a class="sectref" href="module-keyword.html" tppabs="http://www.python.org/doc/current/lib/module-keyword.html">17.4 keyword </A>
<b class="navlabel">Up:</b> <a class="sectref" href="language.html" tppabs="http://www.python.org/doc/current/lib/language.html">17. Python Language Services</A>
<b class="navlabel">Next:</b> <a class="sectref" href="module-tabnanny.html" tppabs="http://www.python.org/doc/current/lib/module-tabnanny.html">17.6 tabnanny </A>
<br><hr></DIV>
<!--End of Navigation Panel-->
<H1><A NAME="SECTION0019500000000000000000">
17.5 <tt class="module">tokenize</tt> --
Tokenizer for Python source</A>
</H1>
<P>
<P>
The <tt class="module">tokenize</tt> module provides a lexical scanner for Python
source code, implemented in Python. The scanner in this module
returns comments as tokens as well, making it useful for implementing
``pretty-printers,'' including colorizers for on-screen displays.
<P>
The scanner is exposed by a single function:
<P>
<dl><dt><b><a name='l2h-3042'><tt class='function'>tokenize</tt></a></b> (<var>readline</var><big>[</big><var>, tokeneater</var><big>]</big>)
<dd>
The <tt class="function">tokenize()</tt> function accepts two parameters: one
representing the input stream, and one providing an output mechanism
for <tt class="function">tokenize()</tt>.
<P>
The first parameter, <var>readline</var>, must be a callable object which
provides the same interface as the <tt class="method">readline()</tt> method of
built-in file objects (see section <A href="bltin-file-objects.html#bltin-file-objects" tppabs="http://www.python.org/doc/current/lib/bltin-file-objects.html#bltin-file-objects">2.1.7</A>). Each
call to the function should return one line of input as a string.
<P>
The second parameter, <var>tokeneater</var>, must also be a callable
object. It is called with five parameters: the token type, the
token string, a tuple <code>(<var>srow</var>, <var>scol</var>)</code> specifying the
row and column where the token begins in the source, a tuple
<code>(<var>erow</var>, <var>ecol</var>)</code> giving the ending position of the
token, and the line on which the token was found. The line passed
is the <i>logical</i> line; continuation lines are included.
</dl>
<P>
All constants from the <tt class='module'><a href="module-token.html" tppabs="http://www.python.org/doc/current/lib/module-token.html">token</a></tt> module are also exported from
<tt class="module">tokenize</tt>, as is one additional token type value that might be
passed to the <var>tokeneater</var> function by <tt class="function">tokenize()</tt>:
<P>
<dl><dt><b><a name='l2h-3043'><tt>COMMENT</tt></a></b>
<dd>
Token value used to indicate a comment.
</dl>
<DIV CLASS="navigation"><p><hr><table align="center" width="100%" cellpadding="0" cellspacing="2">
<tr>
<td><A href="module-keyword.html" tppabs="http://www.python.org/doc/current/lib/module-keyword.html"><img src="previous.gif" tppabs="http://www.python.org/doc/current/icons/previous.gif" border="0" height="32"
alt="Previous Page" width="32"></A></td>
<td><A href="language.html" tppabs="http://www.python.org/doc/current/lib/language.html"><img src="up.gif" tppabs="http://www.python.org/doc/current/icons/up.gif" border="0" height="32"
alt="Up One Level" width="32"></A></td>
<td><A href="module-tabnanny.html" tppabs="http://www.python.org/doc/current/lib/module-tabnanny.html"><img src="next.gif" tppabs="http://www.python.org/doc/current/icons/next.gif" border="0" height="32"
alt="Next Page" width="32"></A></td>
<td align="center" width="100%">Python Library Reference</td>
<td><A href="contents.html" tppabs="http://www.python.org/doc/current/lib/contents.html"><img src="contents.gif" tppabs="http://www.python.org/doc/current/icons/contents.gif" border="0" height="32"
alt="Contents" width="32"></A></td>
<td><a href="modindex.html" tppabs="http://www.python.org/doc/current/lib/modindex.html" title="Module Index"><img src="modules.gif" tppabs="http://www.python.org/doc/current/icons/modules.gif" border="0" height="32"
alt="Module Index" width="32"></a></td>
<td><A href="genindex.html" tppabs="http://www.python.org/doc/current/lib/genindex.html"><img src="index.gif" tppabs="http://www.python.org/doc/current/icons/index.gif" border="0" height="32"
alt="Index" width="32"></A></td>
</tr></table>
<b class="navlabel">Previous:</b> <a class="sectref" href="module-keyword.html" tppabs="http://www.python.org/doc/current/lib/module-keyword.html">17.4 keyword </A>
<b class="navlabel">Up:</b> <a class="sectref" href="language.html" tppabs="http://www.python.org/doc/current/lib/language.html">17. Python Language Services</A>
<b class="navlabel">Next:</b> <a class="sectref" href="module-tabnanny.html" tppabs="http://www.python.org/doc/current/lib/module-tabnanny.html">17.6 tabnanny </A>
</DIV>
<!--End of Navigation Panel-->
<ADDRESS>
<hr>See <i><a href="about.html" tppabs="http://www.python.org/doc/current/lib/about.html">About this document...</a></i> for information on suggesting changes.
</ADDRESS>
</BODY>
</HTML>
⌨️ 快捷键说明
复制代码
Ctrl + C
搜索代码
Ctrl + F
全屏模式
F11
切换主题
Ctrl + Shift + D
显示快捷键
?
增大字号
Ctrl + =
减小字号
Ctrl + -