📄 tokenreq.html
字号:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><html><head><title>SQLite Tokenizer Requirements</title><style type="text/css">body { margin: auto; font-family: "Verdana" "sans-serif"; padding: 8px 1%;}a { color: #45735f }a:visited { color: #734559 }.logo { position:absolute; margin:3px; }.tagline { float:right; text-align:right; font-style:italic; width:240px; margin:12px; margin-top:58px;}.toolbar { font-variant: small-caps; text-align: center; line-height: 1.6em; margin: 0; padding:1px 8px;}.toolbar a { color: white; text-decoration: none; padding: 6px 12px; }.toolbar a:visited { color: white; }.toolbar a:hover { color: #80a796; background: white; }.content { margin: 5%; }.content dt { font-weight:bold; }.content dd { margin-bottom: 25px; margin-left:20%; }.content ul { padding:0px; padding-left: 15px; margin:0px; }/* rounded corners */.se { background: url(images/se.png) 100% 100% no-repeat #80a796}.sw { background: url(images/sw.png) 0% 100% no-repeat }.ne { background: url(images/ne.png) 100% 0% no-repeat }.nw { background: url(images/nw.png) 0% 0% no-repeat }</style><meta http-equiv="content-type" content="text/html; charset=UTF-8"> </head><body><div><!-- container div to satisfy validator --><a href="index.html"><img class="logo" src="images/SQLite.gif" alt="SQLite Logo" border="0"></a><div><!-- IE hack to prevent disappearing logo--></div><div class="tagline">Small. Fast. Reliable.<br>Choose any three.</div><table width=100% style="clear:both"><tr><td> <div class="se"><div class="sw"><div class="ne"><div class="nw"> <div class="toolbar"> <a href="about.html">About</a> <a href="sitemap.html">Sitemap</a> <a href="docs.html">Documentation</a> <a href="download.html">Download</a> <a href="copyright.html">License</a> <a href="news.html">News</a> <a href="http://www.sqlite.org/cvstrac/index">Developers</a> <a href="support.html">Support</a> </div></div></div></div></div></td></tr></table> <h1>Requirements For The SQLite Tokenizer</h1><p>When processing SQL statements, SQLite (as does every other SQLdatabase engine) breaks the SQL statement up into tokens which arethen forwarded to the parser component. SQL statements are splitinto tokens by the "tokenizer" component of SQLite. This document specifiesrequirements that precisely define the operation of the SQLite tokenizer.</p><h2>Character classes</h2><p>SQL statements are composed of unicode characters. Specificindividual characters many be described using a notation consisting ofthe character "u" followed by four hexadecimal digits. Forexample, the lower-case letter "a" can be expressed as "u0061"and the dollar sign can be expressed as "u0024". For notational convenience, the following character classes aredefined:</p><blockquote><dl><dt><b>WHITESPACE</b></dt><dd><p>One of these five characters: u0009, u000a, u000c, u000d, or u0020</p></dd><dt><b>ALPHABETIC</b></dt><dd><p>Any of the characters in the range u0041 through u005a (letters "A" through "Z") or in the range u0061 through u007a (letters "a" through "z") or the character u005f ("_") or any other character larger than u007f.</p></dd><dt><b>NUMERIC</b></dt><dd><p>Any of the characters in the range u0030 through u0039 (digits "0" through "9")</p></dd><dt><b>ALPHANUMERIC</b></dt><dd><p>Any character which is either ALPHABETIC or NUMERIC</p></dd><dt><b>HEXADECIMAL</b></dt><dd><p>Any NUMERIC character or a characters in the range u0041 through u0046 ("A" through "F") or in the range u0061 through u0066 ("a" through "f")</p></dd><dt><b>SPECIAL</b></dt><dd></p>Any character that is not WHITESPACE, ALPHABETIC, nor NUMERIC</p></dd></dl></blockquote><h2>Token requirements</h2><a name="H41010"></a><p> Processing is left-to-right. This seems obvious, but it needs to be explicitly stated.</p><blockquote><b>H41010:</b> SQLite shall divide input SQL text into tokens working from left to right.</b></blockquote><a name="H41020"></a><p> The standard practice in SQL, as with most context-free grammar based programming languages, is to resolve ambiguities in tokenizing by selecting the option that results in the longest tokens.</p><blockquote><b>H41020:</b> At each step in the SQL tokenization process, SQLite shall extract the longest possible token from the remaining input text.</b></blockquote><a name="H41030"></a><p> The tokenizer recognizes tokens one by one and passes them on to the parser. Except whitespace is ignored. The only use for whitespace is as a separator between tokens.</p><blockquote><b>H41030:</b> The tokenizer shall pass each non-WHITESPACE token seen on to the parser in the order in which the tokens are seen.</b></blockquote><a name="H41040"></a><p> The tokenizer appends a semicolon to the end of input if necessary. This ensures that every SQL statement is terminated by a semicolon.</p><blockquote><b>H41040:</b> When the tokenizer reaches the end of input where the last token sent to the parser was not a SEMI token, it shall send a SEMI token to the parser.⌨️ 快捷键说明
复制代码
Ctrl + C
搜索代码
Ctrl + F
全屏模式
F11
切换主题
Ctrl + Shift + D
显示快捷键
?
增大字号
Ctrl + =
减小字号
Ctrl + -