代码搜索：Tokenize
找到约 228 项符合「Tokenize」的源代码
代码结果 228
www.eeworm.com/read/134443/13990513
c tokenize.c
/*
** 2001 September 15
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiven
www.eeworm.com/read/205013/15329855
c tokenize.c
/*
** 2001 September 15
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiven
www.eeworm.com/read/221047/14760065
html module-tokenize.html
17.5 tokenize -- Tokenizer for Python source
www.eeworm.com/read/193974/5138238
py test_tokenize.py
from test_support import verbose, findfile
import tokenize, os, sys
if verbose:
print 'starting...'
file = open(findfile('tokenize_tests'+os.extsep+'py'))
tokenize.tokenize(file.readline)
if verb
www.eeworm.com/read/193974/5138394
py tokenize_tests.py
# Tests for the 'tokenize' module.
# Large bits stolen from test_grammar.py.
# Comments
"#"
#'
#"
#\
#
# abc
'''#
#'''
x = 1 #
# Balancing continuation
a = (3, 4,
5, 6)
y = [3, 4,
www.eeworm.com/read/438718/1822892
py test_tokenize.py
from test_support import verbose, findfile
import tokenize, os, sys
if verbose:
print 'starting...'
file = open(findfile('tokenize_tests.py'))
tokenize.tokenize(file.readline)
if verbose:
pri
www.eeworm.com/read/438718/1822938
py tokenize_tests.py
# Tests for the 'tokenize' module.
# Large bits stolen from test_grammar.py.
# Comments
"#"
#'
#"
#\
#
# abc
'''#
#'''
x = 1 #
# Balancing continuation
a = (3, 4,
5, 6)
y = [3, 4,
www.eeworm.com/read/375190/2727061
hpp tokenize_and_parse.hpp
/*=============================================================================
Copyright (c) 2001-2008 Hartmut Kaiser
Distributed under the Boost Software License, Version 1.0. (See accompan
www.eeworm.com/read/375190/2727601
hpp lex_tokenize_and_parse.hpp
/*=============================================================================
Copyright (c) 2001-2008 Joel de Guzman
Copyright (c) 2001-2008 Hartmut Kaiser
http://spirit.sourceforge.net/
www.eeworm.com/read/260388/11729684
notes
These are notes from D. Richard Hipp; but, I couldn't
get this exact sample to work. Instead, see example4.y
There's a bug in lemon such that it doesn't like the primary
non-terminal to be recursiv