chap04.htm.kbk
来自「c++设计思想」· KBK 代码 · 共 1,135 行 · 第 1/5 页
KBK
1,135 行
<B>wordlist</B>.</FONT><A NAME="_Toc519042036"></A><BR></P></DIV>
<A NAME="Heading224"></A><FONT FACE = "Verdana, Tahoma, Arial, Helvetica, Sans"><H3 ALIGN="LEFT">
StreamTokenizer: <BR>a more flexible solution</H3></FONT>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia">The above program parses its input into
strings of words containing only alpha characters, but that’s still a
special case compared to the generality of <B>strtok( )</B>. What
we’d like now is an actual replacement for <B>strtok( )</B> so
we’re never tempted to use it. <B>WordList2.cpp</B> can be modified to
create a class called <B>StreamTokenizer</B> that delivers a new token as a
<B>string</B> whenever you call <B>next( )</B>, according to the delimiters
you give it upon construction (very similar to
<B>strtok( )</B>):</FONT><BR></P></DIV>
<BLOCKQUOTE><FONT SIZE = "+1"><PRE><font color=#009900>//: C07:StreamTokenizer.h</font>
<font color=#009900>// C++ Replacement for Standard C strtok()</font>
#ifndef STREAMTOKENIZER_H
#define STREAMTOKENIZER_H
#include <string>
#include <iostream>
#include <iterator>
<font color=#0000ff>class</font> StreamTokenizer {
<font color=#0000ff>typedef</font> std::istreambuf_iterator<<font color=#0000ff>char</font>> It;
It p, end;
std::string delimiters;
<font color=#0000ff>bool</font> isDelimiter(<font color=#0000ff>char</font> c) {
<font color=#0000ff>return</font>
delimiters.find(c) != std::string::npos;
}
<font color=#0000ff>public</font>:
StreamTokenizer(std::istream& is,
std::string delim = <font color=#004488>" \t\n;()\"</font><>:{}[]+-=&*#"
<font color=#004488>".,</font><font color=#004488>/\\~!0123456789"</font>) : p(is), end(It()),
delimiters(delim) {}
std::string next(); <font color=#009900>// Get next token</font>
};
#endif STREAMTOKENIZER_H <font color=#009900>///:~</font></PRE></FONT></BLOCKQUOTE>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia">The default delimiters for the
<B>StreamTokenizer</B> constructor extract words with only alpha characters, as
before, but now you can choose different delimiters to parse different tokens.
The implementation of <B>next( )</B> looks similar to
<B>Wordlist2.cpp</B>:</FONT><BR></P></DIV>
<BLOCKQUOTE><FONT SIZE = "+1"><PRE><font color=#009900>//: C07:StreamTokenizer.cpp {O}</font>
<font color=#009900>//{-g++295} </font>
#include <font color=#004488>"StreamTokenizer.h"</font>
<font color=#0000ff>using</font> <font color=#0000ff>namespace</font> std;
string StreamTokenizer::next() {
string result;
<font color=#0000ff>if</font>(p != end) {
insert_iterator<string>
ii(result, result.begin());
<font color=#0000ff>while</font>(isDelimiter(*p) && p != end)
p++;
<font color=#0000ff>while</font> (!isDelimiter(*p) && p != end)
*ii++ = *p++;
}
<font color=#0000ff>return</font> result;
} <font color=#009900>///:~</font></PRE></FONT></BLOCKQUOTE>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia">The first non-delimiter is found, then
characters are copied until a delimiter is found, and the resulting
<B>string</B> is returned. Here’s a test:</FONT><BR></P></DIV>
<BLOCKQUOTE><FONT SIZE = "+1"><PRE><font color=#009900>//: C07:TokenizeTest.cpp</font>
<font color=#009900>// Test StreamTokenizer</font>
<font color=#009900>//{L} StreamTokenizer ../TestSuite/Test</font>
<font color=#009900>//{-g++295} </font>
#include <font color=#004488>"StreamTokenizer.h"</font>
#include <font color=#004488>"..</font><font color=#004488>/require.h"</font>
#include <iostream>
#include <fstream>
#include <set>
<font color=#0000ff>using</font> <font color=#0000ff>namespace</font> std;
<font color=#0000ff>int</font> main(<font color=#0000ff>int</font> argc, <font color=#0000ff>char</font>* argv[]) {
<font color=#0000ff>char</font>* fname = <font color=#004488>"TokenizeTest.cpp"</font>;
<font color=#0000ff>if</font>(argc > 1) fname = argv[1];
ifstream in(fname);
assure(in, fname);
StreamTokenizer words(in);
set<string> wordlist;
string word;
<font color=#0000ff>while</font>((word = words.next()).size() != 0)
wordlist.insert(word);
<font color=#009900>// Output results:</font>
copy(wordlist.begin(), wordlist.end(),
ostream_iterator<string>(cout, <font color=#004488>"\n"</font>));
} <font color=#009900>///:~</font></PRE></FONT></BLOCKQUOTE>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia">Now the tool is more reusable than
before, but it’s still inflexible, because it can only work with an
<B>istream</B>. This isn’t as bad as it first seems, since a <B>string</B>
can be turned into an <B>istream</B> via an <B>istringstream</B>. But in the
next section we’ll come up with the most general, reusable tokenizing
tool, and this should give you a feeling of what “reusable” really
means, and the effort necessary to create truly reusable
code.</FONT><A NAME="_Toc519042037"></A><BR></P></DIV>
<A NAME="Heading225"></A><FONT FACE = "Verdana, Tahoma, Arial, Helvetica, Sans"><H3 ALIGN="LEFT">
A completely reusable tokenizer</H3></FONT>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia">Since the STL containers and algorithms
all revolve around iterators, the most flexible solution will itself be an
iterator. You could think of the <B>TokenIterator</B> as an iterator that wraps
itself around any other iterator that can produce characters. Because it is
designed as an input iterator (the most primitive type of iterator) it can be
used with any STL algorithm. Not only is it a useful tool in itself, the
<B>TokenIterator</B> is also a good example of how you can design your own
iterators.</FONT><A NAME="fnB22" HREF="#fn22">[22]</A><BR></P></DIV>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia">The <B>TokenIterator</B> is doubly
flexible: first, you can choose the type of iterator that will produce the
<B>char</B> input. Second, instead of just saying what characters represent the
delimiters, <B>TokenIterator</B> will use a predicate which is a function object
whose <B>operator( )</B> takes a <B>char</B> and decides if it should be in
the token or not. Although the two examples given here have a static concept of
what characters belong in a token, you could easily design your own function
object to change its state as the characters are read, producing a more
sophisticated parser.</FONT><BR></P></DIV>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia">The following header file contains the
two basic predicates <B>Isalpha</B> and <B>Delimiters</B>, along with the
template for <B>TokenIterator</B>:</FONT><BR></P></DIV>
<BLOCKQUOTE><FONT SIZE = "+1"><PRE><font color=#009900>//: C07:TokenIterator.h</font>
#ifndef TOKENITERATOR_H
#define TOKENITERATOR_H
#include <string>
#include <iterator>
#include <algorithm>
#include <cctype>
<font color=#0000ff>struct</font> Isalpha {
<font color=#0000ff>bool</font> <font color=#0000ff>operator</font>()(<font color=#0000ff>char</font> c) {
<font color=#0000ff>using</font> <font color=#0000ff>namespace</font> std; <font color=#009900>//[[For a compiler bug]]</font>
<font color=#0000ff>return</font> isalpha(c);
}
};
<font color=#0000ff>class</font> Delimiters {
std::string exclude;
<font color=#0000ff>public</font>:
Delimiters() {}
Delimiters(<font color=#0000ff>const</font> std::string& excl)
: exclude(excl) {}
<font color=#0000ff>bool</font> <font color=#0000ff>operator</font>()(<font color=#0000ff>char</font> c) {
<font color=#0000ff>return</font> exclude.find(c) == std::string::npos;
}
};
<font color=#0000ff>template</font> <<font color=#0000ff>class</font> InputIter, <font color=#0000ff>class</font> Pred = Isalpha>
<font color=#0000ff>class</font> TokenIterator: <font color=#0000ff>public</font> std::iterator<
std::input_iterator_tag,std::string,ptrdiff_t>{
InputIter first;
InputIter last;
std::string word;
Pred predicate;
<font color=#0000ff>public</font>:
TokenIterator(InputIter begin, InputIter end,
Pred pred = Pred())
: first(begin), last(end), predicate(pred) {
++*<font color=#0000ff>this</font>;
}
TokenIterator() {} <font color=#009900>// End sentinel</font>
<font color=#009900>// Prefix increment:</font>
TokenIterator& <font color=#0000ff>operator</font>++() {
word.resize(0);
first = std::find_if(first, last, predicate);
<font color=#0000ff>while</font> (first != last && predicate(*first))
word += *first++;
<font color=#0000ff>return</font> *<font color=#0000ff>this</font>;
}
<font color=#009900>// Postfix increment</font>
<font color=#0000ff>class</font> Proxy {
std::string word;
<font color=#0000ff>public</font>:
Proxy(<font color=#0000ff>const</font> std::string& w) : word(w) {}
std::string <font color=#0000ff>operator</font>*() { <font color=#0000ff>return</font> word; }
};
Proxy <font color=#0000ff>operator</font>++(<font color=#0000ff>int</font>) {
Proxy d(word);
++*<font color=#0000ff>this</font>;
<font color=#0000ff>return</font> d;
}
<font color=#009900>// Produce the actual value:</font>
std::string <font color=#0000ff>operator</font>*() <font color=#0000ff>const</font> { <font color=#0000ff>return</font> word; }
std::string* <font color=#0000ff>operator</font>->() <font color=#0000ff>const</font> {
<font color=#0000ff>return</font> &(<font color=#0000ff>operator</font>*());
}
<font color=#009900>// Compare iterators:</font>
<font color=#0000ff>bool</font> <font color=#0000ff>operator</font>==(<font color=#0000ff>const</font> TokenIterator&) {
<font color=#0000ff>return</font> word.size() == 0 && first == last;
}
<font color=#0000ff>bool</font> <font color=#0000ff>operator</font>!=(<font color=#0000ff>const</font> TokenIterator& rv) {
<font color=#0000ff>return</font> !(*<font color=#0000ff>this</font> == rv);
}
};
#endif <font color=#009900>// TOKENITERATOR_H ///:~</font></PRE></FONT></BLOCKQUOTE>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia"><B>TokenIterator</B> is inherited from
the <B>std::iterator</B> template. It might appear that there’s some kind
of functionality that comes with <B>std::iterator</B>, but it is purely a way of
tagging an iterator so that a container that uses it knows what it’s
capable of. Here, you can see <B>input_iterator_tag</B> as a template argument
– this tells anyone who asks that a <B>TokenIterator</B> only has the
capabilities of an input iterator, and cannot be used with algorithms requiring
more sophisticated iterators. Apart from the tagging, <B>std::iterator</B>
doesn’t do anything else, which means you must design all the other
functionality in yourself.</FONT><BR></P></DIV>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia"><B>TokenIterator</B> may look a little
strange at first, because the first constructor requires both a
“begin” and “end” iterator as arguments, along with the
predicate. Remember that this is a “wrapper” iterator that has no
idea of how to tell whether it’s at the end of its input source, so the
ending iterator is necessary in the first constructor. The reason for the second
(default) constructor is that the STL algorithms (and any algorithms you write)
need a <B>TokenIterator </B>sentinel to be the past-the-end value. Since all the
information necessary to see if the <B>TokenIterator</B> has reached the end of
its input is collected in the first constructor, this second constructor creates
a <B>TokenIterator</B> that is merely used as a placeholder in
algorithms.</FONT><BR></P></DIV>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia">The core of the behavior happens in
<B>operator++</B>. This erases the current value of <B>word</B> using
<B>string::resize( )</B>, then finds the first character that satisfies the
predicate (thus discovering the beginning of the new token) using
<B>find_if( )</B> (from the STL algorithms, discussed in the following
chapter). The resulting iterator is assigned to <B>first</B>, thus moving
<B>first</B> forward to the beginning of the token. Then, as long as the end of
the input is not reached and the predicate is satisfied, characters are copied
into the word from the input. Finally, the TokenIterator object is returned, and
must be dereferenced to access the new token.</FONT><BR></P></DIV>
<DIV ALIGN="LEFT"><P><FONT FACE="Georgia">The postfix increment requires a proxy
⌨️ 快捷键说明
复制代码Ctrl + C
搜索代码Ctrl + F
全屏模式F11
增大字号Ctrl + =
减小字号Ctrl + -
显示快捷键?