mbyte.texi

一个C源代码分析器

@node Extended Characters, Locales, String and Array Utilities, Top
@chapter Extended Characters

A number of languages use character sets that are larger than the range
of values of type @code{char}.  Japanese and Chinese are probably the
most familiar examples.

The GNU C library includes support for two mechanisms for dealing with
extended character sets: multibyte characters and wide characters.  This
chapter describes how to use these mechanisms, and the functions for
converting between them.
@cindex extended character sets

The behavior of the functions in this chapter is affected by the current
locale for character classification---the @code{LC_CTYPE} category; see
@ref{Locale Categories}.  This choice of locale selects which multibyte
code is used, and also controls the meanings and characteristics of wide
character codes.

@menu
* Extended Char Intro::         Multibyte codes versus wide characters.
* Locales and Extended Chars::  The locale selects the character codes.
* Multibyte Char Intro::        How multibyte codes are represented.
* Wide Char Intro::             How wide characters are represented.
* Wide String Conversion::      Converting wide strings to multibyte code
                                 and vice versa.
* Length of Char::              how many bytes make up one multibyte char.
* Converting One Char::         Converting a string character by character.
* Example of Conversion::       Example showing why converting 
				 one character at a time may be useful.
* Shift State::                 Multibyte codes with "shift characters".
@end menu

@node Extended Char Intro, Locales and Extended Chars,  , Extended Characters
@section Introduction to Extended Characters

You can represent extended characters in either of two ways:

@itemize @bullet
@item
As @dfn{multibyte characters} which can be embedded in an ordinary
string, an array of @code{char} objects.  Their advantage is that many
programs and operating systems can handle occasional multibyte
characters scattered among ordinary ASCII characters, without any
change.

@item
@cindex wide characters
As @dfn{wide characters}, which are like ordinary characters except that
they occupy more bits.  The wide character data type, @code{wchar_t},
has a range large enough to hold extended character codes as well as
old-fashioned ASCII codes.

An advantage of wide characters is that each character is a single data
object, just like ordinary ASCII characters.  There are a few
disadvantages:

@itemize @bullet
@item
Each existing program must be modified and recompiled to make it use
wide characters.

@item
Files of wide characters cannot be read by programs that expect ordinary
characters.
@end itemize
@end itemize

Typically, you use the multibyte character representation as part of the
external program interface, such as reading or writing text to files.
However, it's usually easier to perform internal manipulations on
strings containing extended characters on arrays of @code{wchar_t}
objects, since the uniform representation makes most editing operations
easier.  If you do use multibyte characters for files and wide
characters for internal operations, you need to convert between them
when you read and write data.

If your system supports extended characters, then it supports them both
as multibyte characters and as wide characters.  The library includes
functions you can use to convert between the two representations.
These functions are described in this chapter.

@node Locales and Extended Chars, Multibyte Char Intro, Extended Char Intro, Extended Characters
@section Locales and Extended Characters

A computer system can support more than one multibyte character code,
and more than one wide character code.  The user controls the choice of
codes through the current locale for character classification
(@pxref{Locales}).  Each locale specifies a particular multibyte
character code and a particular wide character code.  The choice of locale
influences the behavior of the conversion functions in the library.

Some locales support neither wide characters nor nontrivial multibyte
characters.  In these locales, the library conversion functions still
work, even though what they do is basically trivial.

If you select a new locale for character classification, the internal
shift state maintained by these functions can become confused, so it's
not a good idea to change the locale while you are in the middle of
processing a string.

@node Multibyte Char Intro, Wide Char Intro, Locales and Extended Chars, Extended Characters
@section Multibyte Characters
@cindex multibyte characters

In the ordinary ASCII code, a sequence of characters is a sequence of
bytes, and each character is one byte.  This is very simple, but
allows for only 256 distinct characters.

In a @dfn{multibyte character code}, a sequence of characters is a
sequence of bytes, but each character may occupy one or more consecutive
bytes of the sequence.

@cindex basic byte sequence
There are many different ways of designing a multibyte character code;
different systems use different codes.  To specify a particular code
means designating the @dfn{basic} byte sequences---those which represent
a single character---and what characters they stand for.  A code that a
computer can actually use must have a finite number of these basic
sequences, and typically none of them is more than a few characters
long.

These sequences need not all have the same length.  In fact, many of
them are just one byte long.  Because the basic ASCII characters in the
range from @code{0} to @code{0177} are so important, they stand for
themselves in all multibyte character codes.  That is to say, a byte
whose value is @code{0} through @code{0177} is always a character in
itself.  The characters which are more than one byte must always start
with a byte in the range from @code{0200} through @code{0377}.

The byte value @code{0} can be used to terminate a string, just as it is
often used in a string of ASCII characters.

Specifying the basic byte sequences that represent single characters
automatically gives meanings to many longer byte sequences, as more than
one character.  For example, if the two byte sequence @code{0205 049}
stands for the Greek letter alpha, then @code{0205 049 065} must stand
for an alpha followed by an @samp{A} (ASCII code 065), and @code{0205 049
0205 049} must stand for two alphas in a row.

If any byte sequence can have more than one meaning as a sequence of
characters, then the multibyte code is ambiguous---and no good.  The
codes that systems actually use are all unambiguous.

In most codes, there are certain sequences of bytes that have no meaning
as a character or characters.  These are called @dfn{invalid}.

The simplest possible multibyte code is a trivial one:

@quotation
The basic sequences consist of single bytes.
@end quotation

This particular code is equivalent to not using multibyte characters at
all.  It has no invalid sequences.  But it can handle only 256 different
characters.

Here is another possible code which can handle 9376 different
characters:

@quotation
The basic sequences consist of

@itemize @bullet
@item
single bytes with values in the range @code{0} through @code{0237}.

@item
two-byte sequences, in which both of the bytes have values in the range
from @code{0240} through @code{0377}.
@end itemize
@end quotation

@noindent
This code or a similar one is used on some systems to represent Japanese
characters.  The invalid sequences are those which consist of an odd
number of consecutive bytes in the range from @code{0240} through
@code{0377}.

Here is another multibyte code which can handle more distinct extended
characters---in fact, almost thirty million:

@quotation
The basic sequences consist of

@itemize @bullet
@item
single bytes with values in the range @code{0} through @code{0177}.

@item
sequences of up to four bytes in which the first byte is in the range
from @code{0200} through @code{0237}, and the remaining bytes are in the
range from @code{0240} through @code{0377}.
@end itemize
@end quotation

@noindent
In this code, any sequence that starts with a byte in the range
from @code{0240} through @code{0377} is invalid.

And here is another variant which has the advantage that removing the
last byte or bytes from a valid character can never produce another
valid character.  (This property is convenient when you want to search
strings for particular characters.)

@quotation
The basic sequences consist of

@itemize @bullet
@item
single bytes with values in the range @code{0} through @code{0177}.

@item
two-byte sequences in which the first byte is in the range from
@code{0200} through @code{0207}, and the second byte is in the range
from @code{0240} through @code{0377}.

@item
three-byte sequences in which the first byte is in the range from
@code{0210} through @code{0217}, and the other bytes are in the range
from @code{0240} through @code{0377}.

@item
four-byte sequences in which the first byte is in the range from
@code{0220} through @code{0227}, and the other bytes are in the range
from @code{0240} through @code{0377}.
@end itemize
@end quotation

@noindent
The list of invalid sequences for this code is long and not worth
stating in full; examples of invalid sequences include @code{0240} and
@code{0220 0300 065}.

The number of @emph{possible} multibyte codes is astronomical.  But a
given computer system will support at most a few different codes.  (One
of these codes may allow for thousands of different characters.)
Another computer system may support a completely different code.  The
library facilities described in this chapter are helpful because they
package up the knowledge of the details of a particular computer
system's multibyte code, so your programs need not know them.

You can use special standard macros to find out the maximum possible
number of bytes in a character in the currently selected multibyte
code with @code{MB_CUR_MAX}, and the maximum for @emph{any} multibyte
code supported on your computer with @code{MB_LEN_MAX}.

@comment limits.h
@comment ANSI
@deftypevr Macro int MB_LEN_MAX
This is the maximum length of a multibyte character for any supported
locale.  It is defined in @file{limits.h}.
@pindex limits.h
@end deftypevr

@comment stdlib.h
@comment ANSI
@deftypevr Macro int MB_CUR_MAX
This macro expands into a (possibly non-constant) positive integer
expression that is the maximum number of bytes in a multibyte character
in the current locale.  The value is never greater than @code{MB_LEN_MAX}.

@pindex stdlib.h
@code{MB_CUR_MAX} is defined in @file{stdlib.h}.
@end deftypevr

Normally, each basic sequence in a particular character code stands for
one character, the same character regardless of context.  Some multibyte
character codes have a concept of @dfn{shift state}; certain codes,
called @dfn{shift sequences}, change to a different shift state, and the
meaning of some or all basic sequences varies according to the current
shift state.  In fact, the set of basic sequences might even be
different depending on the current shift state.  @xref{Shift State}, for
more information on handling this sort of code.

What happens if you try to pass a string containing multibyte characters
to a function that doesn't know about them?  Normally, such a function
treats a string as a sequence of bytes, and interprets certain byte
values specially; all other byte values are ``ordinary''.  As long as a
multibyte character doesn't contain any of the special byte values, the
function should pass it through as if it were several ordinary
characters.

For example, let's figure out what happens if you use multibyte
characters in a file name.  The functions such as @code{open} and
@code{unlink} that operate on file names treat the name as a sequence of
byte values, with @samp{/} as the only special value.  Any other byte
values are copied, or compared, in sequence, and all byte values are
treated alike.  Thus, you may think of the file name as a sequence of
bytes or as a string containing multibyte characters; the same behavior
makes sense equally either way, provided no multibyte character contains
a @samp{/}.

@node Wide Char Intro, Wide String Conversion, Multibyte Char Intro, Extended Characters
@section Wide Character Introduction

@dfn{Wide characters} are much simpler than multibyte characters.  They
are simply characters with more than eight bits, so that they have room
for more than 256 distinct codes.  The wide character data type,
@code{wchar_t}, has a range large enough to hold extended character
codes as well as old-fashioned ASCII codes.

An advantage of wide characters is that each character is a single data
object, just like ordinary ASCII characters.  Wide characters also have
some disadvantages:

@itemize @bullet
@item
A program must be modified and recompiled in order to use wide
characters at all.

@item
Files of wide characters cannot be read by programs that expect ordinary
characters.
@end itemize

Wide character values @code{0} through @code{0177} are always identical
in meaning to the ASCII character codes.  The wide character value zero
is often used to terminate a string of wide characters, just as a single
byte with value zero often terminates a string of ordinary characters.

@comment stddef.h
@comment ANSI
@deftp {Data Type} wchar_t
This is the ``wide character'' type, an integer type whose range is
large enough to represent all distinct values in any extended character
set in the supported locales.  @xref{Locales}, for more information
about locales.  This type is defined in the header file @file{stddef.h}.
@pindex stddef.h
@end deftp

If your system supports extended characters, then each extended
character has both a wide character code and a corresponding multibyte
basic sequence.

@cindex code, character
@cindex character code
In this chapter, the term @dfn{code} is used to refer to a single
extended character object to emphasize the distinction from the
@code{char} data type.

@node Wide String Conversion, Length of Char, Wide Char Intro, Extended Characters
@section Conversion of Extended Strings
@cindex extended strings, converting representations
@cindex converting extended strings

@pindex stdlib.h
The @code{mbstowcs} function converts a string of multibyte characters