iconv.tex

Newlib 嵌入式 C库 标准实现代码

@tab
ansi_x3.4_1968, ansi_x3.4_1986, iso_646.irv:1991, ascii, iso646_us, us, ibm367, cp367, csascii
@tab
us_ascii / (ASCII)
@tab
7-bit ASCII.


@item
utf_16
@tab
utf16
@tab
utf_16 / (UCS)
@tab
RFC 2781 UTF-16. The very first NBSP code in stream is interpreted as BOM.


@item
utf_16be
@tab
utf16be
@tab
utf_16 / (UCS)
@tab
Big Endian version of RFC 2781 UTF-16.
NBSP is always interpreted as NBSP (BOM isn't supported).


@item
utf_16le
@tab
utf16le
@tab
utf_16 / (UCS)
@tab
Little Endian version of RFC 2781 UTF-16.
NBSP is always interpreted as NBSP (BOM isn't supported).


@item
utf_8
@tab
utf8
@tab
utf_8 / (UCS)
@tab
RFC 3629 UTF-8.


@item
win_1250
@tab
cp1250
@tab
@tab
Win-1250 Croatian.


@item
win_1251
@tab
cp1251
@tab
table / win_1251
@tab
Win-1251 - Cyrillic.


@item
win_1252
@tab
cp1252
@tab
table / win_1252
@tab
Win-1252 - Latin 1.


@item
win_1253
@tab
cp1253
@tab
table / win_1253
@tab
Win-1253 - Greek.


@item
win_1254
@tab
cp1254
@tab
table / win_1254
@tab
Win-1254 - Turkish.


@item
win_1255
@tab
cp1255
@tab
table / win_1255
@tab
Win-1255 - Hebrew.


@item
win_1256
@tab
cp1256
@tab
table / win_1256
@tab
Win-1256 - Arabic.


@item
win_1257
@tab
cp1257
@tab
table / win_1257
@tab
Win-1257 - Baltic.


@item
win_1258
@tab
cp1258
@tab
table / win_1258
@tab
Win-1258 - Vietnamese7 that supports Cyrillic.
@end multitable





@page
@node iconv design decisions
@section iconv design decisions
@findex CCS table
@findex CES converter
@findex Speed-optimized tables
@findex Size-optimized tables
@*
The first iconv library design issue arises when considering the
following two design approaches:

@enumerate
@item
Have modules which implement conversion from the encoding A to the encoding B
and vice versa i.e., one conversion module relates to any two encodings.
@item
Have modules which implement conversion from the encoding A to the fixed
encoding C and vice versa i.e., one conversion module relates to any
one encoding A and one fixed encoding C. In this case, to convert from
the encoding A to the encoding B, two modules are needed (in order to convert
from A to C and then from C to B).
@end enumerate

@*
It's obvious, that we have tradeoff between commonality/flexibility and
efficiency: the first method is more efficient since it converts
directly; however, it isn't so flexible since for each
encoding pair a distinct module is needed.

@*
The Newlib iconv model uses the second method and always converts through the 32-bit
UCS but its design also allows one to write specialized conversion
modules if the conversion speed is critical.

@*
The second design issue is how to break down (decompose) encodings.
The Newlib iconv library uses the fact that any encoding may be
considered as one or more CCS plus a CES. It also decomposes its
conversion modules on @dfn{CES converter} plus one or more @dfn{CCS
tables}. CCS tables map CCS to UCS and vice versa; the CES converters
map CCS to the encoding and vice versa.

@*
As the example, let's consider the conversion from the big5 encoding to
the EUC-TW encoding. The big5 encoding may be decomposed to the ASCII and BIG5
CCS-es plus the BIG5 CES. EUC-TW may be decomposed on the CNS11643_PLANE1, CNS11643_PLANE2,
and CNS11643_PLANE14 CCS-es plus the EUC CES.

@*
The euc_jp -> big5 conversion is performed as follows:

@enumerate
@item
The EUC converter performs the EUC-TW encoding to the corresponding CCS-es
transformation (CNS11643_PLANE1, CNS11643_PLANE2 and CNS11643_PLANE14
CCS-es);
@item
The obtained CCS codes are transformed to the UCS codes using the CNS11643_PLANE1,
CNS11643_PLANE2 and CNS11643_PLANE14 CCS tables;
@item
The resulting UCS codes are transformed to the ASCII and BIG5 codes using
the corresponding CCS tables;
@item
The obtained CCS codes are transformed to the big5 encoding using the corresponding
CES converter.
@end enumerate

@*
Analogously, the backward conversion is performed as follows:

@enumerate
@item
The BIG5 converter performs the big5 encoding to the corresponding CCS-es transformation
(the ASCII and BIG5 CCS-es);
@item
The obtained CCS codes are transformed to the UCS codes using the ASCII and BIG5 CCS tables;
@item
The resulting UCS codes are transformed to the ASCII and BIG5 codes using
the corresponding CCS tables;
@item
The obtained CCS codes are transformed to the EUC-TW encoding using the corresponding
CES converter.
@end enumerate

@*
Note, the above is just an example and real names (which are implemented
in the Newlib iconv) of the CES converters and the CCS tables are slightly different.

@*
The third design issue also relates to flexibility. Obviously, it isn't
desirable to always link all the CES converters and the CCS tables to the library
but instead, we want to be able to load the needed converters and tables
dynamically on demand. This isn't a problem on "big" machines such as
a PC, but it may be very problematical within "small" embedded systems.

@*
Since the CCS tables are just data, it is possible to load them
dynamically from external files.  The CES converters, on the other hand
are algorithms with some code so a dynamic library loading 
capability is required.

@*
Apart from possible restrictions applied by embedded systems (small
RAM for example), Newlib itself has no dynamic library support and
therefore, all the CES converters which will ever be used must be linked into
the library.   However, loading of the dynamic CCS tables is possible and is
implemented in the Newlib iconv library.  It may be enabled via the Newlib
configure script options.

@*
The next design issue is fine-tuning the iconv library
configuration.  One important ability is for iconv to not link all it's
converters and tables (if dynamic loading is not enabled) but instead,
enable only those encodings which are specified at configuration
time (see the section about the configure script options).

@*
In addition, the Newlib iconv library configure options distinguish between
conversion directions. This means that not only are supported encodings
selectable, the conversion direction is as well. For example, if user wants
the configuration which allows conversions from UTF-8 to UTF-16 and
doesn't plan using the "UTF-16 to UTF-8" conversions, he or she can 
enable only
this conversion direction (i.e., no "UTF-16 -> UTF-8"-related code will
be included) thus, saving some memory (note, that such technique allows to
exclude one half of a CCS table from linking which may be big enough).

@*
One more design aspect are the speed- and size- optimized tables. Users can
select between them using configure script options. The
speed-optimized CCS tables are the same as the size-optimized ones in
case of 8-bit CCS (e.g.m KOI8-R), but for 16-bit CCS-es the size-optimized
CCS tables may be 1.5 to 2 times less then the speed-optimized ones. On the
other hand, conversion with speed tables is several times faster.

@*
Its worth to stress that the new encoding support can't be
dynamically added into an already compiled Newlib library, even if it
needs only an additional CCS table and iconv is configured to use
the external files with CCS tables (this isn't the fundamental restriction
and the possibility to add new Table-based encoding support dynamically, by
means of just adding new .cct file, may be easily added).

@*
Theoretically, the compiled-in CCS tables should be more appropriate for
embedded systems than dynamically loaded CCS tables.  This is because the compiled-in tables are read-only and can be placed in ROM
whereas dynamic loading requires RAM.  Moreover, in the current iconv
implementation, a distinct copy of the dynamic CCS file is loaded for each opened iconv descriptor even in case of the same encoding.
This means, for example, that if two iconv descriptors for
"KOI8-R -> UCS-4BE" and "KOI8-R -> UTF-16BE" are opened, two copies of
koi8-r .cct file will be loaded (actually, iconv loads only the needed part
of these files).  On the other hand, in the case of compiled-in CCS tables, there will always be only one copy.

@page
@node iconv configuration
@section iconv configuration
@findex iconv configuration
@findex --enable-newlib-iconv-encodings
@findex --enable-newlib-iconv-from-encodings
@findex --enable-newlib-iconv-to-encodings
@findex --enable-newlib-iconv-external-ccs
@findex NLSPATH
@*
To enable an encoding, the @emph{--enable-newlib-iconv-encodings} configure
script option should be used. This option accepts a comma-separated list
of @emph{encodings} that should be enabled. The option enables each encoding in both
("to" and "from") directions.

@*
The @option{--enable-newlib-iconv-from-encodings} configure script option enables
"from" support for each encoding that was passed to it.

@*
The @option{--enable-newlib-iconv-to-encodings} configure script option enables
"to" support for each encoding that was passed to it.

@*
Example: if user plans only the "KOI8-R -> UTF-8", "UTF-8 -> ISO-8859-5" and
"KOI8-R -> UCS-2" conversions, the most optimal way (minimal iconv
code and data will be linked) is to configure Newlib with the following
options:
@*
@code{--enable-newlib-iconv-encodings=UTF-8
--enable-newlib-iconv-from-encodings=KOI8-R
--enable-newlib-iconv-to-encodings=UCS-2,ISO-8859-5}
@*
which is the same as
@*
@code{--enable-newlib-iconv-from-encodings=KOI8-R,UTF-8
--enable-newlib-iconv-to-encodings=UCS-2,ISO-8859-5,UTF-8}
@*
User may also just use the
@*
@code{--enable-newlib-iconv-encodings=KOI8-R,ISO-8859-5,UTF-8,UCS-2}
@*
configure script option, but it isn't so optimal since there will be
some unneeded data and code.

@*
The @option{--enable-newlib-iconv-external-ccs} option enables iconv's
capabilities to work with the external CCS files.

@*
The @option{--enable-target-optspace} Newlib configure script option also affects
the iconv library. If this option is present, the library uses the size
optimized CCS tables. This means, that only the size-optimized CCS
tables will be linked or, if the
@option{--enable-newlib-iconv-external-ccs} configure script option was used,
the iconv library will load the size-optimized tables. If the
@option{--enable-target-optspace}configure script option is disabled,
the speed-optimized CCS tables are used.

@*
Note: .cct files are searched by iconv_open in the $NLSPATH/iconv_data/ directory.
Thus, the NLSPATH environment variable should be set.





@page
@node Encoding names
@section Encoding names
@findex encoding name
@findex encoding alias
@findex normalized name
@*
Each encoding has one @dfn{name} and a number of @dfn{aliases}. When
user works with the iconv library (i.e., when the @code{iconv_open} call
is used) both name or aliases may be used. The same is when encoding
names are used in configure script options.

@*
Names and aliases may be specified in any case (small or capital
letters) and the @kbd{-} symbol is equivalent to the @kbd{_} symbol.
Also, when working with the iconv library,

@*
Internally the Newlib iconv library always converts aliases to names. It
also converts names and aliases in the @dfn{normalized} form which means
that all capital letters are converted to small letters and the @kbd{-}
symbols are converted to @kbd{_} symbols.




@page
@node CCS tables
@section CCS tables
@findex Size-optimized CCS table
@findex Speed-optimized CCS table
@findex mktbl.pl Perl script
@findex .cct files
@findex The CCT tables source files
@findex CCS source files
@*
The iconv library stores files with CCS tables in the the @emph{ccs/}
subdirectory. The CCS tables for any CCS may be kept in two forms - in the binary form
(@dfn{.cct files}, see the @emph{ccs/binary/} subdirectory) and in form
of compilable .c source files. The .cct files are only used when the
@option{--enable-newlib-iconv-external-ccs} configure script option is enabled.
The .c files are linked to the Newlib library if the corresponding
encoding is enabled.

@*
As stated earlier, the Newlib iconv library performs all
conversions through the 32-bit UCS, but the codes which are used
in most CCS-es, fit into the first 16-bit subset of the 32-bit UCS set.
Thus, in order to make the CCS tables more compact, the 16-bit UCS-2 is
used instead of the 32-bit UCS-4.

@*
CCS tables may be 8- or 16-bit wide. 8-bit CCS tables map 8-bit CCS to
16-bit UCS-2 and vice versa while 16-bit CCS tables map
16-bit CCS to 16-bit UCS-2 and vice versa.
8-bit tables are small (in size) while 16-bit tables may be big enough.
Because of this, 16-bit CCS tables may be
either speed- or size-optimized. Size-optimized CCS tables are
smaller then speed-optimized ones, but the conversion process is
slower if the size-optimized CCS tables are used. 8-bit CCS tables have only
size-optimized variant.

Each CCS table (both speed- and size-optimized) consists of
@dfn{from_ucs} and @dfn{to_ucs} subtables. "from_ucs" subtable maps
UCS-2 codes to CCS codes, while "to_ucs" subtable maps CCS codes to
UCS-2 codes.

@*
Almost all 16-bit CCS tables contain less then 0xFFFF codes and
a lot of gaps exist.

@subsection Speed-optimized tables format
@*
In case of 8-bit speed-optimized CCS tables the "to_ucs" subtables format is
trivial - it is just the array of 256 16-bit UCS codes. Therefore, an
UCS-2 code @emph{Y} corresponding to a @emph{X} CCS code is calculates
as @emph{Y = to_ucs[X]}.

@*
Obviously, the simplest way to create the "from_ucs" table or the
16-bit "to_ucs" table is to use the huge 16-bit array like in case
of the 8-bit "to_ucs" table. But almost all the 16-bit CCS tables contain
less then 0xFFFF code maps and this fact may be exploited to reduce
the size of the CCS tables.

@*
In this chapter the "UCS-2 -> CCS" 8-bit CCS table format is described. The
16-bit "CCS -> UCS-2" CCS table format is the same, except the mapping
direction and the CCS bits number.

@*
In case of the 8-bit speed-optimized table the "from_ucs" subtable
corresponds the "from_ucs" array and has the following layout:

@*
from_ucs array:
@*
-------------------------------------
@*
0xFF mapping (2 bytes) (only for
8-bit table).
@*
-------------------------------------
@*
Heading block
@*
-------------------------------------
@*
Block 1
@*
-------------------------------------
@*
Block 2
@*
-------------------------------------
@*
  ...
@*
-------------------------------------
@*
Block N
@*
-------------------------------------

@*
The 0x0000-0xFFFF 16-bit code range is divided to 256 code subranges. Each
subrange is represented by an 256-element @dfn{block} (256 1-byte
elements or 256 2-byte element in case of 16-bit CCS table) with
elements which are equivalent to the CCS codes of this subrange.
If the "UCS-2 -> CCS" mapping has big enough gaps, some blocks will be
absent and there will be less then 256 blocks.

@*
Any element number @emph{m} of @dfn{the heading block} (which contains
256 2-byte elements) corresponds to the @emph{m}-th 256-element subrange.
If the subrange contains some codes, the value of the @emph{m}-th element of
the heading block contains the offset of the corresponding block in the
"from_ucs" array. If there is no codes in the subrange, the heading
block element contains 0xFFFF.

@*
If there are some gaps in a block, the corresponding block elements have
the 0xFF value. If there is an 0xFF code present in the CCS, it's mapping
is defined in the first 2-byte element of the "from_ucs" array.

@*
Having such a table format, the algorithm of searching the CCS code
@emph{X} which corresponds to the UCS-2 code @emph{Y} is as follows.

@*
@enumerate
@item If @emph{Y} is equivalent to the value of the first 2-byte element
of the "from_ucs" array, @emph{X} is 0xFF. Else, continue to search.

@item Calculate the block number: @emph{BlkN = (Y & 0xFF00) >> 8}.

@item If the heading block element with number @emph{BlkN} is 0xFFFF, there
is no corresponding CCS code (error, wrong input data). Else, fetch the
"flom_ucs" array index of the @emph{BlkN}-th block.

@item Calculate the offset of the @emph{X} code in its block: 
@emph{Xindex = Y & 0xFF}

@item If the @emph{Xintex}-th element of the block (which is equivalent to
@emph{from_ucs[BlkN+Xindex]}) value is 0xFF, there is no corresponding
CCS code (error, wrong input data). Else, @emph{X = from_ucs[BlkN+Xindex]}.
@end enumerate

@subsection Size-optimized tables format
@*
As it is stated above, size-optimized tables exist only for 16-bit CCS-es.
This is because there is too small difference between the speed-optimized
and the size-optimized table sizes in case of 8-bit CCS-es.

@*
Formats of the "to_ucs" and "from_ucs" subtables are equivalent in case of
size-optimized tables.

This sections describes the format of the "UCS-2 -> CCS" size-optimized
CCS table. The format of "CCS -> UCS-2" table is the same.

The idea of the size-optimized tables is to split the UCS-2 codes
("from" codes) on @dfn{ranges} (@dfn{range} is a number of consecutive UCS-2 codes).
Then CCS codes ("to" codes) are stored only for the codes from these
ranges. Distinct "from" codes, which have no range (@dfn{unranged codes}, are stored
together with the corresponding "to" codes.

@*
The following is the layout of the size-optimized table array:

@*
size_arr array:
@*
-------------------------------------
@*
Ranges number (2 bytes)
@*
-------------------------------------
@*
Unranged codes number (2 bytes)
@*
-------------------------------------
@*
Unranged codes array index (2 bytes)
@*
-------------------------------------
@*
Ranges indexes (triads)