iconv.tex

Newlib 嵌入式 C库 标准实现代码

@node Iconv
@chapter Encoding conversions (@file{iconv.h})

This chapter describes the Newlib iconv library.
The iconv functions declarations are in
@file{iconv.h}.

@menu
* iconv::                           Encoding conversion routines
* Introduction::                    Introduction to iconv and encodings
* Supported encodings::             The list of currently supported encodings
* iconv design decisions::          General iconv library design issues
* iconv configuration::             iconv-related configure script options
* Encoding names::                  How encodings are named.
* CCS tables::                      CCS tables format and 'mktbl.pl' Perl script
* CES converters::                  CES converters description
* The encodings description file::  The 'encoding.deps' file and 'mkdeps.pl'
* How to add new encoding::         The steps to add new encoding support
* The locale support interfaces::   Locale-related iconv interfaces
* Contact::                         The author contact
@end menu

@page
@include iconv/iconv.def

@page
@node Introduction
@section Introduction
@findex encoding
@findex character set
@findex charset
@findex CES
@findex CCS
@*
The iconv library is intended to convert characters from one encoding to
another. It implements iconv(), iconv_open() and iconv_close()
calls, which are defined by the Single Unix Specification.

@*
In addition to these user-level interfaces, the iconv library also has
several useful interfaces which are needed to support coding
capabilities of the Newlib Locale infrastructure.  Since Locale 
support also needs to
convert various character sets to and from the @emph{wide characters
set}, the iconv library shares it's capabilities with the Newlib Locale
subsystem. Moreover, the iconv library supports several features which are
only needed for the Locale infrastructure (for example, the MB_CUR_MAX value).

@*
The Newlib iconv library was created using concepts from another iconv
library implemented by Konstantin Chuguev (ver 2.0). The Newlib iconv library
was rewritten from scratch and contains a lot of improvements with respect to
the original iconv library. 

@*
Terms like @dfn{encoding} or @dfn{character set} aren't well defined and
are often used with various meanings. The following are the definitions of terms
which are used in this documentation as well as in the iconv library
implementation:

@itemize @bullet
@item
@dfn{encoding} - a machine representation of characters by means of bits;

@item
@dfn{Character Set} or @dfn{Charset} - just a collection of
characters, i.e. the encoding is the machine representation of the character set; 

@item
@dfn{CCS} (@dfn{Coded Character Set}) - a mapping from an character set to a
set of integers @dfn{character codes};

@item
@dfn{CES} (@dfn{Character Encoding Scheme}) - a mapping from a set of character
codes to a sequence of bytes;
@end itemize

@*
Users usually deal with encodings, for example, KOI8-R, Unicode, UTF-8,
ASCII, etc. Encodings are formed by the following chain of steps:

@enumerate
@item
User has a set of characters which are specific to his or her language (character set).

@item
Each character from this set is uniquely numbered, resulting in an CCS.

@item
Each number from the CCS is converted to a sequence of bits or bytes by means
of a CES and form some encoding. Thus, CES may be considered as a
function of CCS which produces some encoding. Note, that CES may be
applied to more than one CCS.
@end enumerate

@*
Thus, an encoding may be considered as one or more CCS + CES.

@*
Sometimes, there is no CES and in such cases encoding is equivalent
to CCS, e.g. KOI8-R or ASCII.

@*
An example of a more complicated encoding is UTF-8 which is the UCS
(or Unicode) CCS plus the UTF-8 CES.

@*
The following is a brief list of iconv library features:
@itemize
@item
Generic architecture;
@item
Locale infrastructure support;
@item
Automatic generation of the program code which handles
CES/CCS/Encoding/Names/Aliases dependencies;
@item
The ability to choose size- or speed-optimazed
configuration;
@item
The ability to exclude a lot of unneeded code and data from the linking step.
@end itemize




@page
@node Supported encodings
@section Supported encodings
@findex big5
@findex cp775
@findex cp850
@findex cp852
@findex cp855
@findex cp866
@findex euc_jp
@findex euc_kr
@findex euc_tw
@findex iso_8859_1
@findex iso_8859_10
@findex iso_8859_11
@findex iso_8859_13
@findex iso_8859_14
@findex iso_8859_15
@findex iso_8859_2
@findex iso_8859_3
@findex iso_8859_4
@findex iso_8859_5
@findex iso_8859_6
@findex iso_8859_7
@findex iso_8859_8
@findex iso_8859_9
@findex iso_ir_111
@findex koi8_r
@findex koi8_ru
@findex koi8_u
@findex koi8_uni
@findex ucs_2
@findex ucs_2_internal
@findex ucs_2be
@findex ucs_2le
@findex ucs_4
@findex ucs_4_internal
@findex ucs_4be
@findex ucs_4le
@findex us_ascii
@findex utf_16
@findex utf_16be
@findex utf_16le
@findex utf_8
@findex win_1250
@findex win_1251
@findex win_1252
@findex win_1253
@findex win_1254
@findex win_1255
@findex win_1256
@findex win_1257
@findex win_1258
@*
The following is the list of currently supported encodings. The first column
corresponds to the encoding name, the second column is the list of aliases,
the third column is its CES and CCS components names, and the fourth column
is a short description.

@multitable @columnfractions .20 .26 .24 .30
@item
Name
@tab
Aliases
@tab
CES/CCS
@tab
Short description
@item
@tab
@tab
@tab


@item
big5
@tab
csbig5, big_five, bigfive, cn_big5, cp950
@tab
table_pcs / big5, us_ascii 
@tab
The encoding for the Traditional Chinese.


@item
cp775
@tab
ibm775, cspc775baltic
@tab
table / cp775
@tab
The updated version of CP 437 that supports the balitic languages.


@item
cp850
@tab
ibm850, 850, cspc850multilingual
@tab
table / cp850
@tab
IBM 850 - the updated version of CP 437 where several Latin 1 characters have been
added instead of some less-often used characters like the line-drawing
and the greek ones.


@item
cp852
@tab
ibm852, 852, cspcp852
@tab
@tab
IBM 852 - the updated version of CP 437 where several Latin 2 characters have been added
instead of some less-often used characters like the line-drawing and the greek ones.


@item
cp855
@tab
ibm855, 855, csibm855
@tab
table / cp855
@tab
IBM 855 - the updated version of CP 437 that supports Cyrillic.


@item
cp866
@tab
866, IBM866, CSIBM866
@tab
table / cp866
@tab
IBM 866 - the updated version of CP 855 which follows more the logical Russian alphabet 
ordering of the alternative variant that is preferred by many Russian users.


@item
euc_jp
@tab
eucjp
@tab
euc / jis_x0208_1990, jis_x0201_1976, jis_x0212_1990
@tab
EUC-JP - The EUC for Japanese.


@item
euc_kr
@tab
euckr
@tab
euc / ksx1001
@tab
EUC-KR - The EUC for Korean.


@item
euc_tw
@tab
euctw
@tab
euc / cns11643_plane1, cns11643_plane2, cns11643_plane14
@tab
EUC-TW - The EUC for Traditional Chinese.


@item
iso_8859_1
@tab
iso8859_1, iso88591, iso_8859_1:1987, iso_ir_100, latin1, l1, ibm819, cp819, csisolatin1
@tab
table / iso_8859_1
@tab
ISO 8859-1:1987 - Latin 1, West European.


@item
iso_8859_10
@tab
iso_8859_10:1992, iso_ir_157, iso885910, latin6, l6, csisolatin6, iso8859_10
@tab
table / iso_8859_10
@tab
ISO 8859-10:1992 - Latin 6, Nordic.


@item
iso_8859_11
@tab
iso8859_11, iso885911
@tab
table / iso_8859_11
@tab
ISO 8859-11 - Thai.


@item
iso_8859_13
@tab
iso_8859_13:1998, iso8859_13, iso885913
@tab
table / iso_8859_13
@tab
ISO 8859-13:1998 - Latin 7, Baltic Rim.


@item
iso_8859_14
@tab
iso_8859_14:1998, iso885914, iso8859_14
@tab
table / iso_8859_14
@tab
ISO 8859-14:1998 - Latin 8, Celtic.


@item
iso_8859_15
@tab
iso885915, iso_8859_15:1998, iso8859_15, 
@tab
table / iso_8859_15
@tab
ISO 8859-15:1998 - Latin 9, West Europe, successor of Latin 1.


@item
iso_8859_2
@tab
iso8859_2, iso88592, iso_8859_2:1987, iso_ir_101, latin2, l2, csisolatin2
@tab
table / iso_8859_2
@tab
ISO 8859-2:1987 - Latin 2, East European.


@item
iso_8859_3
@tab
iso_8859_3:1988, iso_ir_109, iso8859_3, latin3, l3, csisolatin3, iso88593
@tab
table / iso_8859_3
@tab
ISO 8859-3:1988 - Latin 3, South European.


@item
iso_8859_4
@tab
iso8859_4, iso88594, iso_8859_4:1988, iso_ir_110, latin4, l4, csisolatin4
@tab
table / iso_8859_4
@tab
ISO 8859-4:1988 - Latin 4, North European.


@item
iso_8859_5
@tab
iso8859_5, iso88595, iso_8859_5:1988, iso_ir_144, cyrillic, csisolatincyrillic
@tab
table / iso_8859_5
@tab
ISO 8859-5:1988 - Cyrillic.


@item
iso_8859_6
@tab
iso_8859_6:1987, iso_ir_127, iso8859_6, ecma_114, asmo_708, arabic, csisolatinarabic, iso88596
@tab
table / iso_8859_6
@tab
ISO i8859-6:1987 - Arabic.


@item
iso_8859_7
@tab
iso_8859_7:1987, iso_ir_126, iso8859_7, elot_928, ecma_118, greek, greek8, csisolatingreek, iso88597
@tab
table / iso_8859_7
@tab
ISO 8859-7:1987 - Greek.


@item
iso_8859_8
@tab
iso_8859_8:1988, iso_ir_138, iso8859_8, hebrew, csisolatinhebrew, iso88598
@tab
table / iso_8859_8
@tab
ISO 8859-8:1988 - Hebrew.


@item
iso_8859_9
@tab
iso_8859_9:1989, iso_ir_148, iso8859_9, latin5, l5, csisolatin5, iso88599
@tab
table / iso_8859_9
@tab
ISO 8859-9:1989 - Latin 5, Turkish.


@item
iso_ir_111
@tab
ecma_cyrillic, koi8_e, koi8e, csiso111ecmacyrillic
@tab
table / iso_ir_111
@tab
ISO IR 111/ECMA Cyrillic.


@item
koi8_r
@tab
cskoi8r, koi8r, koi8
@tab
table / koi8_r
@tab
RFC 1489 Cyrillic.


@item
koi8_ru
@tab
koi8ru
@tab
table / koi8_ru
@tab
The obsolete Ukrainian.


@item
koi8_u
@tab
koi8u
@tab
table / koi8_u
@tab
RFC 2319 Ukrainian.


@item
koi8_uni
@tab
koi8uni
@tab
table / koi8_uni
@tab
KOI8 Unified.


@item
ucs_2
@tab
ucs2, iso_10646_ucs_2, iso10646_ucs_2, iso_10646_ucs2, iso10646_ucs2, iso10646ucs2, csUnicode
@tab
ucs_2 / (UCS)
@tab
ISO-10646-UCS-2. Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported).


@item
ucs_2_internal
@tab
ucs2_internal, ucs_2internal, ucs2internal
@tab
ucs_2_internal / (UCS)
@tab
ISO-10646-UCS-2 in system byte order.
NBSP is always interpreted as NBSP (BOM isn't supported).


@item
ucs_2be
@tab
ucs2be
@tab
ucs_2 / (UCS)
@tab
Big Endian version of ISO-10646-UCS-2 (in fact, equivalent to ucs_2).
Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported).


@item
ucs_2le
@tab
ucs2le
@tab
ucs_2 / (UCS)
@tab
Little Endian version of ISO-10646-UCS-2.
Little Endian, NBSP is always interpreted as NBSP (BOM isn't supported).


@item
ucs_4
@tab
ucs4, iso_10646_ucs_4, iso10646_ucs_4, iso_10646_ucs4, iso10646_ucs4, iso10646ucs4
@tab
ucs_4 / (UCS)
@tab
ISO-10646-UCS-4. Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported).


@item
ucs_4_internal
@tab
ucs4_internal, ucs_4internal, ucs4internal
@tab
ucs_4_internal / (UCS)
@tab
ISO-10646-UCS-4 in system byte order.
NBSP is always interpreted as NBSP (BOM isn't supported).


@item
ucs_4be
@tab
ucs4be
@tab
ucs_4 / (UCS)
@tab
Big Endian version of ISO-10646-UCS-4 (in fact, equivalent to ucs_4).
Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported).


@item
ucs_4le
@tab
ucs4le
@tab
ucs_4 / (UCS)
@tab
Little Endian version of ISO-10646-UCS-4.
Little Endian, NBSP is always interpreted as NBSP (BOM isn't supported).


@item
us_ascii