rfc2279.txt

来自「RFC 的详细文档！」· 文本 代码 · 共 564 行 · 第 1/2 页

Network Working Group                                       F. Yergeau
Request for Comments: 2279                           Alis Technologies
Obsoletes: 2044                                           January 1998
Category: Standards Track


              UTF-8, a transformation format of ISO 10646

Status of this Memo

   This document specifies an Internet standards track protocol for the
   Internet community, and requests discussion and suggestions for
   improvements.  Please refer to the current edition of the "Internet
   Official Protocol Standards" (STD 1) for the standardization state
   and status of this protocol.  Distribution of this memo is unlimited.

Copyright Notice

   Copyright (C) The Internet Society (1998).  All Rights Reserved.

Abstract

   ISO/IEC 10646-1 defines a multi-octet character set called the
   Universal Character Set (UCS) which encompasses most of the world's
   writing systems. Multi-octet characters, however, are not compatible
   with many current applications and protocols, and this has led to the
   development of a few so-called UCS transformation formats (UTF), each
   with different characteristics.  UTF-8, the object of this memo, has
   the characteristic of preserving the full US-ASCII range, providing
   compatibility with file systems, parsers and other software that rely
   on US-ASCII values but are transparent to other values. This memo
   updates and replaces RFC 2044, in particular addressing the question
   of versions of the relevant standards.

1.  Introduction

   ISO/IEC 10646-1 [ISO-10646] defines a multi-octet character set
   called the Universal Character Set (UCS), which encompasses most of
   the world's writing systems.  Two multi-octet encodings are defined,
   a four-octet per character encoding called UCS-4 and a two-octet per
   character encoding called UCS-2, able to address only the first 64K
   characters of the UCS (the Basic Multilingual Plane, BMP), outside of
   which there are currently no assignments.

   It is noteworthy that the same set of characters is defined by the
   Unicode standard [UNICODE], which further defines additional
   character properties and other application details of great interest
   to implementors, but does not have the UCS-4 encoding.  Up to the



Yergeau                     Standards Track                     [Page 1]

RFC 2279                         UTF-8                      January 1998


   present time, changes in Unicode and amendments to ISO/IEC 10646 have
   tracked each other, so that the character repertoires and code point
   assignments have remained in sync.  The relevant standardization
   committees have committed to maintain this very useful synchronism.

   The UCS-2 and UCS-4 encodings, however, are hard to use in many
   current applications and protocols that assume 8 or even 7 bit
   characters.  Even newer systems able to deal with 16 bit characters
   cannot process UCS-4 data. This situation has led to the development
   of so-called UCS transformation formats (UTF), each with different
   characteristics.

   UTF-1 has only historical interest, having been removed from ISO/IEC
   10646.  UTF-7 has the quality of encoding the full BMP repertoire
   using only octets with the high-order bit clear (7 bit US-ASCII
   values, [US-ASCII]), and is thus deemed a mail-safe encoding
   ([RFC2152]).  UTF-8, the object of this memo, uses all bits of an
   octet, but has the quality of preserving the full US-ASCII range:
   US-ASCII characters are encoded in one octet having the normal US-
   ASCII value, and any octet with such a value can only stand for an
   US-ASCII character, and nothing else.

   UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire
   into pairs of UCS-2 values from a reserved range.  UTF-16 impacts
   UTF-8 in that UCS-2 values from the reserved range must be treated
   specially in the UTF-8 transformation.

   UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of
   octets, where the number of octets, and the value of each, depend on
   the integer value assigned to the character in ISO/IEC 10646.  This
   transformation format has the following characteristics (all values
   are in hexadecimal):

   -  Character values from 0000 0000 to 0000 007F (US-ASCII repertoire)
      correspond to octets 00 to 7F (7 bit US-ASCII values). A direct
      consequence is that a plain ASCII string is also a valid UTF-8
      string.

   -  US-ASCII values do not appear otherwise in a UTF-8 encoded
      character stream.  This provides compatibility with file systems
      or other software (e.g. the printf() function in C libraries) that
      parse based on US-ASCII values but are transparent to other
      values.

   -  Round-trip conversion is easy between UTF-8 and either of UCS-4,
      UCS-2.





Yergeau                     Standards Track                     [Page 2]

RFC 2279                         UTF-8                      January 1998


   -  The first octet of a multi-octet sequence indicates the number of
      octets in the sequence.

   -  The octet values FE and FF never appear.

   -  Character boundaries are easily found from anywhere in an octet
      stream.

   -  The lexicographic sorting order of UCS-4 strings is preserved.  Of
      course this is of limited interest since the sort order is not
      culturally valid in either case.

   -  The Boyer-Moore fast search algorithm can be used with UTF-8 data.

   -  UTF-8 strings can be fairly reliably recognized as such by a
      simple algorithm, i.e. the probability that a string of characters
      in any other encoding appears as valid UTF-8 is low, diminishing
      with increasing string length.

   UTF-8 was originally a project of the X/Open Joint
   Internationalization Group XOJIG with the objective to specify a File
   System Safe UCS Transformation Format [FSS-UTF] that is compatible
   with UNIX systems, supporting multilingual text in a single encoding.
   The original authors were Gary Miller, Greger Leijonhufvud and John
   Entenmann.  Later, Ken Thompson and Rob Pike did significant work for
   the formal UTF-8.

   A description can also be found in Unicode Technical Report #4 and in
   the Unicode Standard, version 2.0 [UNICODE].  The definitive
   reference, including provisions for UTF-16 data within UTF-8, is
   Annex R of ISO/IEC 10646-1 [ISO-10646].

2.  UTF-8 definition

   In UTF-8, characters are encoded using sequences of 1 to 6 octets.
   The only octet of a "sequence" of one has the higher-order bit set to
   0, the remaining 7 bits being used to encode the character value. In
   a sequence of n octets, n>1, the initial octet has the n higher-order
   bits set to 1, followed by a bit set to 0.  The remaining bit(s) of
   that octet contain bits from the value of the character to be
   encoded.  The following octet(s) all have the higher-order bit set to
   1 and the following bit set to 0, leaving 6 bits in each to contain
   bits from the character to be encoded.

   The table below summarizes the format of these different octet types.
   The letter x indicates bits available for encoding bits of the UCS-4
   character value.




Yergeau                     Standards Track                     [Page 3]

RFC 2279                         UTF-8                      January 1998


   UCS-4 range (hex.)           UTF-8 octet sequence (binary)
   0000 0000-0000 007F   0xxxxxxx
   0000 0080-0000 07FF   110xxxxx 10xxxxxx
   0000 0800-0000 FFFF   1110xxxx 10xxxxxx 10xxxxxx

   0001 0000-001F FFFF   11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
   0020 0000-03FF FFFF   111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
   0400 0000-7FFF FFFF   1111110x 10xxxxxx ... 10xxxxxx

   Encoding from UCS-4 to UTF-8 proceeds as follows:

   1) Determine the number of octets required from the character value
      and the first column of the table above.  It is important to note
      that the rows of the table are mutually exclusive, i.e. there is
      only one valid way to encode a given UCS-4 character.

   2) Prepare the high-order bits of the octets as per the second column
      of the table.

   3) Fill in the bits marked x from the bits of the character value,
      starting from the lower-order bits of the character value and
      putting them first in the last octet of the sequence, then the
      next to last, etc. until all x bits are filled in.

      The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can be
      obtained from the above, in principle, by simply extending each
      UCS-2 character with two zero-valued octets.  However, pairs of
      UCS-2 values between D800 and DFFF (surrogate pairs in Unicode
      parlance), being actually UCS-4 characters transformed through
      UTF-16, need special treatment: the UTF-16 transformation must be
      undone, yielding a UCS-4 character that is then transformed as
      above.

      Decoding from UTF-8 to UCS-4 proceeds as follows:

   1) Initialize the 4 octets of the UCS-4 character with all bits set
      to 0.

   2) Determine which bits encode the character value from the number of
      octets in the sequence and the second column of the table above
      (the bits marked x).

   3) Distribute the bits from the sequence to the UCS-4 character,
      first the lower-order bits from the last octet of the sequence and
      proceeding to the left until no x bits are left.

      If the UTF-8 sequence is no more than three octets long, decoding
      can proceed directly to UCS-2.



Yergeau                     Standards Track                     [Page 4]

RFC 2279                         UTF-8                      January 1998


        NOTE -- actual implementations of the decoding algorithm above
        should protect against decoding invalid sequences.  For
        instance, a naive implementation may (wrongly) decode the
        invalid UTF-8 sequence C0 80 into the character U+0000, which
        may have security consequences and/or cause other problems.  See
        the Security Considerations section below.

   A more detailed algorithm and formulae can be found in [FSS_UTF],
   [UNICODE] or Annex R to [ISO-10646].

3.  Versions of the standards

   ISO/IEC 10646 is updated from time to time by published amendments;
   similarly, different versions of the Unicode standard exist: 1.0, 1.1
   and 2.0 as of this writing.  Each new version obsoletes and replaces
   the previous one, but implementations, and more significantly data,
   are not updated instantly.

   In general, the changes amount to adding new characters, which does
   not pose particular problems with old data.  Amendment 5 to ISO/IEC
   10646, however, has moved and expanded the Korean Hangul block,
   thereby making any previous data containing Hangul characters invalid
   under the new version.  Unicode 2.0 has the same difference from
   Unicode 1.1. The official justification for allowing such an
   incompatible change was that no implementations and no data
   containing Hangul existed, a statement that is likely to be true but
   remains unprovable.  The incident has been dubbed the "Korean mess",
   and the relevant committees have pledged to never, ever again make
   such an incompatible change.

   New versions, and in particular any incompatible changes, have q
   conseuences regarding MIME character encoding labels, to be discussed
   in section 5.

4.  Examples

   The UCS-2 sequence "A<NOT IDENTICAL TO><ALPHA>." (0041, 2262, 0391,
   002E) may be encoded in UTF-8 as follows:

   41 E2 89 A2 CE 91 2E

   The UCS-2 sequence representing the Hangul characters for the Korean
   word "hangugo" (D55C, AD6D, C5B4) may be encoded as follows:

   ED 95 9C EA B5 AD EC 96 B4






Yergeau                     Standards Track                     [Page 5]