draft-ietf-idn-nameprep-03.txt

bind-3.2.

Internet Draft                                          Paul Hoffman
draft-ietf-idn-nameprep-03.txt                            IMC & VPNC
February 24, 2001                                      Marc Blanchet
Expires in six months                                       ViaGenie

               Preparation of Internationalized Host Names

Status of this memo

This document is an Internet-Draft and is in full conformance with all
provisions of Section 10 of RFC2026.

Internet-Drafts are working documents of the Internet Engineering Task
Force (IETF), its areas, and its working groups. Note that other groups
may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference material
or to cite them other than as "work in progress."

To view the list Internet-Draft Shadow Directories, see
http://www.ietf.org/shadow.html.


Abstract

This document describes how to prepare internationalized host names for
use in the DNS. The steps include:
  - mapping characters to other characters, such as to change their case
  - normalizing the characters
  - excluding characters that are prohibited from appearing in
    internationalized host names
This document does not specify a wire protocol. This preparation should
be done before the DNS request.

1. Introduction

When expanding today's DNS to include internationalized host names,
those new names will be handled in many parts of the DNS. The
Internationalized Domain Name (IDN) Working Group's requirements
document [IDNReq] describes a framework for domain name handling as well
as requirements for the new names.

A user can enter a domain name into an application program in a myriad
of fashions. Depending on the input method, the characters entered in
the domain name may or may not be those that are allowed in
internationalized host names. Thus, there must be a way to normalized
the user's input before the name is resolved in the DNS.

It is a design goal of this document to allow users to enter host names
in applications and have the highest chance of getting the name correct.
Another, often conflicting, design goal is to allow as wide of a range
of characters as possible to be allowed in host names. The user should
not be limited to only entering exactly the characters that might have
been used, but to instead be able to enter characters that unambiguously
normalize to characters in the desired host name. Although it would be
easy to use the process in this step to "correct" perceived mis-features
or bugs in the current character standards, this document expressly does
not do so.

This document describes the steps needed to convert a name part from one
that is entered by the user to one that can be used in the DNS.

Within a fully-qualified domain name, some labels may be
internationalized, while others are not. This specification should be
applied to all internationalized labels. An application must be able to
recognize which part is internationalized; the method for such
recognition is outside of the scope of this document. Note that this
specification is harmless to the non-internationalized labels: when the
steps described here are applied to non-internationalized labels, the
label will not change.

1.1 Terminology

The key words "MUST", "SHALL", "REQUIRED", "SHOULD", "RECOMMENDED", and
"MAY" in this document are to be interpreted as described in RFC 2119
[RFC2119].

Examples in this document use the notation for code points and names
from the Unicode Standard [Unicode3] and ISO/IEC 10646 [ISO10646]. For
example, the letter "a" may be represented as either "U+0061" or "LATIN
SMALL LETTER A". In the lists of prohibited characters, the "U+" is left
off to make the lists easier to read. The names of character ranges are
shown in square brackets (such as "[SYMBOLS]") and do not come from the
standards.

Note: A glossary of terms used in Unicode and ISO/IEC 10646 can be found
in [Glossary]. Information on the 10646/Unicode character model can be
found in [CharModel].


2. Preparation Overview

The steps for preparing names are:

1) Input from the application service interface -- This can be done in
many ways and is not specified in this document

2) Map -- For each character in the input, check if it has a mapping
and, if so, replace it with its mapping. The mappings are a combination
of folding uppercase characters to lowercase and hyphen mapping. This is
described in Section 4.

3) Normalize -- Normalize the characters. This is described in Section
5.

4) Look for prohibited output -- Check for any characters that are not
allowed in the output. If any are found, return an error to the
application service interface. This is described in Section 6.

5) Resolution of the prepared name -- This must be specified in a
different IDN document.

The above steps MUST be performed in the order given in order to comply
with this specification.

The steps in this document have associated tables in the document. The
tables are derived from outside sources, and the derivation is briefly
described in the document. Although a great deal of effort has gone into
preparing the tables, there is a chance that the tables do not correctly
reflect the outside sources. Regardless of whether or not the tables
differ from the sources, implementations MUST use the tables in this
document for their processing. That is, if there is an error in the
tables, the tables must still be used. Future versions of this document
may include corrections and additions to the tables.


3. Mapping

Each character in the input stream is checked against the mapping table.
The mapping table can be found in Appendix E of this document. That
table includes all the steps described in the subsections below.

The mappings can be one-to-none, one-to-one, or one-to-many. That is,
some characters may be eliminated or replaced by more than one
character, and the output of this step might be shorter or longer than
the input. Because of this, an application MUST be prepared to receive a
longer or shorter string than the one input in the nameprep algorithm.

Rationale: Characters that are not wanted in internationalized name
parts can either be mapped to nothing in the mapping step, or cause an
error in the prohibition step. The general guideline used to pick
between the two outcomes was that removing alphabetic, non-protocol
characters be done in the mapping step, but all other removals be done
in the prohibition step. This allows for simple linguistic errors on the
part of an input mechanism to be caught in the mapping step, but to not
hide serious errors such as entering protocol characters or invisible
characters from the user.

3.1 Case mapping

The input string is case folded according to [UTR21]. For most
characters, this is the same thing as changing the input character to a
lowercase character. For some characters, however, more complex
transformations occur. The mapping table in Appendix E is derived by
applying the rules for equivalence classes from [UTR21].

Rationale: This step could have been "change all lowercase characters
into uppercase characters". However, the upper-to-lower folding was
chosen because most users of the Internet today enter host names in
lowercase.

3.2 Additional folding mappings

There are some characters that do not have mappings in [UTR21] but still
need processing. These characters include a few Greek characters and
many symbols that contain Latin characters. The list of characters to
add to the mapping table were determined by the following algorithm:

b = NormalizeWithKC(Fold(a));
c = NormalizeWithKC(Fold(b));
if c is not the same as b, add a mapping for "a to c".

Because NormalizeWithKC(Fold(c)) always equals c, the table is stable
from that point on.

3.3 Mapped out

The following characters are simply deleted from the input (that is,
they are mapped to nothing) because their presence or absence should not
make two domain names different.

Some characters are only useful in line-based text, and are otherwise
invisible and ignored.

00AD; SOFT HYPHEN
1806; MONGOLIAN TODO SOFT HYPHEN
200B; ZERO WIDTH SPACE
FEFF; ZERO WIDTH NO-BREAK SPACE

Variation selectors and cursive connectors select different glyphs, but
do not bear semantics.

180B; MONGOLIAN FREE VARIATION SELECTOR ONE
180C; MONGOLIAN FREE VARIATION SELECTOR TWO
180D; MONGOLIAN FREE VARIATION SELECTOR THREE
200C; ZERO WIDTH NON-JOINER
200D; ZERO WIDTH JOINER


4. Normalization

The output of the mapping step is normalized using form KC, as described
in [UTR15]. Using form KC instead of form C causes many characters that
are identical or near-identical to be converted into a single character.
Note that this specification refers to a specific version of [UTR15]. If
a later version of [UTR15] changes the algorithm used for normalizing,
that later version MUST NOT be used with this specification. Note that
it is likely that this specification will be revised if UTR15 is
changed, but until that happens, only the specified version of [UTR15]
must be used.


5. Prohibited Output

Before the text can be emitted, it must be checked for prohibited code
points. There is a variety of prohibited code points, as described in
this section.

One of the goals of IDN is to allow the widest possible set of host
names as long as those host names do not cause other problems, such as
conflict with other standards. Specifically, experience with current DNS
names have shown that there is a desire for host names that include
personal names, company names, and spoken phrases. A goal of this
section is to prohibit as few characters that might be used in these
contexts as possible.

The collected list of prohibited code points can be found in Appendix F
of this document. The list in Appendix F MUST be used by implementations
of this specification. If there are any discrepancies between the list
in Appendix F and subsections below, the list Appendix F always takes
precedence.

Some code points listed in one section would also appear in other
sections. Each code point is only listed once in the table in Appendix
F.

5.1 Currently-prohibited ASCII characters

Some of the ASCII characters that are currently prohibited in host names
by [STD13] are also used in protocol elements such as URIs [URI]. The other
characters in the range U+0000 to U+007F that are not currently allowed
are also prohibited in host name parts to reserve them for future use in
protocol elements.

0000-002C; [ASCII]
002E-002F; [ASCII]
003A-0040; [ASCII]
005B-0060; [ASCII]
007B-007F; [ASCII]

5.2 Space characters

Space characters would make visual transcription of URLs nearly
impossible and could lead to user entry errors in many ways.

0020; SPACE
00A0; NO-BREAK SPACE
2000; EN QUAD
2001; EM QUAD
2002; EN SPACE
2003; EM SPACE
2004; THREE-PER-EM SPACE
2005; FOUR-PER-EM SPACE
2006; SIX-PER-EM SPACE
2007; FIGURE SPACE
2008; PUNCTUATION SPACE
2009; THIN SPACE
200A; HAIR SPACE
202F; NARROW NO-BREAK SPACE
3000; IDEOGRAPHIC SPACE
1680; OGHAM SPACE MARK
200B; ZERO WIDTH SPACE

5.3 Control characters

Control characters cannot be seen and can cause unpredictable results
when displayed.

0000-001F; [CONTROL CHARACTERS]
007F; DELETE
0080-009F; [CONTROL CHARACTERS]
2028; LINE SEPARATOR
2029; PARAGRAPH SEPARATOR

5.4 Private use and replacement characters

Because private-use characters do not have defined meanings, they are
prohibited. The private-use characters are:

E000-F8FF; [PRIVATE USE, PLANE 0]
F0000-FFFFD; [PRIVATE USE, PLANE 15]
100000-10FFFD; [PRIVATE USE, PLANE 16]

The replacement character (U+FFFD) has no known semantic definition in a
name, and is often displayed by renderers to indicate "there would be some
character here, but it cannot be rendered". For example, on a computer
with no Asian fonts, a name with three katakana characters might be
rendered with three replacement characters.

FFFD; REPLACEMENT CHARACTER

5.5 Non-character code points

Non-character code points are code points that have been assigned in
ISO/IEC 10646 but are not characters. Because they are already assigned,
they are guaranteed not to later change into characters.

FFFE-FFFF; [NONCHARACTER CODE POINTS]
1FFFE-1FFFF; [NONCHARACTER CODE POINTS]
2FFFE-2FFFF; [NONCHARACTER CODE POINTS]
3FFFE-3FFFF; [NONCHARACTER CODE POINTS]
4FFFE-4FFFF; [NONCHARACTER CODE POINTS]
5FFFE-5FFFF; [NONCHARACTER CODE POINTS]
6FFFE-6FFFF; [NONCHARACTER CODE POINTS]
7FFFE-7FFFF; [NONCHARACTER CODE POINTS]
8FFFE-8FFFF; [NONCHARACTER CODE POINTS]
9FFFE-9FFFF; [NONCHARACTER CODE POINTS]
AFFFE-AFFFF; [NONCHARACTER CODE POINTS]
BFFFE-BFFFF; [NONCHARACTER CODE POINTS]
CFFFE-CFFFF; [NONCHARACTER CODE POINTS]
DFFFE-DFFFF; [NONCHARACTER CODE POINTS]
EFFFE-EFFFF; [NONCHARACTER CODE POINTS]
FFFFE-FFFFF; [NONCHARACTER CODE POINTS]
10FFFE-10FFFF; [NONCHARACTER CODE POINTS]

5.6 Surrogate codes

The following code points are permanently reserved for use as surrogate
code values in the UTF-16 encoding, will never be assigned to
characters, and are therefore prohibited:

D800-DFFF; [SURROGATE CODES]

5.7 Inappropriate for plain text

The following characters should not appear in regular text.

FFF9; INTERLINEAR ANNOTATION ANCHOR
FFFA; INTERLINEAR ANNOTATION SEPARATOR
FFFB; INTERLINEAR ANNOTATION TERMINATOR
FFFC; OBJECT REPLACEMENT CHARACTER

5.8 Inappropriate for domain names

The ideographic description characters allow different sequences of
characters to be rendered the same way, which makes them inappropriate
for host names that must have a single canonical order.

2FF0-2FFF; [IDEOGRAPHIC DESCRIPTION CHARACTERS]

5.9 Change display properties

The following characters, some of which are deprecated in ISO/IEC 10646,
can cause changes in display or the order in which characters appear
when rendered.

200E; LEFT-TO-RIGHT MARK
200F; RIGHT-TO-LEFT MARK
202A; LEFT-TO-RIGHT EMBEDDING
202B; RIGHT-TO-LEFT EMBEDDING
202C; POP DIRECTIONAL FORMATTING
202D; LEFT-TO-RIGHT OVERRIDE
202E; RIGHT-TO-LEFT OVERRIDE
206A; INHIBIT SYMMETRIC SWAPPING
206B; ACTIVATE SYMMETRIC SWAPPING
206C; INHIBIT ARABIC FORM SHAPING
206D; ACTIVATE ARABIC FORM SHAPING
206E; NATIONAL DIGIT SHAPES
206F; NOMINAL DIGIT SHAPES

5.10 Inappropriate characters from common input mechanisms

U+3002 is used as if it were U+002E in many input mechanisms,
particularly in Asia. This prohibition allows input mechanisms to safely
map U+3002 to U+002E before doing nameprep without worrying about
preventing users from accessing legitimate host name parts.

3002; IDEOGRAPHIC FULL STOP



6. Unassigned Code Points

All code points not assigned in ISO/IEC 10646 are called "unassigned
code points". Authoritative name servers MUST NOT have internationalized
name parts that contain any unassigned code points. DNS requests MAY
contain name parts that contain unassigned code points. Note that this
is the only part of this document where the requirements for queries
differs from the requirements for names in DNS zones.

Using two different policies for where unassigned code points can appear
in the DNS prevents the need for versioning the IDN protocol [IDNrev].
This is very useful since it makes the overall processing simpler and do
not impose a "protocol" to handle versioning. It is expected that ISO/IEC
10646 will be updated fairly frequently; recently, it has happened
approximately once a year. Each time a new version of ISO/IEC 10646 appears,
a new version of this document can be created. Some end users will want
to use the new code points as soon as they are defined.

The list of unassigned code points can be found in Appendix G of this
document. The list in Appendix G MUST be used by implementations of this
specification. If there are any discrepancies between the list in
Appendix G and the ISO/IEC 10646 specification, the list Appendix G
always takes precedence.

Due to the way that versioning is handled in this section, host names
that are embedded in structures that cannot be changed (such as the
signed parts of digital certificates) MUST NOT have internationalized
name parts that contain any unassigned code points.

6.1 Categories of code points

Each code point in ISO/IEC 10646 can be categorized by how it acts in the
process described in earlier sections of this document:

AO      Code points that may be in the output

MN      Code points that cannot be in the output because they are
         mapped to nothing or never appear as output from
         normalization

D       Code points that cannot be in the output because they are
         disallowed in the prohibition step

U       Unassigned code points

A subsequent version of this document that references a newer version of
ISO/IEC 10646 with new code points will inherently have some code points
move from category U to either D, MN, or AO. For backwards
compatibility, no future version of this document will move code points
from any other category. That is, no current AO, MN, or D code points
will ever change to a different category.

Authoritative name servers MUST NOT contain any name that has code
points outside of AO for the latest version of this document. That is,
they are forbidden to contain any IDN names containing code points from
the MN, D, or U categories.

Applications creating name queries MUST treat U code points as if they
were AO when preparing the name parts according to this document. Those
applications MAY optionally have a preprocess that provide stricter
checks: treating unassigned code points in the input as errors, or
warning the user about the fact that the code point is unassigned in the
version of this document that the software is based on; such a choice is
a local matter for the software.

Non-authoritative DNS servers MAY reject names that contain code points
that are in categories MN or D for the version of this document that
they implement, but MUST NOT reject names because they contain name
parts with code points from category U.

6.2 Reasons for difference between authoritative servers and requests

Different software using different versions of this document need to
interoperate with maximal compatibility. The scheme described in this
section (authoritative name servers MUST NOT use unassigned code points,
requests MAY include unassigned code points) allows that compatibility
without introducing any known security or interoperability issues.

The list below shows what happens if a request contains a code point
from category U that is allowed in a newer version of this document. The
request either resolves to the domain name that was intended, or
resolves to no domain at all. In this list, the request comes from an
application using version "oldVersion" of this document, the
authoritative name server is using version "newVersion" of this
document, and the code point X was in category U on oldVersion, and has
changed category to AO, MN, or D. There are 3 possible scenarios:

1. X becomes AO -- In newVersion, X is in category AO. Because the
application passed X through, it gets back correct data from the
authoritative name server. There is one exceptional case, where X is a
combining mark.

The order of combining marks is normalized, so if another combining mark
Y has a lower combining class than X then XY will be put in the
canonical order YX. (Unassigned code points are never reordered, so this
doesn't happen in oldVersion). If the request contains YX, the request
will get correct data from the authoritative name server. However, no
domain name can be registered with XY, so a request with XY will get a
"no such host" error.

2. X becomes MN -- In newVersion, X is normalized to code point "nX" and
therefore X is now put in category MN. This cannot exist in any domain
name, so any request containing X will get back a "no such host" error.
Note, however, if the request had contained the letter nX, it would have
gotten back correct data.

3. X becomes D -- In newVersion, X is in category MN. This cannot exist
in any domain name, so any request containing X will get back a "no such
host" error.

In none of the cases does the request get data for a host name other
than the one it actually wanted.

The processing in this document is always stable. If a string S is the
result of processing on newVersion, then it will remain the same when
processed on oldVersion.

There is always a way for the application to get the correct data from
the authoritative name server. For example, suppose that <ALPHA> was
unassigned in oldVersion, and that it is assigned in newVersion, but
case-folded to <alpha>. As long as the application supplies strings
containing <alpha> instead of <ALPHA>, the correct data will be
returned. Because the processing is stable, a different application
running newVersion can pass a processed host name to the application
running oldVersion. It will only contain <alpha>, and will return the