tpj13.pod

source of perl for linux application,

# This document contains text in Perl "POD" format.
# Use a POD viewer like perldoc or perlman to render it.

=head1 NAME

Locale::Maketext::TPJ13 -- article about software localization

=head1 SYNOPSIS

  # This an article, not a module.

=head1 DESCRIPTION

The following article by Sean M. Burke and Jordan Lachler
first appeared in I<The Perl Journal> #13
and is copyright 1999 The Perl Journal. It appears
courtesy of Jon Orwant and The Perl Journal.  This document may be
distributed under the same terms as Perl itself.

=head1 Localization and Perl: gettext breaks, Maketext fixes

by Sean M. Burke and Jordan Lachler

This article points out cases where gettext (a common system for
localizing software interfaces -- i.e., making them work in the user's
language of choice) fails because of basic differences between human
languages.  This article then describes Maketext, a new system capable
of correctly treating these differences.

=head2 A Localization Horror Story: It Could Happen To You

=over

"There are a number of languages spoken by human beings in this
world."

-- Harald Tveit Alvestrand, in RFC 1766, "Tags for the
Identification of Languages"

=back

Imagine that your task for the day is to localize a piece of software
-- and luckily for you, the only output the program emits is two
messages, like this:

  I scanned 12 directories.

  Your query matched 10 files in 4 directories.

So how hard could that be?  You look at the code that
produces the first item, and it reads:

  printf("I scanned %g directories.",
         $directory_count);

You think about that, and realize that it doesn't even work right for
English, as it can produce this output:

  I scanned 1 directories.

So you rewrite it to read:

  printf("I scanned %g %s.",
         $directory_count,
         $directory_count == 1 ?
           "directory" : "directories",
  );

...which does the Right Thing.  (In case you don't recall, "%g" is for
locale-specific number interpolation, and "%s" is for string
interpolation.)

But you still have to localize it for all the languages you're
producing this software for, so you pull Locale::gettext off of CPAN
so you can access the C<gettext> C functions you've heard are standard
for localization tasks.

And you write:

  printf(gettext("I scanned %g %s."),
         $dir_scan_count,
         $dir_scan_count == 1 ?
           gettext("directory") : gettext("directories"),
  );

But you then read in the gettext manual (Drepper, Miller, and Pinard 1995)
that this is not a good idea, since how a single word like "directory"
or "directories" is translated may depend on context -- and this is
true, since in a case language like German or Russian, you'd may need
these words with a different case ending in the first instance (where the
word is the object of a verb) than in the second instance, which you haven't even
gotten to yet (where the word is the object of a preposition, "in %g
directories") -- assuming these keep the same syntax when translated
into those languages.

So, on the advice of the gettext manual, you rewrite:

  printf( $dir_scan_count == 1 ?
           gettext("I scanned %g directory.") :
           gettext("I scanned %g directories."),
         $dir_scan_count );

So, you email your various translators (the boss decides that the
languages du jour are Chinese, Arabic, Russian, and Italian, so you
have one translator for each), asking for translations for "I scanned
%g directory." and "I scanned %g directories.".  When they reply,
you'll put that in the lexicons for gettext to use when it localizes
your software, so that when the user is running under the "zh"
(Chinese) locale, gettext("I scanned %g directory.") will return the
appropriate Chinese text, with a "%g" in there where printf can then
interpolate $dir_scan.

Your Chinese translator emails right back -- he says both of these
phrases translate to the same thing in Chinese, because, in linguistic
jargon, Chinese "doesn't have number as a grammatical category" --
whereas English does.  That is, English has grammatical rules that
refer to "number", i.e., whether something is grammatically singular
or plural; and one of these rules is the one that forces nouns to take
a plural suffix (generally "s") when in a plural context, as they are when
they follow a number other than "one" (including, oddly enough, "zero").
Chinese has no such rules, and so has just the one phrase where English
has two.  But, no problem, you can have this one Chinese phrase appear
as the translation for the two English phrases in the "zh" gettext
lexicon for your program.

Emboldened by this, you dive into the second phrase that your software
needs to output: "Your query matched 10 files in 4 directories.".  You notice
that if you want to treat phrases as indivisible, as the gettext
manual wisely advises, you need four cases now, instead of two, to
cover the permutations of singular and plural on the two items,
$dir_count and $file_count.  So you try this:

  printf( $file_count == 1 ?
    ( $directory_count == 1 ?
     gettext("Your query matched %g file in %g directory.") :
     gettext("Your query matched %g file in %g directories.") ) :
    ( $directory_count == 1 ?
     gettext("Your query matched %g files in %g directory.") :
     gettext("Your query matched %g files in %g directories.") ),
   $file_count, $directory_count,
  );

(The case of "1 file in 2 [or more] directories" could, I suppose,
occur in the case of symlinking or something of the sort.)

It occurs to you that this is not the prettiest code you've ever
written, but this seems the way to go.  You mail off to the
translators asking for translations for these four cases.  The
Chinese guy replies with the one phrase that these all translate to in
Chinese, and that phrase has two "%g"s in it, as it should -- but
there's a problem.  He translates it word-for-word back: "In %g
directories contains %g files match your query."  The %g
slots are in an order reverse to what they are in English.  You wonder
how you'll get gettext to handle that.

But you put it aside for the moment, and optimistically hope that the
other translators won't have this problem, and that their languages
will be better behaved -- i.e., that they will be just like English.

But the Arabic translator is the next to write back.  First off, your
code for "I scanned %g directory." or "I scanned %g directories."
assumes there's only singular or plural.  But, to use linguistic
jargon again, Arabic has grammatical number, like English (but unlike
Chinese), but it's a three-term category: singular, dual, and plural.
In other words, the way you say "directory" depends on whether there's
one directory, or I<two> of them, or I<more than two> of them.  Your
test of C<($directory == 1)> no longer does the job.  And it means
that where English's grammatical category of number necessitates
only the two permutations of the first sentence based on "directory
[singular]" and "directories [plural]", Arabic has three -- and,
worse, in the second sentence ("Your query matched %g file in %g
directory."), where English has four, Arabic has nine.  You sense
an unwelcome, exponential trend taking shape.

Your Italian translator emails you back and says that "I searched 0
directories" (a possible English output of your program) is stilted,
and if you think that's fine English, that's your problem, but that
I<just will not do> in the language of Dante.  He insists that where
$directory_count is 0, your program should produce the Italian text
for "I I<didn't> scan I<any> directories.".  And ditto for "I didn't
match any files in any directories", although he says the last part
about "in any directories" should probably just be left off.

You wonder how you'll get gettext to handle this; to accomodate the
ways Arabic, Chinese, and Italian deal with numbers in just these few
very simple phrases, you need to write code that will ask gettext for
different queries depending on whether the numerical values in
question are 1, 2, more than 2, or in some cases 0, and you still haven't
figured out the problem with the different word order in Chinese.

Then your Russian translator calls on the phone, to I<personally> tell
you the bad news about how really unpleasant your life is about to
become:

Russian, like German or Latin, is an inflectional language; that is, nouns
and adjectives have to take endings that depend on their case
(i.e., nominative, accusative, genitive, etc...) -- which is roughly a matter of
what role they have in syntax of the sentence --
as well as on the grammatical gender (i.e., masculine, feminine, neuter)
and number (i.e., singular or plural) of the noun, as well as on the
declension class of the noun.  But unlike with most other inflected languages,
putting a number-phrase (like "ten" or "forty-three", or their Arabic
numeral equivalents) in front of noun in Russian can change the case and
number that noun is, and therefore the endings you have to put on it.

He elaborates:  In "I scanned %g directories", you'd I<expect>
"directories" to be in the accusative case (since it is the direct
object in the sentence) and the plural number,
except where $directory_count is 1, then you'd expect the singular, of
course.  Just like Latin or German.  I<But!>  Where $directory_count %
10 is 1 ("%" for modulo, remember), assuming $directory count is an
integer, and except where $directory_count % 100 is 11, "directories"
is forced to become grammatically singular, which means it gets the
ending for the accusative singular...  You begin to visualize the code
it'd take to test for the problem so far, I<and still work for Chinese
and Arabic and Italian>, and how many gettext items that'd take, but
he keeps going...  But where $directory_count % 10 is 2, 3, or 4
(except where $directory_count % 100 is 12, 13, or 14), the word for
"directories" is forced to be genitive singular -- which means another
ending... The room begins to spin around you, slowly at first...  But
with I<all other> integer values, since "directory" is an inanimate
noun, when preceded by a number and in the nominative or accusative
cases (as it is here, just your luck!), it does stay plural, but it is
forced into the genitive case -- yet another ending...  And
you never hear him get to the part about how you're going to run into
similar (but maybe subtly different) problems with other Slavic
languages like Polish, because the floor comes up to meet you, and you
fade into unconsciousness.


The above cautionary tale relates how an attempt at localization can
lead from programmer consternation, to program obfuscation, to a need
for sedation.  But careful evaluation shows that your choice of tools
merely needed further consideration.

=head2 The Linguistic View

=over

"It is more complicated than you think." 

-- The Eighth Networking Truth, from RFC 1925

=back

The field of Linguistics has expended a great deal of effort over the
past century trying to find grammatical patterns which hold across
languages; it's been a constant process
of people making generalizations that should apply to all languages,
only to find out that, all too often, these generalizations fail --
sometimes failing for just a few languages, sometimes whole classes of
languages, and sometimes nearly every language in the world except
English.  Broad statistical trends are evident in what the "average
language" is like as far as what its rules can look like, must look
like, and cannot look like.  But the "average language" is just as
unreal a concept as the "average person" -- it runs up against the
fact no language (or person) is, in fact, average.  The wisdom of past
experience leads us to believe that any given language can do whatever
it wants, in any order, with appeal to any kind of grammatical