tpj13.pod

source of perl for linux application,

categories wants -- case, number, tense, real or metaphoric
characteristics of the things that words refer to, arbitrary or
predictable classifications of words based on what endings or prefixes
they can take, degree or means of certainty about the truth of
statements expressed, and so on, ad infinitum.

Mercifully, most localization tasks are a matter of finding ways to
translate whole phrases, generally sentences, where the context is
relatively set, and where the only variation in content is I<usually>
in a number being expressed -- as in the example sentences above.
Translating specific, fully-formed sentences is, in practice, fairly
foolproof -- which is good, because that's what's in the phrasebooks
that so many tourists rely on.  Now, a given phrase (whether in a
phrasebook or in a gettext lexicon) in one language I<might> have a
greater or lesser applicability than that phrase's translation into
another language -- for example, strictly speaking, in Arabic, the
"your" in "Your query matched..." would take a different form
depending on whether the user is male or female; so the Arabic
translation "your[feminine] query" is applicable in fewer cases than
the corresponding English phrase, which doesn't distinguish the user's
gender.  (In practice, it's not feasable to have a program know the
user's gender, so the masculine "you" in Arabic is usually used, by
default.)

But in general, such surprises are rare when entire sentences are
being translated, especially when the functional context is restricted
to that of a computer interacting with a user either to convey a fact
or to prompt for a piece of information.  So, for purposes of
localization, translation by phrase (generally by sentence) is both the
simplest and the least problematic.

=head2 Breaking gettext

=over

"It Has To Work."

-- First Networking Truth, RFC 1925

=back

Consider that sentences in a tourist phrasebook are of two types: ones
like "How do I get to the marketplace?" that don't have any blanks to
fill in, and ones like "How much do these ___ cost?", where there's
one or more blanks to fill in (and these are usually linked to a
list of words that you can put in that blank: "fish", "potatoes",
"tomatoes", etc.)  The ones with no blanks are no problem, but the
fill-in-the-blank ones may not be really straightforward. If it's a
Swahili phrasebook, for example, the authors probably didn't bother to
tell you the complicated ways that the verb "cost" changes its
inflectional prefix depending on the noun you're putting in the blank.
The trader in the marketplace will still understand what you're saying if
you say "how much do these potatoes cost?" with the wrong
inflectional prefix on "cost".  After all, I<you> can't speak proper Swahili,
I<you're> just a tourist.  But while tourists can be stupid, computers
are supposed to be smart; the computer should be able to fill in the
blank, and still have the results be grammatical.

In other words, a phrasebook entry takes some values as parameters
(the things that you fill in the blank or blanks), and provides a value
based on these parameters, where the way you get that final value from
the given values can, properly speaking, involve an arbitrarily
complex series of operations.  (In the case of Chinese, it'd be not at
all complex, at least in cases like the examples at the beginning of
this article; whereas in the case of Russian it'd be a rather complex
series of operations.  And in some languages, the
complexity could be spread around differently: while the act of
putting a number-expression in front of a noun phrase might not be
complex by itself, it may change how you have to, for example, inflect
a verb elsewhere in the sentence.  This is what in syntax is called
"long-distance dependencies".)

This talk of parameters and arbitrary complexity is just another way
to say that an entry in a phrasebook is what in a programming language
would be called a "function".  Just so you don't miss it, this is the
crux of this article: I<A phrase is a function; a phrasebook is a
bunch of functions.>

The reason that using gettext runs into walls (as in the above
second-person horror story) is that you're trying to use a string (or
worse, a choice among a bunch of strings) to do what you really need a
function for -- which is futile.  Preforming (s)printf interpolation
on the strings which you get back from gettext does allow you to do I<some>
common things passably well... sometimes... sort of; but, to paraphrase
what some people say about C<csh> script programming, "it fools you
into thinking you can use it for real things, but you can't, and you
don't discover this until you've already spent too much time trying,
and by then it's too late."

=head2 Replacing gettext

So, what needs to replace gettext is a system that supports lexicons
of functions instead of lexicons of strings.  An entry in a lexicon
from such a system should I<not> look like this:

  "J'ai trouv\xE9 %g fichiers dans %g r\xE9pertoires"

[\xE9 is e-acute in Latin-1.  Some pod renderers would
scream if I used the actual character here. -- SB]

but instead like this, bearing in mind that this is just a first stab:

  sub I_found_X1_files_in_X2_directories {
    my( $files, $dirs ) = @_[0,1];
    $files = sprintf("%g %s", $files,
      $files == 1 ? 'fichier' : 'fichiers');
    $dirs = sprintf("%g %s", $dirs,
      $dirs == 1 ? "r\xE9pertoire" : "r\xE9pertoires");
    return "J'ai trouv\xE9 $files dans $dirs.";
  }

Now, there's no particularly obvious way to store anything but strings
in a gettext lexicon; so it looks like we just have to start over and
make something better, from scratch.  I call my shot at a
gettext-replacement system "Maketext", or, in CPAN terms,
Locale::Maketext.

When designing Maketext, I chose to plan its main features in terms of
"buzzword compliance".  And here are the buzzwords:

=head2 Buzzwords: Abstraction and Encapsulation

The complexity of the language you're trying to output a phrase in is
entirely abstracted inside (and encapsulated within) the Maketext module
for that interface.  When you call:

  print $lang->maketext("You have [quant,_1,piece] of new mail.",
                       scalar(@messages));

you don't know (and in fact can't easily find out) whether this will
involve lots of figuring, as in Russian (if $lang is a handle to the
Russian module), or relatively little, as in Chinese.  That kind of
abstraction and encapsulation may encourage other pleasant buzzwords
like modularization and stratification, depending on what design
decisions you make.

=head2 Buzzword: Isomorphism

"Isomorphism" means "having the same structure or form"; in discussions
of program design, the word takes on the special, specific meaning that
your implementation of a solution to a problem I<has the same
structure> as, say, an informal verbal description of the solution, or
maybe of the problem itself.  Isomorphism is, all things considered,
a good thing -- it's what problem-solving (and solution-implementing)
should look like.

What's wrong the with gettext-using code like this...

  printf( $file_count == 1 ?
    ( $directory_count == 1 ?
     "Your query matched %g file in %g directory." :
     "Your query matched %g file in %g directories." ) :
    ( $directory_count == 1 ?
     "Your query matched %g files in %g directory." :
     "Your query matched %g files in %g directories." ),
   $file_count, $directory_count,
  );

is first off that it's not well abstracted -- these ways of testing
for grammatical number (as in the expressions like C<foo == 1 ?
singular_form : plural_form>) should be abstracted to each language
module, since how you get grammatical number is language-specific.

But second off, it's not isomorphic -- the "solution" (i.e., the
phrasebook entries) for Chinese maps from these four English phrases to
the one Chinese phrase that fits for all of them.  In other words, the
informal solution would be "The way to say what you want in Chinese is
with the one phrase 'For your question, in Y directories you would
find X files'" -- and so the implemented solution should be,
isomorphically, just a straightforward way to spit out that one
phrase, with numerals properly interpolated.  It shouldn't have to map
from the complexity of other languages to the simplicity of this one.

=head2 Buzzword: Inheritance

There's a great deal of reuse possible for sharing of phrases between
modules for related dialects, or for sharing of auxiliary functions
between related languages.  (By "auxiliary functions", I mean
functions that don't produce phrase-text, but which, say, return an
answer to "does this number require a plural noun after it?".  Such
auxiliary functions would be used in the internal logic of functions
that actually do produce phrase-text.)

In the case of sharing phrases, consider that you have an interface
already localized for American English (probably by having been
written with that as the native locale, but that's incidental).
Localizing it for UK English should, in practical terms, be just a
matter of running it past a British person with the instructions to
indicate what few phrases would benefit from a change in spelling or
possibly minor rewording.  In that case, you should be able to put in
the UK English localization module I<only> those phrases that are
UK-specific, and for all the rest, I<inherit> from the American
English module.  (And I expect this same situation would apply with
Brazilian and Continental Portugese, possbily with some I<very>
closely related languages like Czech and Slovak, and possibly with the
slightly different "versions" of written Mandarin Chinese, as I hear exist in
Taiwan and mainland China.)

As to sharing of auxiliary functions, consider the problem of Russian
numbers from the beginning of this article; obviously, you'd want to
write only once the hairy code that, given a numeric value, would
return some specification of which case and number a given quanitified
noun should use.  But suppose that you discover, while localizing an
interface for, say, Ukranian (a Slavic language related to Russian,
spoken by several million people, many of whom would be relieved to
find that your Web site's or software's interface is available in
their language), that the rules in Ukranian are the same as in Russian
for quantification, and probably for many other grammatical functions.
While there may well be no phrases in common between Russian and
Ukranian, you could still choose to have the Ukranian module inherit
from the Russian module, just for the sake of inheriting all the
various grammatical methods.  Or, probably better organizationally,
you could move those functions to a module called C<_E_Slavic> or
something, which Russian and Ukranian could inherit useful functions
from, but which would (presumably) provide no lexicon.

=head2 Buzzword: Concision

Okay, concision isn't a buzzword.  But it should be, so I decree that
as a new buzzword, "concision" means that simple common things should
be expressible in very few lines (or maybe even just a few characters)
of code -- call it a special case of "making simple things easy and
hard things possible", and see also the role it played in the
MIDI::Simple language, discussed elsewhere in this issue [TPJ#13].

Consider our first stab at an entry in our "phrasebook of functions":

  sub I_found_X1_files_in_X2_directories {
    my( $files, $dirs ) = @_[0,1];
    $files = sprintf("%g %s", $files,
      $files == 1 ? 'fichier' : 'fichiers');
    $dirs = sprintf("%g %s", $dirs,
      $dirs == 1 ? "r\xE9pertoire" : "r\xE9pertoires");
    return "J'ai trouv\xE9 $files dans $dirs.";
  }

You may sense that a lexicon (to use a non-committal catch-all term for a
collection of things you know how to say, regardless of whether they're
phrases or words) consisting of functions I<expressed> as above would
make for rather long-winded and repetitive code -- even if you wisely
rewrote this to have quantification (as we call adding a number
expression to a noun phrase) be a function called like:

  sub I_found_X1_files_in_X2_directories {
    my( $files, $dirs ) = @_[0,1];
    $files = quant($files, "fichier");
    $dirs =  quant($dirs,  "r\xE9pertoire");
    return "J'ai trouv\xE9 $files dans $dirs.";
  }

And you may also sense that you do not want to bother your translators
with having to write Perl code -- you'd much rather that they spend
their I<very costly time> on just translation.  And this is to say
nothing of the near impossibility of finding a commercial translator
who would know even simple Perl.

In a first-hack implementation of Maketext, each language-module's
lexicon looked like this: