ch06_19.htm

By Tom Christiansen and Nathan Torkington ISBN 1-56592-243-3 First Edition, published August 1998

>/ox</CODE
> modifiers. The <CODE
CLASS="literal"
>/x</CODE
> modifier is especially crucial due to the whitespace used in the encoding template <CODE
CLASS="literal"
>$eucjp</CODE
>. The <CODE
CLASS="literal"
>/o</CODE
> modifier is for efficiency, since we know <CODE
CLASS="literal"
>$eucjp</CODE
> won't change from use to use.</P
><P
CLASS="para"
>Use in a replacement is similar, but since the text leading to the real match is also part of the overall match, we must capture it with parentheses, being sure to include it in the replacment text. Assuming that <CODE
CLASS="literal"
>$Tokyo</CODE
> and <CODE
CLASS="literal"
>$Osaka</CODE
> have been set to the bytes sequences for their respective words in the EUC-JP encoding, we could use the following to replace Osaka for Tokyo:</P
><PRE
CLASS="programlisting"
>/^ (  (?:eucjp)*? ) $Tokyo/$1$Osaka/ox</PRE
><P
CLASS="para"
>If used with <CODE
CLASS="literal"
>/g</CODE
>, we want to anchor the match to the end of the previous match, rather than to the start of the string. That's as simple as changing <CODE
CLASS="literal"
>^</CODE
> to <CODE
CLASS="literal"
>\G</CODE
>:</P
><PRE
CLASS="programlisting"
>/\G (  (?:eucjp)*? ) $Tokyo/$1$Osaka/gox</PRE
></DIV
><DIV
CLASS="sect3"
><H4
CLASS="sect3"
><A
CLASS="title"
NAME="ch06-pgfId-1000010009"
>Splitting multiple-byte strings</A
></H4
><P
CLASS="para"
>Another common task is to split an input string into its individual charcters. With a one-byte-per-character encoding, you can simply split <CODE
CLASS="literal"
>//</CODE
>, but with a multiple-byte encoding, we need something like:</P
><PRE
CLASS="programlisting"
>@chars = /$eucjp/gox; # One character per list element</PRE
><P
CLASS="para"
>Now, <CODE
CLASS="literal"
>@chars</CODE
> contains one character per element. The following snippet shows how you might use this to write a filter of some sort:</P
><PRE
CLASS="programlisting"
>while (&lt;&gt;) {
  my @chars = /$eucjp/gox; # One character per list element
  for my $char (@chars) {
    if (length($char) == 1) {
      # Do something interesting with this one-byte character
    } else {
      # Do something interesting with this multiple-byte character
    }
  }
  my $line = join(&quot;&quot;,@chars); # Glue list back together
  print $line;
}</PRE
><P
CLASS="para"
>In the two "do something interesting" parts, any change to <CODE
CLASS="literal"
>$char</CODE
> will be reflected in the output when <CODE
CLASS="literal"
>@chars</CODE
> is glued back together.</P
></DIV
><DIV
CLASS="sect3"
><H4
CLASS="sect3"
><A
CLASS="title"
NAME="ch06-pgfId-1000010032"
>Validating multiple-byte strings</A
></H4
><P
CLASS="para"
>The use of <CODE
CLASS="literal"
>/$eucjp/gox</CODE
> in this kind of technique relies strongly on the input string indeed being properly formatted in our target encoding, EUC-JP. If it's not, the template <CODE
CLASS="literal"
>/$eucjp/</CODE
> won't be able to match, and bytes will be skipped.</P
><P
CLASS="para"
>One way to address this is to use <CODE
CLASS="literal"
>/\G$eucjp/gox</CODE
> instead. This prohibits the pattern matching engine from skipping bytes in order to find a match (since the use of <CODE
CLASS="literal"
>\G</CODE
> indicates that any match must immediately follow the previous match). This is still not a perfect approach, since it will simply stop matching on ill-formatted input data.</P
><P
CLASS="para"
>A better approach to confirm that a string is valid with respect to an encoding is to use something like:</P
><PRE
CLASS="programlisting"
>$is_eucjp = m/^(?:$eucjp)*$/xo;</PRE
><P
CLASS="para"
>If a string has only valid characters from start to end, you know the string as a whole is valid.</P
><P
CLASS="para"
>There is one potential for a problem, and that's due to how the end-of-string metacharacter <CODE
CLASS="literal"
>$</CODE
> works: it can be true at the end of the string (as we want), and also just before a newline at the end of the string. That means you can still match sucessfully even if the newline is not a valid character in the encoding. To get around this problem, you could use the more-complicated <CODE
CLASS="literal"
>(?!\n)$</CODE
> instead of <CODE
CLASS="literal"
>$</CODE
>.</P
><P
CLASS="para"
>You can use the basic validation technique to detect which encoding is being used. For example, Japanese is commonly encoded with either EUC-JP, or another encoding called Shift-JIS. If you've set up the templates, as with <CODE
CLASS="literal"
>$eucjp</CODE
>, you can do something like:</P
><PRE
CLASS="programlisting"
>$is_eucjp = m/^(?:$eucjp)*$/xo;
$is_sjis  = m/^(?:$sjis)*$/xo;</PRE
><P
CLASS="para"
>If both are true, the text is likely ASCII (since, essentially, ASCII is a sub-component of both encodings). (It's not quite fool-proof, though, since some strings with multi-byte characters might appear to be valid in both encodings. In such a case, automatic detection becomes impossible, although one might use character-frequency data to make an educated guess.)</P
></DIV
><DIV
CLASS="sect3"
><H4
CLASS="sect3"
><A
CLASS="title"
NAME="ch06-pgfId-1000010053"
>Converting between encodings</A
></H4
><P
CLASS="para"
>Converting from one encoding to another can be as simple as an extension of the process-each-character routine above. Conversions for some closely related encodings can be done by a simple mathematical computation on the bytes, while others might require huge mapping tables. In either case, you insert the code at the "do something interesting" points in the routine.</P
><P
CLASS="para"
>Here's an example to convert from EUC-JP to Unicode, using a <CODE
CLASS="literal"
>%euc2uni</CODE
> hash as a mapping table:</P
><PRE
CLASS="programlisting"
>while (&lt;&gt;) {
  my @chars = /$eucjp/gox; # One character per list element
  for my $euc (@chars) {
    my $uni = $euc2uni{$char};
    if (defined $uni) {
        $euc = $uni;
    } else {
        ## deal with unknown EUC-&gt;Unicode mapping here.
    }
  }
  my $line = join(&quot;&quot;,@chars);
  print $line;
}</PRE
><P
CLASS="para"
>The topic of multiple-byte matching and processing is of particular importance when dealing with Unicode, which has a variety of possible representations. UCS-2 and UCS-4 are fixed-length encodings. UTF-8 defines a mixed one- through six-byte encoding. UTF-16, which represents the most common instance of Unicode encoding, is a variable-length 16-bit encoding.<A
CLASS="indexterm"
NAME="ch06-idx-1000010159-0"
></A
><A
CLASS="indexterm"
NAME="ch06-idx-1000010159-1"
></A
><A
CLASS="indexterm"
NAME="ch06-idx-1000010159-2"
></A
><A
CLASS="indexterm"
NAME="ch06-idx-1000010159-3"
></A
><A
CLASS="indexterm"
NAME="ch06-idx-1000010159-4"
></A
></P
></DIV
></DIV
><DIV
CLASS="sect2"
><H3
CLASS="sect2"
><A
CLASS="title"
NAME="ch06-pgfId-1000010163"
>See Also</A
></H3
><P
CLASS="para"
>Jeffrey Friedl's article in Issue 5 of <CITE
CLASS="citetitle"
>The Perl Journal </CITE
>; <CITE
CLASS="citetitle"
>CJKV Information Processing</CITE
> by Ken Lunde; O'Reilly &amp; Associates, (due 1999)</P
></DIV
></DIV
><DIV
CLASS="htmlnav"
><P
></P
><HR
ALIGN="LEFT"
WIDTH="684"
TITLE="footer"><TABLE
WIDTH="684"
BORDER="0"
CELLSPACING="0"
CELLPADDING="0"
><TR
><TD
ALIGN="LEFT"
VALIGN="TOP"
WIDTH="228"
><A
CLASS="sect1"
HREF="ch06_18.htm"
TITLE="6.17. Expressing AND, OR, and NOT in a Single Pattern"
><IMG
SRC="../gifs/txtpreva.gif"
ALT="Previous: 6.17. Expressing AND, OR, and NOT in a Single Pattern"
BORDER="0"></A
></TD
><TD
ALIGN="CENTER"
VALIGN="TOP"
WIDTH="228"
><A
CLASS="book"
HREF="index.htm"
TITLE="Perl Cookbook"
><IMG
SRC="../gifs/txthome.gif"
ALT="Perl Cookbook"
BORDER="0"></A
></TD
><TD
ALIGN="RIGHT"
VALIGN="TOP"
WIDTH="228"
><A
CLASS="sect1"
HREF="ch06_20.htm"
TITLE="6.19. Matching a Valid Mail Address"
><IMG
SRC="../gifs/txtnexta.gif"
ALT="Next: 6.19. Matching a Valid Mail Address"
BORDER="0"></A
></TD
></TR
><TR
><TD
ALIGN="LEFT"
VALIGN="TOP"
WIDTH="228"
>6.17. Expressing AND, OR, and NOT in a Single Pattern</TD
><TD
ALIGN="CENTER"
VALIGN="TOP"
WIDTH="228"
><A
CLASS="index"
HREF="index/index.htm"
TITLE="Book Index"
><IMG
SRC="../gifs/index.gif"
ALT="Book Index"
BORDER="0"></A
></TD
><TD
ALIGN="RIGHT"
VALIGN="TOP"
WIDTH="228"
>6.19. Matching a Valid Mail Address</TD
></TR
></TABLE
><HR
ALIGN="LEFT"
WIDTH="684"
TITLE="footer"><FONT
SIZE="-1"
></DIV<!-- LIBRARY NAV BAR --> <img src="../gifs/smnavbar.gif" usemap="#library-map" border="0" alt="Library Navigation Links"><p> <a href="copyrght.htm">Copyright &copy; 2002</a> O'Reilly &amp; Associates. All rights reserved.</font> </p> <map name="library-map"> <area shape="rect" coords="1,0,85,94" href="../index.htm"><area shape="rect" coords="86,1,178,103" href="../lwp/index.htm"><area shape="rect" coords="180,0,265,103" href="../lperl/index.htm"><area shape="rect" coords="267,0,353,105" href="../perlnut/index.htm"><area shape="rect" coords="354,1,446,115" href="../prog/index.htm"><area shape="rect" coords="448,0,526,132" href="../tk/index.htm"><area shape="rect" coords="528,1,615,119" href="../cookbook/index.htm"><area shape="rect" coords="617,0,690,135" href="../pxml/index.htm"></map> </BODY
></HTML
>