=head1 NAME

ODF::lpOD_Helper::Unicode - Unicode and ODF::lpOD (and XML::Twig)

=head1 SYNOPSIS

  use ODF::lpOD;
  use ODF::lpOD_Helper;
  use feature 'unicode_strings';

=head1 INTRODUCTION

We once thought Unicode forced us
to fiddle with bytes to handle "international" characters.
That thinking came from low-level languages like C.

Perl saved us but it took years before everyone believed.
And more years before Perl's Unicode paradigm clearly emerged.

Meanwhile lots of pod and code was written which in hindsight
was confused or misleading.

=head1 THE PERL UNICODE PARADIGM

  1. "Decode" input from binary into Perl characters as soon as possible
     after getting it from the outside world (e.g. from a disk file).

  2. As much of the application as possible works with Perl characters,
     paying absolutely no attention to encoding.

  3. "Encode" Perl characters into binary data as late as possible,
     just before sending the data out.

See "Not always so tidy" below for more discussion.

=head1 ODF::lpOD

For historical reasons
ODF::lpOD by default is incompatible with the above paradigm
because every method encodes result strings into UTF-8
before returning them to you, and attempts to decode strings you pass in before
using them.  Therefore you must work with binary octets rather than
abstract characters;
Regex match, substr(), length(), and comparison with string literals
do not work with non-ASCII/latin1 characters.
Also, you can't print such already-encoded binary to STDOUT if that
file handle auto-encodes because the data will be encoded twice.

B<< use ODF::lpOD_Helper >>
disables ODF::lpOD's internal encoding and decoding.
Methods then speak and listen in characters, not octets.

You should also B<< use feature 'unicode_strings'; >> to
avoid problematic aspects of legacy Perl behavior.

=head1 IF LEGACY BEHAVIOR IS NEEDED

The import tag B<:bytes> will tell ODF::lpOD_Helper to not disable
internal encoding & decoding, i.e. retain the original ODF::lpOD behavior.

It is also possible to toggle between the old and new behavior at
run-time:
I<< lpod->Huse_octet_strings() >> will will re-enable implicit
decoding/encoding of method arguments (using UTF-8 encoding by default)
and I<< lpod->Huse_character_strings() >> will disable the old behavior
and restore transparent Unicode support.

Prior to version 6.000 character mode was not the default, but
required a B<:chars> import tag.  That tag is now deprecated and
produces a warning.

=head1 XML::Twig & XML::Parser

XML::Twig's docs wrongly say that it stores strings in memory encoded
as UTF-8, implying that a I<decode> operation would be needed to obtain
characters; in fact XML::Twig stores strings as characters
(the result of the I<decode> done by C<parse()>),
and most methods return those Perl characters, not encoded binary.

An exception is XML::Twig's B<print()> and B<sprint> which encode
their results.  The encoding used is the same as was found when
parsing the input file unless you change it,
and the generated XML header specifies the encoding used.
This makes sense when printing directly to a disk file (via a non-encoding
file handle), but will corrupt non-ASCII characters printed to STDOUT
if STDOUT is set to auto-encode, as is usually the case.

Going the other way, XML::Twig's B<parse()> method expects binary octets as
input and uses the "encoding=..." specification in the XML
header to know how to decode the remainder of the file.

=head1 Not always so tidy

Perl's data model now has two kinds of strings: B<binary octets>,
which might or might not be some representation of Unicode characters,
and B<abstract characters> which are opaque atoms.
In most cases these two kinds should never be combined, see L<perlunifaq>.

I<Decoding> means converting possibly-multiple binary octets into
abstract characters, and I<encoding> does the reverse.

Obviously Perl somehow represents abstract characters internally,
and it often (not always) uses a variation of UTF-8;
as a result, "decoding" or "encoding" between abstract characters
and UTF-8 octet streams can be very fast.

Non-Unicode-aware scripts usually "just work" as long as they never
see characters outside the ASCII range.  This is because,
without a I<decode> operation or literal strings with "wide" characters,
Perl treats all strings as binary octets; the behaviour of such strings is
usually correct if all characters have single-byte representations.

=head2 The "wide character" warning

If a non-ASCII/latin1 I<character> gets into a Perl string
which is written to a non-encoding file handle, Perl issues
a warning because that should never happen.  It means
the program is both Unicode-unaware and uses characters which
require Unicode awareness to work correctly.
See L<perlunifaq>.

Unicode-aware programs always I<encode> character strings before
writing to disk either explicitly or by using an encoding file handle.

Non-unicode-aware programs (which neither decode or encode),
will "just work" with ASCII data as described earlier.

=head1 SEE ALSO

L<Encode>, L<Encode::Locale>, L<PerlIO::locale>, L<perlunifaq>

(END)

=cut
