package Treex::Tutorial::FirstSteps;
$Treex::Tutorial::FirstSteps::VERSION = '2.20150928';
# TODO: opravit linky, aby ukazovali do CPANu (az budou bloky released)

=encoding utf8

=head1 NAME

Treex::Tutorial::FirstSteps - First steps after installing Treex

=head1 VERSION

version 2.20150928

=head1 SHORT INTRODUCTION

The elementary unit of code in Treex is called B<block>.
Each block should solve some well defined and usually lingustically motivated
task, e.g. tokenization, tagging or parsing.

A sequence of blocks is called C<scenario> and it can describe end-to-end NLP
application, e.g. machine translation or preprocessing of a parallel treebank. 

Treex applications can be executed from Perl.
However, usually they are executed using the command line interface C<treex>.

=head2 Other Sources

For more specific tutorials, feel free to browse L<Treex::Tutorial::|https://github.com/ufal/treex/tree/master/lib/Treex/Tutorial> directory.

You might also want to read this L<presentation|https://ufal.mff.cuni.cz/~popel/treex.pdf> about Treex framework.

=head1 HELLO WORLD

We will start traditionally with the "Hello, world!" example :-). 

 echo 'Hello, world!' | treex Read::Text language=en Write::Text language=en

The desired output was printed to STDOUT, but there are some info messages
around printed to STDERR. To filter out these messages you can either use the
C<--quiet> option (C<-q>) or the standard redirection of STDERR.

 echo 'Hello, world!' | treex -q Read::Text language=en Write::Text language=en
 echo 'Hello, world!' | treex Read::Text language=en Write::Text language=en 2>/dev/null

=over

=item What does the syntax mean?

I<Read::Text language=en Write::Text language=en> is a scenario definition.
The scenario consists of two blocks: I<Read::Text> and I<Write::Text>.
Each block has one parameter set, the name of the parameter is C<language>
and its value is C<en> (which is an ISO 639-1 code for English).

=item Why is the C<language> parameter needed?

One Treex document can contain sentences in more languages
(which is useful for tasks like word alignment or machine translation),
so it is necessary to instruct each block on which language it should operate.

=item Can I make the scenario description shorter? 

It is not necessary to repeat the same parameter specification for every block.
You can use a special block C<Util::SetGlobal>:

 echo 'Hello, world!' | treex -q Util::SetGlobal language=en Read::Text Write::Text

=item Can I make it even shorter?

Yes. (And I know the previous example was not actually shorter.)
There is an option C<--language> (C<-L>) which is just a shortcut
for C<Util::SetGlobal language=..>.

 echo 'Hello, world!' | treex -q --language=en Read::Text Write::Text
 echo 'Hello, world!' | treex -q -Len Read::Text Write::Text
  
=back

The "Hello, world!" example is silly. The first block (so-called B<reader>)
reads the plain text input, converts it to the Treex in-memory document
representation and this document is passed to the second block
(so-called B<writer>) that converts it to plain text and prints to STDOUT.
No (linguistic) processing was done.

There are readers and writers for various other formats than plain text
(e.g. HTML, CoNLL, PennTB MRG, PDT PML), so you can use it for format
conversions (see L<Treex::Tutorial::ReadersAndWriters>).
You can also create you own readers and writers for new formats
(see L<Treex::Tutorial::WritingNewReaders).

For simplicity, we'll continue to use plain text format
in this tutorial chapter, but we'll try to do something slightly more interesting. 

=head1 SEGMENTATION TO SENTENCES

To segment a text into sentences we can use block C<W2A::Segment>
and writer C<Write::Sentences> that prints each sentence on a separate line.

 echo "Hello! Mr. Brown, how are you?" \
  | treex -Len Read::Text W2A::Segment Write::Sentences

You can see, that the text was segmented into three sentences:
"I<Hello!>", "I<Mr.>", and "I<Brown, how are you?>".
Block C<W2A::Segment> is language independent (at least for languages using
Latin alphabet) and it finds sentence boundaries just based on a regex rules
that detect end-sentence punctuation ([.?!]) followed by a capital letter.
To get the correct segmentation we must use C<W2A::EN::Segment> which has a list
of English words (tokens) that usually do not end a sentence
even if they are followed by a fullstop and a capital letter.
By the way, Treex is object-oriented, blocks are classes and
C<W2A::EN::Segment> is a descendant of the C<W2A::Segment> base class.

 echo "Hello! Mr. Brown, how are you?" \
  | treex -Len Read::Text W2A::EN::Segment Write::Sentences

=over

=item Where can I find blocks' source code?

The blocks are actually Perl modules and if you followed L<Treex::Tutorial::Install>
you can find them in F<~/perl5/lib/perl5/Treex/Block/>.
Generally, you can find the real location of perl module with C<perldoc -l>.
The full name of the C<W2A::EN::Segment> module is actually
C<Treex::Block::W2A::EN::Segment>, but since the prefix C<Treex::Block::>
is common to all blocks, it is not written in the scenario description.
So the location of C<W2A::EN::Segment> can be found with

 perldoc -l Treex::Block::W2A::EN::Segment

=item What does the name C<W2A::EN::Segment> mean?

All Treex blocks that do shallow linguistic analysis
(segmentation, tokenization, lemmatization, PoS tagging, dependency parsing)
are grouped in a directory C<W2A> (W and A are names of the two layers of
language description). 
Language specific blocks are stored in a subdirectory with a uppercase ISO code
of the given language (C<EN>) in this case.

=item How to read already segmented input?

If you have C<sample.txt> with one sentence per line, you can load it to Treex using

 cat sample.txt | treex -Len Read::Sentences ...

=back

There are many other options for segmentation, see (perldoc for) modules
L<Treex::Block::W2A::Segment>,
L<Treex::Block::W2A::SegmentOnNewlines>, and
L<Treex::Block::W2A::ResegmentSentences>.

=head2 Task 1

You have an input plain text (e.g. F<data/news.txt>)
where each paragraph (including headlines)
is on a separate line. Load this file into Treex and print one sentence per line.
Note that headlines do not end with a fullstop,
but they should be treated as separate sentences

HINT: See documentation of L<Treex::Block::W2A::Segment>.

=head1 TOKENIZATION, LEMMATIZATION, TAGGING

Try these scenarios and check the differences:

 echo "Mr. Brown, we'll start tagging." |\
  treex -Len Read::Sentences W2A::TokenizeOnWhitespace Write::CoNLLX

 echo "Mr. Brown, we'll start tagging." |\
  treex -Len Read::Sentences W2A::Tokenize Write::CoNLLX

 echo "Mr. Brown, we'll start tagging." |\
  treex -Len Read::Sentences W2A::EN::Tokenize Write::CoNLLX

 echo "Mr. Brown, we'll start tagging." |\
  treex -Len Read::Sentences W2A::EN::TagLinguaEn Write::CoNLLX

Now, the fourth column in CoNLLX format contains PoS (part-of-speech) tags,
but the tokenization is different than with C<W2A::EN::Tokenize>.
The reason is that L<W2A::EN::TagLinguaEn|Treex::Block::W2A::EN::TagLinguaEn>
is actually a thin wrapper for a popular Perl module L<Lingua::EN::Tagger>,
which does tokenization and tagging in one step.

We can try another tagger which is better suited for the modularity idea.
Note that C<W2A::TagTreeTagger> was not released on CPAN yet
(it is a wrapper for a binary of L<TreeTagger|http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger>)
and for C<W2A::EN::TagMorphoDiTa> you need to have installed L<Ufal::MorphoDiTa> from CPAN.

 echo "Mr. Brown, we'll start tagging." |\
  treex -Len Read::Sentences\
             W2A::EN::Tokenize\
             W2A::TagTreeTagger\
             W2A::EN::Lemmatize\
             Write::CoNLLX

Now, the third column contains lemmas, but the tags are not from the standard PennTB tagset.
As a result, also the lemmas for proper nouns are lowercased
(because L<W2A::EN::Lemmatize|Treex::Block::W2A::EN::Lemmatize> expect I<NNP> tag for proper nouns).
For English and Czech, Treex offers a pre-trained model for a high-quality L<MorphoDiTa|http://ufal.cz/morphodita> tagger.
For many other languages, there are pre-trained TreeTagger models.

 echo "Mr. Brown, we'll start tagging." |\
  treex -Len Read::Sentences\
             W2A::EN::Tokenize\
             W2A::EN::TagMorphoDiTa\
             W2A::EN::Lemmatize\
             Write::CoNLLX

 echo "Es tut mir leid." |\
  treex -Lde Read::Sentences W2A::Tokenize W2A::TagTreeTagger Write::CoNLLX
 echo "Lo siento" |\
  treex -Les Read::Sentences W2A::Tokenize W2A::TagTreeTagger Write::CoNLLX
 echo "Mi dispiace" |\
  treex -Lit Read::Sentences W2A::Tokenize W2A::TagTreeTagger Write::CoNLLX
 echo "Je suis desolée" |\
  treex -Lfr Read::Sentences W2A::Tokenize W2A::TagTreeTagger Write::CoNLLX
 echo "Bohužel jsem tento tutorial nedokončil." |\
  treex -Lcs Read::Sentences W2A::CS::Tokenize W2A::CS::TagMorphoDiTa lemmatize=1 Write::CoNLLX

=head2 Bonus Task 2

Implement a rule-based (or statistical if you dare) tokenizer for a language of your choice.
We will be happy to give you write permission to Treex SVN, so you can commit the block.

HINT: Find the source code of L<W2A::EN::Tokenize|Treex::Block::W2A::EN::Tokenize>,
copy it to F<W2A/XY/Tokenize.pm> (substitute XY for the uppercased ISO code of your language),
and change the implementation of C<tokenize_sentence> method.

=head1 DEPENDENCY PARSING

 echo "John loves Mary" |\
  treex -Len Read::Sentences\
             W2A::EN::TagLinguaEn\
             W2A::EN::Lemmatize\
             W2A::EN::ParseMSTperl\
             Write::CoNLLX

=head2 Task 3

Try to use different taggers and find sentences where different tagging leads to different parsing.
You can also try to use different parsers:
C<W2A::EN::ParseMST> (the original R. McDonald's implementation),
or C<W2A::EN::ParseMalt> (use parameter C<memory=1g>),
but those blocks (and wrappers for the Java implementation) are not released on CPAN yet,
so if you are not using the preinstalled Treex in SU2, you may need to install the parsers first.

=head2 Task 4

Treex native format F<*.treex.gz> is a actually a gzipped XML.
During the following section on readers and writers,
look inside the files (C<zless my.treex.gz>.
Check what happends when lemmatization is added.
Try to continue in the analysis and add tagging and parsing.
Visualize the individual steps using TrEd with

 ttred my.treex.gz

=head1 READERS, WRITERS AND TREEX NATIVE FORMAT

So far, we have printed all the results to STDOUT in CoNLLX format.
You can easily forward the output to a file using a standard redirection,
but you can also specify the output file with a writer's parameter C<to>.

 echo "John loves Mary" | treex -Len Read::Sentences W2A::EN::TagLinguaEn\
      Write::CoNLLX to=my.conll

Similarly, you can specify the input files with a reader's parameter C<from>.

 treex -Len Read::CoNLLX from=my.conll Write::Sentences

The parameter C<from> can contain a list of comma or space separated files.
If a file starts with I<@> character, it is interpreted as a file list
with one file name per line.
So you can do e.g.:

 ls data/pcedt*.treex.gz > my.list
 treex -Len Read::Treex from=@my.list Write::Sentences to=out.txt

For Treex file format (F<*.treex> or F<*.treex.gz>) there is a shortcut,
which automatically adds the reader to the beginning of the scenario.

 treex -Len Write::Sentences -- data/pcedt*.treex.gz

You can use treex CLI as an format convertor.

 treex -Len Read::CoNLLX from=my.conll Write::Treex to=my.treex.gz

Finally, there is another shortcut, that allows you to modify treex files in situ.

 treex -s -Len W2A::EN::Lemmatize -- my.treex.gz
 # check that lemmas were added
 treex -Len -q Write::CoNLLX -- my.treex.gz

=head1 CREATING SCENARIOS

We already know what scenario is, however there are many methods how to pass the desired scenario to the Treex.
So far, we just passed a pipeline of blocks we wanted to apply to our documents to the command line interface:

 echo "John loves Mary" |\
 treex -Len Read::Sentences\
 W2A::EN::Tokenize\
 W2A::TagTreeTagger\
 W2A::EN::Lemmatize\
 Write::CoNLLX

With larger scenarios, this input method can be quite annoying (e.g. when you want to modify previously executed scenario).
However, you can store the pipeline of blocks into a separate file (usually with .scen suffix, see F<scenarios/tagging_en_1.scen>) and substitute the blocks with that file:

 echo "John loves Mary" |\
 treex -Len Read::Sentences\
 data/tagging_en_1.scen\
 Write::CoNLLX

It is also possible to include other scenario files in the scenarios, simply insert its the relative path into the (in this case, tagging_en_1.scen) file:

 ...
 # You can also insert comments like this into the .scen files
 W2A::EN::Tokenize
 W2A::TagTreeTagger
 W2A::EN::Lemmatize

 # call "some_additional_blocks.scen" in the same directory
 some_additional_blocks.scen
 ...

This way you can write scenarios which are easier to read.
It is recommended to exclude the readers/writers from the scenario files, because you may want to use different readers/writers depending on the currently processed files.

The above method still have some drawbacks.
For example, when you are writing a description for your scenario, you can only use comments.
Also, you might want to give the user a choice of a tagger in the F<data/tagging_en_1.scen> depending on the input parameters.
For those (and other) reasons, treex supports scenarios in the format of a Perl module. For more information, see L<Treex::Tutorial::Scen>.

=head1 NEXT TASKS

Now, you can try to do some programming task,
both templates (containing specifications) and solutions are provided in C<Treex::Block::Tutorial>.

=head1 AUTHOR

Martin Popel <popel@ufal.mff.cuni.cz>

=head1 COPYRIGHT AND LICENSE

Copyright © 2011-2012 by Institute of Formal and Applied Linguistics, Charles University in Prague

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
