package Treex::Tutorial::FirstSteps;

=pod

=head1 NAME

Treex::Tutorial::FirstSteps - First steps after installing Treex


=head1 VERSION

version 0.07297

=head1 SHORT INTRODUCTION

The elementary unit of code in Treex is called B<block>.
Each block should solve some well defined and usually lingustically motivated
task, e.g. tokenization, tagging or parsing.

A sequence of blocks is called C<scenario> and it can describe end-to-end NLP
application, e.g. machine translation or preprocessing of a parallel treebank. 

Treex applications can be executed from Perl.
However, in this tutorial we'll start with the command line interface C<treex>.

=head1 HELLO WORLD

We will start traditionally with the "Hello, world!" example :-). 

  echo 'Hello, world!' | treex Read::Text language=en Write::Text language=en

The desired output was printed to STDOUT, but there are some info messages
around printed to STDERR. To filter out these messages you can either use the
C<--quite> option (C<-q>) or the standard redirection of STDERR.

  echo 'Hello, world!' | treex -q Read::Text language=en Write::Text language=en
  echo 'Hello, world!' | treex Read::Text language=en Write::Text language=en 2>/dev/null

=over

=item What does the syntax mean?

I<Read::Text language=en Write::Text language=en> is a scenario definition.
The scenario consists of two blocks: I<Read::Text> and I<Write::Text>.
Each block has one parameter set, the name of the parameter is C<language>
and its value is C<en> (which is ISO 639-1 code for English).

=item Why is the C<language> parameter needed?

One Treex document can contain sentences in more language
(which is useful for task like word alignment or machine translation),
so it is necessary to instruct each block on which language it should operate.

=item Can I make the scenario description shorter? 

It is not necessary to repeat the same parameter specification for every block.
You can use a special block C<Util::SetGlobal>:

  echo 'Hello, world!' | treex -q Util::SetGlobal language=en Read::Text Write::Text

=item Can I make it even shorter?

Yes. (And I know the previous example was not actually shorter.)
There is an option C<--language> (C<-L>) which is just a shortcut
for C<Util::SetGlobal language=..>.

  echo 'Hello, world!' | treex -q --language=en Read::Text Write::Text
  echo 'Hello, world!' | treex -q -Len Read::Text Write::Text
  
=back

The "Hello, world!" example is silly. The first block (so-called B<reader>)
read the plain text input, converted it to the Treex in-memory document
representation and this document was passed to the second block
(so-called B<writer>) that converted it to plain text and printed on STDOUT.
No (linguistic) processing was done.

There are readers and writers for various other formats than plain text
(e.g. HTML, CoNLL, PennTB MRG, PDT PML), so you can use it for format
conversions (see L<Treex::Tutorial::ReadersAndWriters>).
You can also create you own readers and writers for new formats
(see L<Treex::Tutorial::WritingNewReaders>).

For simplicity, we'll continue to use plain text format
in this tutorial chapter, but we'll try to do something slightly more interesting. 

=head1 SEGMENTATION TO SENTENCES

To segment a text into sentences we can use block C<W2A::Segment>
and writer C<Write::Sentences> that prints each sentence on a separate line.

  echo "Hello! Mr. Brown, how are you?" \
   | treex -Len Read::Text W2A::Segment Write::Sentences

You can see, that the text was segmented into three sentences:
"I<Hello!>", "I<Mr.>", and "I<Brown, how are you?>".
Block C<W2A::Segment> is language independent (at least for languages using
Latin alphabet) and it finds sentence boundaries just based on a regex rules
that detect end-sentence punctuation ([.?!]) followed by a capital letter.
To get the correct segmentation we must use C<W2A::EN::Segment> which has a list
of English words (tokens) that usually do not end a sentence
even if they are followed by a fullstop and a capital letter.
By the way, Treex is object-oriented, blocks are classes and
C<W2A::EN::Segment> is a descendant of the C<W2A::Segment> base class.

  echo "Hello! Mr. Brown, how are you?" \
   | treex -Len Read::Text W2A::EN::Segment Write::Sentences

=over

=item Where can I find blocks' source code?

All Treex blocks are stored in C<$TMT_ROOT/treex/lib/Treex/Block/>.
The full name of the C<W2A::EN::Segment> module is actually
C<Treex::Block::W2A::EN::Segment>, but since the prefix C<Treex::Block::>
is common to all blocks, it is not written in the scenario description.

=item What does the name C<W2A::EN::Segment> mean?

All Treex blocks that do shallow linguistic analysis
(segmentation, tokenization, lemmatization, PoS tagging, dependency parsing)
are grouped in a directory C<W2A> (W and A are names of the two layers of
language description, but this will be explained L<later|TODO>). 
Language specific blocks are stored in a subdirectory with a uppercase ISO code
of the given language (C<EN>) in this case.

=item How to read already segmented input?

If you have C<sample.txt> with one sentence per line, you can load it to Treex using

  cat sample.txt | treex -Len Read::Sentences ...

=back

There are many other options for segmentation, see (perldoc for) modules
L<Treex::Block::W2A::Segment>,
L<Treex::Block::W2A::SegmentOnNewlines>, and
L<Treex::Block::W2A::ResegmentSentences>.

=head1 TOKENIZATION, LEMMATIZATION, TAGGING

  echo "Mr. Brown, we'll start tagging." |\
   treex -Len Read::Sentences W2A::TokenizeOnWhitespace Write::CoNLLX

  echo "Mr. Brown, we'll start tagging." |\
   treex -Len Read::Sentences W2A::Tokenize Write::CoNLLX

  echo "Mr. Brown, we'll start tagging." |\
   treex -Len Read::Sentences W2A::EN::Tokenize Write::CoNLLX

  echo "Mr. Brown, we'll start tagging." |\
   treex -Len Read::Sentences\
              W2A::EN::Tokenize\
              W2A::TagTreeTagger\
              W2A::EN::Lemmatize\
              Write::CoNLLX

  echo "Mr. Brown, we'll start tagging." |\
   treex -Len Read::Sentences\
              W2A::EN::Tokenize\
              W2A::EN::TagMorce\
              W2A::EN::Lemmatize\
              Write::CoNLLX

  echo "Es tut mir leid." |\
   treex -Lde Read::Sentences W2A::Tokenize W2A::TagTreeTagger Write::CoNLLX
  echo "Lo siento" |\
   treex -Les Read::Sentences W2A::Tokenize W2A::TagTreeTagger Write::CoNLLX
  echo "Mi dispiace" |\
   treex -Lit Read::Sentences W2A::Tokenize W2A::TagTreeTagger Write::CoNLLX
  echo "Je suis desolée" |\
   treex -Lfr Read::Sentences W2A::Tokenize W2A::TagTreeTagger Write::CoNLLX
  echo "Bohužel jsem tento tutorial nedokončil." |\
   treex -Lcs Read::Sentences W2A::CS::Tokenize W2A::CS::TagMorce Write::CoNLLX

=head1 TASKS

=head2 Task1:

You have an input plain text (TODO: add paragraphs_sample.txt)
where each paragraph (including headlines)
is on a separate line. Load this file into Treex and print one sentence per line.
Note that headlines do not end with a fullstop,
but they should be treated as separated sentences

HINT: See documentation of L<Treex::Block::W2A::Segment>.

=head1 AUTHOR

Martin Popel <popel@ufal.mff.cuni.cz>

=head1 COPYRIGHT AND LICENSE

Copyright © 2011 by Institute of Formal and Applied Linguistics, Charles University in Prague

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
