package Treex::Tutorial::Install;
$Treex::Tutorial::Install::VERSION = '2.20150928';
=pod

=encoding utf8

=head1 NAME

Treex::Tutorial::Install - Installation Guide for the Treex NLP framework

=head1 VERSION

version 2.20150928

=head1 SYNOPSIS

This synopsis is B<just an overview> of the six steps,
which are described L<below|/1. Prepare local Perl environment> in more detail.

=over

=item 1. Prepare your local Perl environment, so Perl modules will be installed to F<~/perl5>.

We expect no admin rights and no previous local Perl environment.
If C<env | grep PERL> prints something or directory F<~/.cpan> exists,
there is a risk that your previously installed local Perl environment will be in conflict with the new one.

 wget -O- http://cpanmin.us | perl - -l ~/perl5 App::cpanminus local::lib
 eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib`
 echo '## Treex installation ##'                        >> ~/.bashrc
 echo 'eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib`'   >> ~/.bashrc
 grep bashrc ~/.bash_profile || echo 'source ~/.bashrc' >> ~/.bash_profile

=item 2. Install Treex::Core and its dependencies from CPAN

 # First, try to install XML::LibXML
 cpanm XML::LibXML
 # If it fails and the build.log contains "looking for -lxml2... no",
 # you are probably missing libxml2 header files (or the whole libxml2).
 # On Ubuntu/Debian you can install it with
 #  sudo apt-get install libxml2-dev zlib1g-dev
 # Few more possibly problematic modules
 cpanm -n PerlIO::Util
 cpanm Moose
 moose-outdated | cpanm
 # and finally the Treex::Core and its dependencies
 cpanm Treex::Core # this may take about 10 minutes
 treex -h          # just to check if it was installed correctly

=item 3. Install Treex modules for processing English

 cpanm Treex::EN
 cpanm Lingua::Interset URI::Find Cache::LRU

=item 4. Install TrEd tree viewer and editor (optional)

See L<TrEd home page|http://ufal.mff.cuni.cz/tred> for details.
To install Perl Tk module, you need several header files,
on Ubuntu/Debian you can install them with
C<sudo apt-get install libx11-dev libxft-dev libfontconfig1-dev libpng12-dev patch>.

 # Get a script which automatically downloads and builds everything else
 wget http://ufal.mff.cuni.cz/tred/install_tred.bash
 bash install_tred.bash --tred-dir ~/tred
 # Instruct Treex where to find TrEd and its dependencies
 echo "tred_dir: $HOME/tred" >> ~/.treex/config.yaml
 echo "source ~/tred/bin/init_tred_environment" >> ~/.bashrc
 source ~/tred/bin/init_tred_environment
 ttred # run TrEd with Treex extension

=item 5. Download the newest version of the whole Treex from GIT repository (optional)

 git clone https://github.com/ufal/treex.git ~/treex
 # Add the following lines to your ~/.bashrc
 export PATH="$HOME/treex/bin:$PATH"
 export PERL5LIB="$HOME/treex/lib:$PERL5LIB"
 export TMT_ROOT=$HOME/.treex

=item 6. Install MorphoDiTa tagger and NameTag NER (optional)

 cpanm Ufal::MorphoDiTa Ufal::NameTag

=back

=head1 Prerequisites

In this tutorial, we expect Linux OS with Bash shell and Perl 5.10 or higher.
Also basic development tools, such as C<make>, C<patch>, and a C compiler (C<gcc>), are required.
You can easily use different shell (e.g. C<csh>), just modify accordingly the shell commands.
It is possible to install Treex also on MacOS and Windows+L<StrawberryPerl|http://strawberryperl.com>,
but it is less tested so far.
If you have a Perl version older than 5.10 (or if you just want to try the newest Perl),
you can install your own Perl using L<perlbrew|http://perlbrew.pl> -- it is really simple.

Note that if you have Windows and I<only> want to browse F<*.treex> files,
you can install L<TrEd|http://ufal.mff.cuni.cz/tred>
and (in menu Setup - Manage Extensions - Get New Extension) select EasyTreex extension.
However, for completing tutorial you need to install Treex (and setup TrEd)
as described below, so EasyTreex is superfluous.

=head1 1. Prepare local Perl environment

In order to install Treex, you must be able to install Perl modules from L<CPAN|http://cpan.org>.
This step is not specific to Treex, it is a basic Perl skill.
There are several ways how to achieve the goal, but I consider this the easiest one.
There are two things you should be aware of:

=over

=item *

If C<env | grep PERL> prints something or directory F<~/.cpan> exists (C<ls -l ~/.cpan>),
it is probable that you have already configured a local Perl environment.
In such a case B<it is important> to

=over

=item either reuse the environment and B<skip this step>

If you used L<local::lib> or L<perlbrew|http://perlbrew.pl> to set the environment,
it should be configured properly and you can continue with L<step 2|/2. Install Treex::Core from CPAN>
(if you want to use C<cpanm>, install it by C<cpan App::cpanminus>).
If you used another method, such as modifying C<$PERL5LIB> in your F<~/.bashrc>
or setting PREFIX or INSTALL_BASE options in C<cpan> configuration,
there is a possibility that your previously installed local Perl environment
is configured only partially and the procedure described here may fail.
If you decide to reuse your previous local Perl environment,
the modules will be installed to whatever path you had chosen (instead of F<~/perl5>)
and you should skip this step 1
(otherwise the installation fails with "I<WHOA THERE! It looks like you've got ...>" in F<~/.cpanm/build.log>).

=item or deactivate completely the environment before doing this step.

If you do not need/want to use your previous local Perl environment,
you should delete (rename) the F<~/.cpan> directory
and edit your shell profile (F<~/.bashrc>, F<~/.profile> etc),
so no Perl-related variables (such as PERL5LIB, PERL_MB_OPT, PERL_MM_OPT)
are exported. After running a new shell (new ssh session),
C<env | grep PERL> should print nothing.

=back

=item *

If you have a root access and I<really> want to install Treex and its dependencies I<for all users to system paths> (F</usr/lib> etc.),
just skip this step (you can install C<cpanm> by C<sudo cpan App::cpanminus>).
However, in course of this tutorial you will be advised to modify some of the modules (F<Treex::Block::Tutorial::*>),
so it may be a good compromise to install only the dependencies to system paths using
C<sudo cpanm --installdeps Treex::Core>, but otherwise follow this local Perl setup.

=back

Download and install locally two useful tools (Perl modules) – C<cpanm> and C<local::lib>:

 wget -O- http://cpanmin.us | perl - -l ~/perl5 App::cpanminus local::lib

L<App::cpanminus> provides L<cpanm> script which is a fast, dependency free, zero-configuration substitute for the standard L<cpan>.
L<local::lib> takes care of setting all the environment variables needed to install modules without administrative privileges.

Instead of C<wget -0->, you can use C<curl -L> or simply download F<cpanm> from L<http://cpanmin.us>, save it as F<cpanm> and run
C<perl cpanm -l ~/perl5 App::cpanminus local::lib>.
Instead of F<~/perl5>, you can use any path you like, but F<~/perl5> is a common standard used in this tutorial.

In the following steps, you can use C<cpan> instead of C<cpanm>.
The advantage is that you can start an interactive C<cpan> shell which provides more features
(I recommend to install first L<Bundle::CPAN> and L<Term::ReadLine::Perl>, so you can browse the history using up/down keys).
The disadvantage is that you cannot use it for installing C<local::lib> locally before C<local::lib> is installed :-).
Also, you will need to go through a configuration dialogue when C<cpan> is executed for the first time.

 eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib`
 echo '## Treex installation ##'                        >> ~/.bashrc
 echo 'eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib`'   >> ~/.bashrc
 grep bashrc ~/.bash_profile || echo 'source ~/.bashrc' >> ~/.bash_profile

The first line sets up the environment variables C<$PERL5LIB>, C<$PATH>, C<$PERL_MB_OPT> etc. for the current shell session.
It enables you to use the modules installed in F<~/perl5> (without specifying this path using C<perl -I>)
and also it ensures that new modules will be installed (using C<cpanm> or C<cpan>) to F<~/perl5> (not to the system paths).
The third line ensures that this setting will be applied also in other (non-login) shell sessions.
The fourth line ensures that this setting will be applied also in "login" shell sessions (e.g. when you log in via C<ssh>).
If you prefer to use F<~/.profile> instead of F<~/.bash_profile>, adapt the fourth line accordingly.

=head1 2. Install Treex::Core from CPAN

Treex is divided into several CPAN distributions.
L<Treex::Core> contains the main ("core") functionality and almost all other Treex modules depend on it.
C<Treex::Core> itself has many dependencies, most notably L<Moose> and L<Treex::PML> (which have many dependencies and so on),
so the installation takes several minutes.
One of the most frequent problems in installation is that the Perl module L<XML::LibXML>,
which is a binding for C<libxml2> library, needs apart from the library also its header files (*.h).
So let's check first, whether you can install C<XML::LibXML>:

 cpanm XML::LibXML

If it fails and F<~/.cpanm/build.log> contains "Cannot write to /usr/lib/ ... XML/SAX.pm line 191",
try to run it again and it should show that it was L<actually installed|https://rt.cpan.org/Ticket/Display.html?id=75007>.
If it fails and F<~/.cpanm/build.log> contains "looking for -lxml2... no",
you are probably missing the header files or the whole library.
On Ubuntu/Debian you can install it with:

 sudo apt-get install libxml2-dev zlib1g-dev

If you know a simple way how to do this without admin privileges, let me know.
You can check for the packages with C<< LANG=C dpkg-query -s libxml2-dev zlib1g-dev 2>&1 | grep Package >>.
On other systems (e.g. RPM based), try to find similarly named packages (libxml2-devel),
or look at L<http://xmlsoft.org>.

There are few other possibly problematic modules.
C<PerlIO::Util> has known (and reported)

 cpanm -n PerlIO::Util
 cpanm Moose
 moose-outdated | cpanm

Now, the installation of L<Treex::Core> should be smooth
(but it takes more than 8 minutes if no dependencies were installed before):

 cpanm Treex::Core

Rarely, you may encounter problems with installing some modules.
In that case, you should find the first module where something went wrong.
You can read the documentation of the module, check its L<bug tracker|https://rt.cpan.org>, try to install it manually etc.
If you cannot diagnose and fix the failure, you may try to install it with C<--prompt>, C<--force> or C<--notest> options,
but this may cause troubles later on.

 treex -h

C<treex> is the main Treex script. C<treex -h> should just print the usage information and exit.
Its actual usage will be described later on in this tutorial (L<Treex::Tutorial::FirstSteps>);
running the command serves here only as a check that C<treex> was installed and can be found in the C<$PATH>.
The installation created a configuration file F<~/.treex/config.yaml> which will be described in L<Treex::Tutorial::Config>.

=head1 3. Install Treex::EN from CPAN

Treex Core itself has no modules for any particular NLP task.
There is a separate distribution C<Treex-Unilang> for such modules that are language independent.
In this tutorial, we will mainly work with English, so you need to install a distribution C<Treex-EN>,
which contains only modules specific to English. It is dependent on C<Treex-Unilang>,
so both the distributions can be installed by:

 cpanm Treex::EN
 cpanm Lingua::Interset URI::Find Cache::LRU

=head1 4. Install TrEd

L<TrEd|http://ufal.mff.cuni.cz/tred> is a fully customizable and programmable graphical editor and viewer for tree-like structures.
Although TrEd visualization of the linguistic trees produced by Treex can be very helpful, it is not required,
i.e. Treex is fully functional even without installing TrEd.

To install Perl Tk module, you may need to install some header files and TrEd needs also the C<patch> tool.
On Ubuntu/Debian you can install these prerequisites using:

 sudo apt-get install libx11-dev libxft-dev libfontconfig1-dev libpng12-dev patch

Now, download a small installation script

 wget http://ufal.mff.cuni.cz/tred/install_tred.bash

You can type C<bash install_tred.bash -h> to see the installation options. 
To automatically download and build the latest TrEd and its dependencies to F<~/tred>, use:

 bash install_tred.bash --tred-dir ~/tred

You can run C<~/tred/bin/start_tred> to check the GUI.
When a dialog box "Manage extensions" appears, you can ignore it (click on "Later").

Treex Core contains an L<extension|Treex::Core::TredView> for TrEd,
which enables it to open F<*.treex>, F<*.treex.gz> and F<*.streex> files and use the Treex stylesheet.
Treex Core also contains a simple wrapper script C<ttred>
which runs TrEd with this extension enabled (I<pre-installed>).
We must instruct Treex where to find TrEd:

 echo "tred_dir: $HOME/tred" >> ~/.treex/config.yaml

TrEd installed some of its dependencies to F<~/tred/dependencies>,
but we want to make them permanently available for Treex (and all Perl modules):

 echo "source ~/tred/bin/init_tred_environment" >> ~/.bashrc
 source ~/tred/bin/init_tred_environment

Finally, you can run TrEd with the Treex extension enabled:

 ttred

=head1 5. Download Treex from GIT repository

Some Treex modules are not mature enough to be released on CPAN.
You may also want to test the newest Treex version or commit your own code to the repository.
So let's create your local clone of Treex in F<~/treex>.

 git clone https://github.com/ufal/treex.git ~/treex

You need to include the path to the downloaded modules in your $PERL5LIB. Add the following lines to the end of your ~/.bashrc:

 export PATH="$HOME/treex/bin:$PATH"
 export PERL5LIB="$HOME/treex/lib:$HOME/treex/oldlib:$PERL5LIB"
 export TMT_ROOT=$HOME/.treex

It is important that these lines follow eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib` in your ~/.bashrc, so a GIT module is preferred over a CPAN modules of the same name. To apply the setting for the current bash session, type the three export commands or start a new session. You can check it with:

 echo $PERL5LIB  # ~/treex/lib should precede ~/perl5/...
 treex -v        # should print "Treex version: DEV from..."

Now you can use Perl modules that were not installed from CPAN (but were downloaded from GIT).
Some of the modules may have dependencies that you do not have (installed).
When you load such a module (e.g. by running C<treex>) it will fail
with an error message like
C<Can't locate Acme/Time/Baby.pm in @INC (@INC contains:...>
You can install the missing dependencies (Acme::Time::Baby in this imaginary example) simply with

 cpanm Acme::Time::Baby

If you happen to need any of the modules C<CzechMorpho>, C<Morce::Czech> and C<Morce::English>,
you must install them manually, because these modules were not released on CPAN,
but they are XS-based (involve compiling C code), so you cannot just download them.

 svn --username public export $SVN_TRUNK/libs/packaged /tmp/packaged
 cd /tmp/packaged/Morce-English
 perl Build.PL
 ./Build
 ./Build test
 ./Build install --prefix $HOME/perl5/lib/perl5

In the same way, you can install C<CzechMorpho> and C<Morce-Czech>
(in this order because the latter depends on the former).

=head1 6. Install MorphoDiTa tagger and NameTag NER (optional)

L<MorphoDiTa|http://ufal.mff.cuni.cz/morphodita> is an open-source tool for morphological analysis of natural language texts.
Currently there is a Perl module, C<Ufal::MorphoDiTa>, available on CPAN providing bindings to the MorphoDiTa library.
This module is necessary for running C<Treex::Tool::Tagger::MorphoDiTa> and consequently C<Treex::Block::W2A::EN::TagMorphoDiTa>.

To compile the module, C++11 compiler is needed, either g++ 4.7 or newer, alternatively clang 3.2 or newer.
You may check if you have the required compiler installed on your computer.

 g++ --version
 # Or alternatively ...
 clang --version

When not installed, install it. On Ubuntu/Debian etc. use this command:

 sudo apt-get install g++ 

When the installed compiler version is too old, upgrade it. On Ubuntu/Debian etc. use this command:

 sudo apt-get upgrade g++

Finally, you can install the module:

 cpanm Ufal::MorphoDiTa


Another useful tool is L<Ufal::NameTag|http://ufal.mff.cuni.cz/nametag>, a tool for named entity recognition.
It should have similar prerequisities as Ufal::MorphoDiTa, so if you followed the previous steps, just install the module.

 cpanm Ufal::NameTag

=head1 Uninstall

Although there is no standardized way to uninstall Perl modules,
in most cases it is enough to delete the respective files and directories.
If you followed this installation guide
and you want to remove all the installed stuff
and B<if you had nothing in F<~/perl5> before>,
you can delete the directories
F<~/perl5>, F<~/treex>, F<~/.treex>, F<~/.tred> and F<~/.cpanm>.
You can also delete the added lines from F<~/.bashrc>
(starting with I<## Treex installation ##>) and F<~/.bash_profile>.

=head1 AUTHOR

Martin Popel <popel@ufal.mff.cuni.cz>

Dušan Variš <varis@ufal.mff.cuni.cz>

=head1 COPYRIGHT AND LICENSE

Copyright © 2012 by Institute of Formal and Applied Linguistics, Charles University in Prague

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
