=encoding utf8


=head1 Name

Outthentic


=head1 Synopsis

Multipurpose scenarios framework.


=head1 Build status

L<![Build Status](https://travis-ci.org/melezhik/outthentic.svg)|https://travis-ci.org/melezhik/outthentic>


=head1 Install

    $ cpanm Outthentic


=head1 Short introduction

This is a quick tutorial on outthentic usage.


=head2 Run your scenarios

Scenario is just a script that you B<run> and that yields something into B<stdout>.

Perl scenario example:

    $ cat story.pl
    
    print "I am OK\n";
    print "I am outthentic\n";

Ruby scenario example:

    $ cat story.rb
    
    puts "I am OK"
    puts "I am outthentic"

Bash scenario example:

    $ cat story.bash
    
    echo I am OK
    echo I am outthentic

Outthentic scenarios could be written on one of three languages:

=over

=item *

Perl 


=item *

Ruby


=item *

Bash


=back

Choose you favorite language ;) !

Outthentic relies on file names to determine scenario language. 

This is the table to describe language / file name conventions:

    +-----------+--------------+
    | Language  | File         |
    +-----------+--------------+
    | Perl      | story.pl     |
    | Ruby      | story.rb     |
    | Bash      | story.bash   |
    +-----------+--------------+


=head2 Check file

Check file contains rules to B<verify> a stdout produced by scenario script. 

Here we require that our scenario should produce  `I am OK' and `I am outthentic' lines in stdout:

    $ cat story.check
    
    I am OK
    I am outthentic


=head2 Story

Outthentic story is a scenarios + check file. When outthentic B<run> story it:

=over

=item *

executes scenario script and saves stdout into file.


=item *

verifies stdout against a check file


=back

See also L<story runner|#story-runner>.


=head2 Suite

Outthentic suite is a bunch of related stories. You may also call suite as projects.

One may have more then one story at the project.

Just create a new directories with a story data inside:

    $ mkdir perl-story
    $ echo 'print "hello from perl";' > perl-story/story.pl
    $ echo 'hello from perl' > perl-story/story.check
    
    $ mkdir ruby-story
    $ echo 'puts "hello from ruby"' > ruby-story/story.rb
    $ echo 'hello from ruby' > ruby-story/story.check
    
    $ mkdir bash-story
    $ echo 'echo hello from bash' > bash-story/story.bash
    $ echo 'hello from bash' > bash-story/story.check

Now, let's use C<strun> command to run suite:

    $ strun
    
    /tmp/.outthentic/19311/home/melezhik/projects/outthentic-demo/bash-story/story.t .. 
    ok 1 - output match 'hello from bash'
    1..1
    ok
    /tmp/.outthentic/19311/home/melezhik/projects/outthentic-demo/perl-story/story.t .. 
    ok 1 - output match 'hello from perl'
    1..1
    ok
    /tmp/.outthentic/19311/home/melezhik/projects/outthentic-demo/ruby-story/story.t .. 
    ok 1 - output match 'hello from ruby'
    1..1
    ok
    All tests successful.
    Files=3, Tests=3,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.18 cusr  0.04 csys =  0.24 CPU)
    Result: PASS

Summary:

=over

=item *

Stories are executed independently, means I<generally> one story B<is not dependent upon> another story (but see "downstream stories section").      




=item *

Outthentic suite is bunch of related scenarios ( one or many ) to B<be run> and to B<be verified> ( by analyzing scenarios output against rules )



=item *

strun - suite runner - produces result in L<TAP|https://testanything.org/> format, which is  is a simple text-based interface between testing modules in a test harness.



=back


=head1 Calculator project example

Here is more detailed tutorial where we will build a test suite for calculator program.

Let's repeat it again - there are three basic outthentic entities: 

=over

=item *

suite


=item *

scenarios 


=item *

check files


=back


=head2 Project

Outthentic project is a bunch of related stories. Every project is I<represented> by a directory.

Let's create a project to test a simple calculator application:

    $ mkdir calc-app
    $ cd calc-app


=head2 Scenarios

Scenarios are just a scripts to be executed so that their output to be verified by rules defined at check files.

In other words, every story is like a small program to be executed and then gets tested ( by it's output )

Let's create two stories for our calc project. One story for `addition' operation and another for `multiplication':

    # story directories
    
    $ mkdir addition # a+b
    $ mkdir multiplication # a*b
    
    
    # scenarios
    
    $ cat  addition/story.pl
    use MyCalc;
    my $calc = MyCalc->new();
    print $calc->add(2,2), "\n";
    print $calc->add(3,3), "\n";
    
    $ cat  multiplication/story.pl
    use MyCalc;
    my $calc = MyCalc->new();
    print $calc->mult(2,3), "\n";
    print $calc->mult(3,4), "\n";


=head2 Check files

Check file contains validation rules to test script output. Every scenario B<is always accompanied by> story check file. 

Check files should be placed at the same directory as scenario and be named as C<story.check>.

Lets add some rules for `multiplication' and `addition' stories:

    $ cat addition/story.check
    4
    6
     
    $ cat multiplication/story.check
    6
    12

And finally lets run our suite:

    $ strun


=head1 Story runner

Story runner - is a script to run outthentic stories. It is called C<strun>.

Runner consequentially goes several phases:


=head2 A compilation phase. 

Stories are converted into perl test files *.t ( compilation phase ) and saved into temporary directory.


=head2 An execution phase. 

L<Prove|https://metacpan.org/pod/distribution/Test-Harness/bin/prove> utility recursively executes 
test files under temporary directory and thus produces a B<final suite execution status>.

So, after all outthentic project I<is very like> a perl test project with *.t files inside, with the difference that
outthentic *.t files I<are generated  by story files> dynamically.


=head1 Check files syntax

Outthentic consumes L<Outthentic::DSL|https://github.com/melezhik/outthentic-dsl>, so check files contain
rules defined in terms of Outthentic DSL - a language to validate unstructured text data.

Below some examples of check file syntax, you may learn more at Outthentic::DSL documentation.


=head2 plain strings checks

Often all you need is to ensure that stdout has some strings in:

    # scenario stdout
    
    HELLO
    HELLO WORLD
    123456
    
    
    # check file
    
    HELLO
    123
    
    # verification output
    
    OK - output matches 'HELLO'
    OK - output matches 'HELLO WORLD'
    OK - output matches '123'


=head2 regular expressions

You may use regular expressions as well:

    # check file
    
    regexp: L+
    regexp: \d
    
    
    # verification output
    
    OK - output matches /L+/
    OK - output matches /\d/

See L<check-expressions|https://github.com/melezhik/outthentic-dsl#check-expressions> in Outthentic::DSL documentation pages.


=head2 inline code, generators and asserts

You may inline code from other language to add some extra logic into your check file:


=head3 Inline code

    # check file
    
    code: <<CODE
    !bash
    echo '# this is debug message will be shown at console'
    CODE
    
    code: <<CODE
    !ruby
    puts '# this is debug message will be shown at console'
    CODE
    
    code: <<CODE
    # by default Perl language is used
    note('this is debug message will be shown at console')
    CODE


=head3 generators

You may generate new B<check entries> on runtime:

    # check file
    # with 2 check entries
       
    Say
    HELLO
       
    generator: <<CODE
    !bash
    
    echo say 
    echo hello 
    echo again
    
    CODE
    
    # a new check list would be:
       
    Say
    HELLO
    say
    hello
    again

Here examples on using other languages in generator expressions:

Perl:

    generator: <<CODE
    !perl
    [ 
      qw { say hello again } 
    ]
    
    CODE

Ruby:

    generator: <<CODE
    !ruby
    puts 'say'
    puts 'hello'
    puts 'again'
    
    CODE


=head3 asserts

Asserts are statements returning true of false with some extra text description.

Asserts are very powerful feature when combined with B<captures> and B<generators>:

    # scenario output
    
    ten       for 10
    twenty   for 20
    thirty    for 30
    
    # check file
    
    regexp: \w+\s+for\s(\d+)
    
    generator: <<CODE
    !ruby
      sum=0
      (captures()).each do |c|
        sum+=c.first
      end
    puts "assert: #{sum==60} sum should be 60!"
    CODE  

Follow L<code expressions|https://github.com/melezhik/outthentic-dsl#code-expressions>, L<generators|https://github.com/melezhik/outthentic-dsl#generators> and L<asserts|https://github.com/melezhik/outthentic-dsl#asserts>
in Outthentic::DSL documentation pages to learn more about code expressions, generators and asserts.


=head3 text blocks

Need to validate that some lines goes successively?

    # stdout
    
    this string followed by
    that string followed by
    another one string
    with that string
    at the very end.
    
    
    # check list
    # this text block
    # consists of 5 strings
    # goes consequentially
    # line by line:
    
    begin:
        # plain strings
        this string followed by
        that string followed by
        another one
        # regexps patterns:
    regexp: with (this|that)
        # and the last one in a block
        at the very end
    end:

See L<comments-blank-lines-and-text-blocks|https://github.com/melezhik/outthentic-dsl#comments-blank-lines-and-text-blocks> in Outthentic::DSL documentation pages.


=head1 Hooks

Story hooks are extension points to change L<story runner|#story-runner> behavior. 

It's just a scripts gets executed I<before scenario script>. 

You should name your hooks as C<hook.*> and place them into story directory

    $ cat perl/hook.pl
    
    print "this is a story hook!";

Hooks could be written on one of three languages:

=over

=item *

Perl 


=item *

Ruby


=item *

Bash


=back

Here is naming convention for hook files:

    +-----------+--------------+
    | Language  | File         |
    +-----------+--------------+
    | Perl      | hook.pl      |
    | Ruby      | hook.rb      |
    | Bash      | hook.bash    |
    +-----------+--------------+

Reasons why you might need a hooks:

=over

=item *

execute some I<initialization code> before running a scenario script


=item *

redefine scenario stdout


=item *

call downstream stories


=back


=head1 Hooks API

Story hooks API provides several functions to hack into story runner execution process:


=head2 Redefine stdout

Redefining stdout feature means you define a scenario output on the hook side ( thus scenario script is never executed ). 

This might be helpful when for some reasons you do not want to run or you don't have by hand a proper scenario script.

This is simple an example:

    $ cat hook.pl
    set_stdout("THIS IS I FAKE RESPONSE \n HELLO WORLD");
    
    $ cat story.check
    THIS IS FAKE RESPONSE
    HELLO WORLD

You may call C<set_stdout()> more then once:

    set_stdout("HELLO WORLD");
    set_stdout("HELLO WORLD2");

An effective scenario stdout will be:

    HELLO WORLD
    HELLO WORLD2

Here is C<set_stdout()> function signatures list for various languages:

    +-----------+-----------------------+
    | Language  | signature             |
    +-----------+-----------------------+
    | Perl      | set_stdout(SCALAR)    |
    | Ruby      | set_stdout(STRING)    |
    | Bash      | set_stdout(STRING)    |
    +-----------+-----------------------+

IMPORTANT: You should only use a set_stdout inside story hook, not scenario file.


=head2 Upstream and downstream stories

It is possible to run one story from another with the help of downstream stories.

Downstream stories are reusable stories ( aka modules ). 

Story runner never executes downstream stories I<directly>.

Downstream story always gets called from the I<upstream> one. This is example:

    $ cat modules/knock-the-door/story.rb
    
    # this is a downstream story
    # to make story downstream
    # simply create story files 
    # in modules/ directory
    
    puts 'knock-knock!'
     
    $ cat modules/knock-the-door/story.check
    knock-knock!
    
     
    $ cat open-the-door/hook.rb
    
    # this is a upstream story
    # to run downstream story
    # call run_story function
    # inside upstream story hook
    
    # with a single parameter - story path,
    # notice that you have to remove
    # `modules/' chunk from story path parameter
    
    run_story( 'knock-the-door' );
    
    $ cat open-the-door/story.rb
    puts 'opening ...' 
    
    $ cat open-the-door/story.check
    opening
     
    $ strun 
    /tmp/.outthentic/3815/home/melezhik/projects/outthentic-dsl-examples/downstream/open-the-door/story.t .. 
    ok 1 - output match 'knock-knock!'
    ok 2 - output match 'opening'
    1..2
    ok
    All tests successful.
    Files=1, Tests=2,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.05 cusr  0.01 csys =  0.08 CPU)
    Result: PASS

Summary:

=over

=item *

to make story a downstream simply create a story  in a C<modules/> directory.



=item *

to run downstream story call C<run_story(story_path)> function inside upstream story hook.



=item *

you can call as many downstream stories as you wish.



=item *

you can call the same downstream story more than once.



=item *

downstream stories in turn may call other downstream stories.



=back

Here is an example of multiple downstream story calls:

    $ mkdir module/up
    $ mkdir module/down
    $ echo 'UP!' > module/up/story.check
    $ echo 'and DOWN!' > module/down/story.check
    $ echo 'print qq{UP!}' > modules/up/story.pl 
    $ echo 'print qq{DOWN!}' > modules/down/story.pl 
    
    $ cat two-jumps/hook.pl
    
    run_story( 'up' );
    run_story( 'down' );
    run_story( 'up' );
    run_story( 'down' );


=head3 story variables 

You may pass a variables to downstream story using second argument of C<run_story()>  function. For example:

    $ mkdir modules/greeting
    
    $ cat hook.pl
    
    run_story( 
      'greeting', {  name => 'Alexey' , message => 'hello' }  
    );

Or using Ruby:

    $ cat hook.rb
    
    run_story  'greeting', {  'name' => 'Alexey' , 'message' => 'hello' }

Or Bash:

    $ cat hook.bash
    
    run_story  greeting name Alexey message hello 

Here is the C<run_story> signature list for various languages:

    +-----------+----------------------------------------------+
    | Language  | signature                                    |
    +-----------+----------------------------------------------+
    | Perl      | run_story(SCALAR,HASHREF)                    |
    | Ruby      | run_story(STRING,HASH)                       | 
    | Bash      | run_story(STORY_NAME NAME VAL NAME2 VAL2 ... | 
    +-----------+----------------------------------------------+

Story variables are accessible via C<story_var()> function. For example:

    $ cat modules/greeting/story.rb
    
    puts "#{story_var('name')} say #{story_var('message')}"

Or if you use Perl:

    $ cat modules/greeting/story.pl
    
    print (story_var('name')).'say '.(story_var('message'))

Or finally Bash (1-st way):

    $ cat modules/greeting/story.bash
    
    echo $name say $message

Bash (2-nd way):

    $ cat modules/greeting/story.bash
    
    echo $(story_var name) say $(story_var message)

You may access story variables inside story hooks and check files as well.

And finally:

=over

=item *

downstream stories may invoke other downstream stories.



=item *

you can't only use story variables inside downstream stories.



=back

Here is the how you access story variable in all three languages

    +------------------+---------------------------------------------+
    | Language         | signature                                   |
    +------------------+---------------------------------------------+
    | Perl             | story_var(SCALAR)                           |
    | Ruby             | story_var(STRING)                           | 
    | Bash (1-st way)  | $foo $bar ...                               |
    | Bash (2-nd way)  | $(story_var foo.bar)                        |
    +------------------+---------------------------------------------+


=head2 Story properties

Some story properties have a proper accessors functions. Here is the list:

=over

=item *

C<project_root_dir()> - Root directory of outthentic project.



=item *

C<test_root_dir()> - Test root directory. Root directory of generated perl test files , see also L<story runner|#story-runner>.



=item *

C<config()> - Returns suite configuration hash object. See also L<suite configuration|#suite-configuration>.



=back


=head2 Ignore unsuccessful story code

Every story is a script gets executed and thus returning an exit code. If exit code is bad (!=0)
this is treated as story verification failure. 

Use C<ignore_story_err()> function to ignore unsuccessful story code:

    $ cat hook.rb
    
    ignore_story_err 1


=head2 Story libraries

Story libraries are files to keep your libraries code to I<automatically required> into story hooks and check files context:

Here are some examples:

Perl:

    $ cat common.pm
    sub abc_generator {
      print $_, "\n" for a..z;
    } 
    
    $ cat story.check
    
    generator: <<CODE;
    !perl
      abc_generator()
    CODE

Ruby:

    $ cat common.rb
    def super_utility arg1, arg2
      # I am cool! But I do nothing!
    end
    
    $ cat hook.pl
    
    super_utility 'foo', 'bar'

Here is the list for library file names for various languages:

    +-----------+-----------------+
    | Language  | file            |
    +-----------+-----------------+
    | Perl      | common.pm       |
    | Ruby      | common.rb       |
    | Bash      | common.bash     |
    +-----------+-----------------+


=head1 Language libraries


=head2 Perl

I<PERL5LIB>

$project_root_directory/lib path gets added to $PERL5LIB variable. 

This make it easy to place custom Perl modules under project root directory:

    $ cat my-app/lib/Foo/Bar/Baz.pm
    package Foo::Bar::Baz;
    1;
    
    $ cat common.pm
    use Foo::Bar::Baz;


=head1 Story runner client

    $ strun <options>


=head2 Options

=over

=item *

C<--root>



=back

Root directory of outthentic project. If root parameter is not set current working directory is assumed as project root directory.

=over

=item *

C<--debug> 


=back

Enable/disable debug mode:

    * Increasing debug value results in more low level information appeared at output
    
    * Default value is 0, which means no debugging 
    
    * Possible values: 0,1,2,3

=over

=item *

C<--silent> 


=back

Run in silent mode. By default strun prints all scenarios output (in green color), to disable this choose C<--silent> option.

=over

=item *

C<--match_l> 


=back

Truncate matching strings. In a TAP output truncate matching strings to {match_l} bytes;  default value is 200.

=over

=item *

C<--story> 


=back

Run only single story. This should be file path without extensions ( .pl, .rb, .check ):

    foo/story.pl
    foo/bar/story.rb
    bar/story.pl
    
    --story foo # runs foo/ stories
    --story foo/story # runs foo/story.pl
    --story foo/bar/ # runs foo/bar/ stories

=over

=item *

C<--prove> 


=back

Prove parameters. See L<prove settings|#prove-settings> section for details.

=over

=item *

C<--ini>



=back

Configuration ini file path.

See L<suite configuration|#suite-configuration> section for details.

=over

=item *

C<--yaml> 


=back

YAML configuration file path. 

See L<suite configuration|#suite-configuration> section for details.

=over

=item *

C<--json> 


=back

JSON configuration file path. 

See L<suite configuration|#suite-configuration> section for details.


=head1 Suite configuration

Outthentic projects are configurable. Configuration data is passed via configuration files.

There are three type of configuration files are supported:

=over

=item *

Config::General format


=item *

YAML format


=item *

JSON format


=back

Config::General  style configuration files are passed by C<--ini> parameter

    $ strun --ini /etc/suites/foo.ini
    
    $ cat /etc/suites/foo.ini
    
    <main>
    
      foo 1
      bar 2
    
    </main>

There is no special magic behind ini files, except this should be L<Config::General|https://metacpan.org/pod/Config::General> compliant configuration file.

Or you can choose YAML format for suite configuration by using C<--yaml> parameter:

    $ strun --yaml /etc/suites/foo.yaml
    
    $ cat /etc/suites/foo.yaml
    
    main :
      foo : 1
      bar : 2

Unless user sets path to configuration file explicitly by C<--ini> or C<--yaml> or C<--json>  story runner looks for the 
files named suite.ini and I<then> ( if suite.ini is not found ) for suite.yaml, suite.json at the current working directory.

If configuration file is passed and read a related configuration data is accessible via config() function, 
for example in story hook file:

    $ cat hook.pl
    
    my $foo = config()->{main}->{foo};
    my $bar = config()->{main}->{bar};

Examples for other languages:

Ruby:

    $ cat hook.rb
    
    foo = config['main']['foo']
    bar = config['main']['bar']

Bash:

    $ cat hook.bash
    
    foo=$(config main.foo )
    bar=$(config main.bar )


=head1 Runtime configuration

Runtime configuration parameters is way to override suite configuration data. Consider this example:

    $ cat suite.ini
    <foo>
      bar 10
    </foo>
      
    $ strun --param foo.bar=20

This way we will override foo.bar to value `20'.


=head1 TAP

Story runner emits results in a L<TAP|https://testanything.org/> format.

You may use your favorite TAP parser to port result to another test / reporting systems.

Follow L<TAP|https://testanything.org/> documentation to get more on this.

Here is example for having output in JUNIT format:

    strun --prove "--formatter TAP::Formatter::JUnit"


=head1 Prove settings

Story runner uses L<prove utility|https://metacpan.org/pod/distribution/Test-Harness/bin/prove> to execute generated perl tests,
you may pass prove related parameters using C<--prove-opts>. Here are some examples:

    strun --prove "-Q" # don't show anythings unless test summary
    strun --prove "-q -s" # run prove tests in random and quite mode


=head1 Environment variables

=over

=item *

C<match_l> - In a suite runner output truncate matching strings to {match_l} bytes. See also C<--match_l> in L<options|#options>.


=back


=head1 Examples

An example outthentic project lives at examples/ directory, to run it say this:

    $ strun --root examples/


=head1 AUTHOR

L<Aleksei Melezhik|mailto:melezhik@gmail.com>


=head1 Home Page

https://github.com/melezhik/outthentic


=head1 See also

=over

=item *

L<Sparrow|https://github.com/melezhik/sparrow> - outthentic suites manager.



=item *

L<Outthentic::DSL|https://github.com/melezhik/outthentic-dsl> - Outthentic::DSL specification.



=item *

L<Swat|https://github.com/melezhik/swat> - web testing framework consuming Outthentic::DSL.



=item *

Perl prove, TAP, Test::More



=back


=head1 Thanks

To God as the One Who inspires me to do my job!
