# Copyright: 2001-2005 The Perl Foundation.  All Rights Reserved.
# $Id: tests.pod 7788 2005-04-06 21:43:11Z bernhard $

=head1 NAME

docs/tests.pod - Testing Parrot

=head1 A basic guide to writing tests for Parrot

This is quick and dirty pointer to how tests for Parrot should be
written.  The testing system is liable to change in the future, but
tests written following the guidelines below should be easy to port
into a new test suite.

=head1 How to write a test

New tests should be added to F<*.t> files. These test files 
can be found in the directories F<t>, F<imcc/t> and F<languages/*/t>.
If a new feature is tested,
it might also make sense to create a new F<*.t> file.

The testing framework needs to know how many tests it should expect.
So the number of planned tests needs to be
incremented when adding a new test.
This is done near the top of a test file, in a line that looks like:

  use Parrot::Test tests => 8;

=head2 Parrot Assembler

PASM test are mostly used for testing ops.
Appropriate test files for basic ops are F<t/op/*.t>.
Perl Magic Cookies are tested in F<t/pmc/*.t>.
Add the new test like this:

    pasm_output_is(<<'CODE', <<'OUTPUT', "name for test");
        *** a big chunk of assembler, eg:
        print   1
        print   "\n" # you can even comment it if it's obscure
        end          # don't forget this...!
    CODE
    *** what you expect the output of the chunk to be, eg.
    1
    OUTPUT

=head2 Parrot Intermediate Representation

Tests can also be written in B<PIR>. This is done with 
C<pir_output_is> and friends.

    pir_output_is(<<'CODE',<<'OUT','nothing useful');
        .include 'library/config.imc'

        .sub main @MAIN
            print "hi\n"
        .end
    CODE
    hi
    OUT

=head2 C source tests

C source tests are usually located in F<t/src/*.t>.
A simple test looks like:
 
    c_output_is(<<'CODE', <<'OUTPUT', "name for test");
    #include <stdio.h>
    #include "parrot/parrot.h"
    #include "parrot/embed.h"

    static opcode_t *the_test(Parrot_Interp, opcode_t *, opcode_t *);

    int main(int argc, char* argv[]) {
        Parrot_Interp interpreter;
        interpreter = Parrot_new(NULL);

        if (!interpreter)
        return 1;

        Parrot_init(interpreter);
        Parrot_run_native(interpreter, the_test);
        printf("done\n");
    fflush(stdout);
        return 0;
    }

    static opcode_t*
    the_test(Parrot_Interp interpreter,
        opcode_t *cur_op, opcode_t *start)
    {
        /* Your test goes here. */

        return NULL;  /* always return NULL */
    }
    CODE
    # Anything that might be output prior to "done".
    done
    OUTPUT

Note that it's always a good idea to output "done" to confirm that the compiled
code executed completely. When mixing C<printf> and C<PIO_printf> always
append a C<fflush(stdout);> after the former.

=head2 Testing language implementations

Language implementations are usually tested with the test function 
C<language_output_is>.

=head1 Ideal tests:

=over 4

=item o

Probe the boundaries (including edge cases, errors thrown etc.) of
whatever code they're testing.  These should include potentially
out of band input unless we decide that compilers should check for this
themselves.

=item o

Are small and self contained, so that if the tested feature breaks we
can identify where and why quickly.

=item o

Are valid. Essentially, they should conform to the additional documentation
that accompanies the feature (if any). [If there isn't any documentation,
then feel free to add some and/or complain to the mailing list].

=item o

Are a chunk of assembler and a chunk of expected output.

=back

=head1 TODO tests

In test driven development, tests are implemented first.
So the tests are initially expected to fail.
This can be expressed by marking the tests as TODO. 
See L<Test::More> on how to do that.

=head1 SEE ALSO

L<http://qa.perl.org>

=cut
