Test Scenario Writing Standards
Author: Denis Walker denis@ripe.net

#
Test Scenario Writing Standards

## Introduction

The purpose of testing is to check that the code is bug free and to prove that the code performs the tasks outlined in the user requirements and specifications documents.
The philosophy of testing for dbupdate is to follow the break down of the whole system into modules according to the initial design. For example the 'main' section, authorisation checks, input processing. These may be further sub-divided to make tests more targeted. For example the authorisation tests may be sub-divided into authorisation methods, pgp, inetnum status, etc. At the lowest level we have the test scenario which perfoms a single test on the smallest unit of code we consider necessary to test individually.
Many of the test scenarios will be written by the developer of a particular module, during the development phase and used for module testing in isolation of the rest of the system. For this a wrapper or test harness will be required to 'feed' the input to the module and receive the output. This document is mainly concerned with the final integration and acceptance testing. The assumption is made that the whole system is available and the tests will be run by the test system. But the standards and format for individual test scenarios should be used when writing them at the development stage to avoid having to re-format them at a later stage.
It is further assumed that we may wish to include the test system and data in future releases of the code. Therefore any use of 'live' data or internal email addresses should be avoided. Also auth: NONE should be avoided as it is quite likely that this will be deprecated in the not too distant future.
This document is only concerned with the writing of the test data for a test scenario. It does not describe how to write the filters and regular expressions used by the test program in order to verify the success or failure of the test. That is covered in the test program documentation.

## Directory Structure and File Naming Convention

The top level directory for all the tests relating to dbupdate will be:
tests/dbupdate
Below this will be six further directories
bin data conf dump log tmp

- bin contains the test program
- conf contains the top level configuration files needed by the test program
- data contains all the test scenarios relating to dbupdate
- dump contans the log files produced by the test program and dbupdate
- log contains the runtime logs for the current local test running
- tmp contains various temporary files

This allows the full set of tests to be run by simply supplying the data directory path to the test program. All tests under this directory will then be run. If an EXCLUDE file exists in the data directory, the directory paths listed there will be excluded from the test run.
Below the data directory we will have directories for the major divisions, eg:
main authorisation input_processing
Below each of these we may have directories for sub-divisions of the major test division, eg in the authorisation directory:
auth_methods inetnum_status pgp
Within the sub-division directory (or the major division if there are no sub-divisions) we have a series of directories for the test scenarios. These will be named as a sequence of numbers starting with 001 which relates to CASE 1 of this set of tests.
001 002 003
Each of these directories will contain the files necessary for one test.
There can be any number of nested sub-divisions of a major test division. The test scenario directories (001, 002) are always at the lowest level of a branch of the directory structure. They should never be mixed with other sub-division directories. So in the case of authorisation, if we had some general test scenarios as well as some well defined sub divisions we would NOT do something like this
auth_methods inetnum_status pgp 001 002 003
It should be like this
auth_methods inetnum_status pgp general
and the 001, 002, 003 above would be in the general directory.
Within a test scenario directory (001) there will be a set of files for this test. These files only exist in this lowest level of directory (001, 002). All other directories within this structure contain only other directories, and must not contain any files. These files will have the standard names loader, dbupdate, test, whois, local_filters.config

- loader - contains the objects to be loaded directly into the database using the loader program before the test is run.
- dbupdate - contains the object(s) that need to be loaded into the database using dbupdate before the test is run.
- test - contains the flags for dbupdate and the object(s) to be submitted to dbupdate to perform the test.
- whois - contains the objects which we expect to be in the database after the test is run. The contents should correspond with filters_local.config.
- filters_local.config - contains the rules for matching the output from a test.

loader, dbupdate and whois are all optional,
test and filters_local.config are mandatory.
Due to local specifics, some names in the directory tree will be ignored. Directories named "doc" and "CVS" will be ignored at any sub-directory level, as well as file with the name "README".

## Standardised Objects Used In Tests

The more standardised the objects can be throughout all the test scenarios, the easier it will be to write and to follow the tests in the long term. How far this standardisation of objects should go I am sure will be a topic of debate. What I will describe is the standard that I adopted during the re-formatting of all the test scenarios recently.
Do not include optional attributes that are not required for the specific test.
These are quite straight forward and need little explanation so I will simply list them. This list may grow as new tests are written including attributes that are not currently used in many tests.

Attribute | Value | Notes
--- | --- | ---
admin-c | TP1-DB-TEST | References standard person object loaded for all tests
tech-c | TP1-DB-TEST | References standard person object loaded for all tests
zone-c | TP1-DB-TEST | References standard person object loaded for all tests
auth | CRYPT-PW | Simplest method of auth for use when auth is not the subject of the test Always include the plain text password as an end of line comment
referral-by | TEST-DBM-MNT | References standard mntner object loaded for all tests
source | DB-TEST | Used in all objects, except when multiple sources is tested
| dbtest@ripe.net | Used wherever an attribute requires an email address, except when multiple email addresses are required (as with notification tests) This email address does not exist.

## Standard Template Objects For loader File

There are a number of objects that are commonly required for most tests. These form the basis of the loader file and can be copied as a template for all new loader files.
Additional objects required for the specific test can then be added after these standard objects.
The standard objects includes the person TP1-DB-TEST that is used for admin-c type attributes in all other objects. In many cases this person is not needed and could suffice as a dummy object. But if any of the objects that reference it need to be updated through dbupdate as part of the test then it would fail if this person object does not exist. For simplicity it is better to always include this person in the load to avoid having to think later if it is needed or not. This person should only be used in admin-c type attributes and not manipulated as part of a test. If the test requires to create/modify/delete a person then include a specific test person object in the additional objects section for this purpose.
Two basic mntner objects are included in the template. TEST-DBM-MNT for use as the referral-by mntner in other mntner objects that you may require in the test. RIPE-NCC-HM-MNT as the allocation control mntner.
The template also includes the root inetnum and inet6num objects as a starting point for any test that requires to use inetnums and routes.
Other standard objects may be added or changes made to these objects if considered necessary at a later date. If so I would suggest we write a perl (or sed) script to ripple the changes/additions throughout all existing loader files under the test data directory. This keeps the standard 'standard'.
If any of the template objects need to be modified for a specific test then it is advisable to move the object from the standard template object section into the additional objects section, for this test. This is to make it clear that it is no longer the standard object from the template.

## File Comments

There are two formats for comments that can be used in any of the data files.

- # comments Lines starting with a # symbol will be copied into the test report file by the test program.
- % comments Lines starting with a % symbol will _not_ be copied into the test report file. The test program will simply ignore them.

This allows for details of what the test is trying to do and the expected results of the test to be included in the report file along with the results from running the test. It also allows for the test scenario writer to include additional comments in the file about writing the test that may not be of interest when checking the results.
The test program will remove all lines from the files that start with a # or % or $ (see below). For tests that require a mail header it is important that this is at the start of the input file to dbupdate. The mail header MUST start at the FIRST line in the update file sent to dbupdate. So do not leave any blank lines in the 'test' file before the 'From:' line. If there is a need for spacing out lines for ease of reading, these padding lines must start with a # or %.

## The loader File

This file contains a list of objects that are to loaded directly into the database before the test can be run. If the loader file contains any key-cert objects the public keyring will also be updated with the keys. The loader file layout is shown in Appendix A, Loader File Template.
It starts with a title of the test group, eg '% inetnum status tests', then states which test the objects are for, eg '% load objects for Case 1 test'. Next are the standard template objects as described above. Then the additional objects required for this test (if any). It ends with a comment, '% End of loader data file'. The main reason for this coment is to ensure that there is a blank line after the last object in the file. Without a blank line the last object will not be loaded.

## The dbupdate File

This file contains an optional string with dbupdate command line flags and a list of objects that are to created in the database by dbupdate before the test can be run. It starts with a title of the test group, eg '% pgp tests', then states which test the objects are for, eg '% dbupdate objects for Case 1 test'. A line starting with '$' will be considered as the dbupdate command line flags string. If it exists, these flags will be used when running dbupdate. If not, the default flags will be used. Please see the documentation for the format of this line. This is followed by the object(s) and credentials that need to be submitted to dbupdate. An example is shown in Appendix B, dbupdate File Sample.

## The test File

This file contains an optional string with dbupdate command line flags and the actual test that is being performed. An example of this can be seen in Appendix C, test File Sample.
It starts with a title followed by a description of what the test will try to do. Additional comments can be placed anywhere in the comment block for the benefit of the test writer. Then we have the action to be performed by dbupdate followed by a simple statement of whether the action should be a SUCCESS or FAILURE. Then there is a description of the outcome of the test. If the optional string with dbupdate command line flags is to be used it should be inserted after the initial comment block on a line starting with a $. If it exists, these flags will be used when running dbupdate. If not, the default flags will be used. Please see the documentation for the format of this line. Following this we have the object(s) and credentials that need to be submitted to dbupdate to perform this test. If the test requires a mail header, this will come after the initial comment block and any optional flags string and before the test objects and credentials. Every empty string occured in test/dbupdate files will be included into dbupdate input, if it is not commented out.
By default, the filename for file which contains input will be added to dbupdate command line. However, this can be overriden by putting '$$' at the beginning of the string containing flags. In this case no filenames will be added.

In fact, any variable described in testing configuration file can be put into flags line. It will be expanded with it's current value.
Examples:

$$ -r -c $RIP_CONFIG -ZZZ
this line can be used to test dbupdate command-line options handling

$$ -r -c $RIP_CONFIG -f $TEST
this line will be effectively expanded to variables from current configuration. $TEST will point to 'test' file for current test.

$ -r -c $RIP_CONFIG $DBUPDATE_FILE
this will effectively do "-r -c [rip_config] -f [local test or dbupdate file]

$DBUPDATE_GEN
this will effectively do the same as above.
There is also a possibility to specify a command to run before the test. It should appear somewhere in the test file, preferrably between the test description and update in order to keep test test file clear and readable. The syntax is:
@ command_line
The testing program will check that script is run and the process is closed successfully. If command line contains some names of defined variables of the testing program, their values will be substituted from testing configuration file.
This feature can be used for example to swap configuration files before running a specific test.

## Appendix A
loader File Template

    % TITLE
    % load objects for Case n tests
    %

    %----------------------------------------------------------------
    %        Standard Template Objects

    % admin-c, tech-c person

    person:       test person
    address:      Singel
    phone:        +0123456789
    nic-hdl:      TP1-DB-TEST
    changed:      dbtest@ripe.net 20020101
    source:       DB-TEST

    % referral-by maintainer

    mntner:       TEST-DBM-MNT
    descr:        Mntner for RIPE DBM objects.
    admin-c:      TP1-DB-TEST
    tech-c:       TP1-DB-TEST
    upd-to:       dbtest@ripe.net
    auth:         CRYPT-PW xzSJqgOS7O2ZI   # test-dbm
    mnt-by:       TEST-DBM-MNT
    referral-by:  TEST-DBM-MNT
    changed:      dbtest@ripe.net 20020101
    source:       DB-TEST

    % allocation maintainer

    mntner:       RIPE-NCC-HM-MNT
    descr:        RIPE-NCC Hostmaster Maintainer
    admin-c:      TP1-DB-TEST
    tech-c:       TP1-DB-TEST
    upd-to:       dbtest@ripe.net
    auth:         CRYPT-PW qw9MK.3f3f5k2    # hostmaster
    mnt-by:       RIPE-NCC-HM-MNT
    referral-by:  TEST-DBM-MNT
    changed:      dbtest@ripe.net 20020101
    source:       DB-TEST

    % root inetnum object

    inetnum:      0.0.0.0 - 255.255.255.255
    netname:      IANA-BLK
    descr:        The whole IPv4 address space
    country:      NL
    admin-c:      TP1-DB-TEST
    tech-c:       TP1-DB-TEST
    status:       ALLOCATED UNSPECIFIED
    remarks:      The country is really worldwide.
    remarks:      This address space is assigned at various other places in
    remarks:      the world and might therefore not be in the RIPE database.
    mnt-by:       RIPE-NCC-HM-MNT
    mnt-lower:    RIPE-NCC-HM-MNT
    mnt-routes:   RIPE-NCC-HM-MNT
    changed:      dbtest@ripe.net 20020101
    source:       DB-TEST

    % root inet6num object

    inet6num:     0::/0
    netname:      ROOT
    descr:        Root inet6num object
    country:      EU
    admin-c:      TP1-DB-TEST
    tech-c:       TP1-DB-TEST
    mnt-by:       RIPE-NCC-HM-MNT
    mnt-lower:    RIPE-NCC-HM-MNT
    status:       ALLOCATED-BY-RIR
    remarks:      This network in not allocated.
                  This object is here for Database
                  consistency and to allow hierarchical
                  authorisation checks.
    changed:      dbtest@ripe.net 20020101
    source:       DB-TEST

    %      End of Standard Template Objects
    %----------------------------------------------------------------

    %  Additional objects required for this test

    % End of loader data file

## Appendix B
dbupdate File Sample

    % pgp auth tests
    % dbupdate objects for Case 1 test
    %

    key-cert: PGPKEY-4A26C717
    certif: -----BEGIN PGP PUBLIC KEY BLOCK-----
    certif: Version: GnuPG v1.0.6 (GNU/Linux)
    certif: Comment: For info see http://www.gnupg.org
    certif:
    certif: mQGiBD1jYZYRBACA4tLa4R7ZuMpQNZVe4QfH5A/Tb0eUeHrxGsR9nfeUr8CEtJe4
    certif: l6BX1jdSc/Yl7LevFx04BRzXDmMV6Xp89xuhV5vpLH/ISpIvF55EtCslW9sKee50
    certif: 4Pefd9+4KgGkMP6r0L+apqyyvMTDl4gPF/SEB6Xa6e/RfNXXCA7YSGZfAwCg33qu
    certif: lnsXAY8qQs/2SgfUROMNetsD/2Ff5NACMQJXfoNqrWOqTHdjrAtw+CrZEHgac1Gk
    certif: wa42NEWD9kUJdQcB8zdXcNfKTZ1gIepgoc0hdh5iVhJaMCiHyTSAmB2yhcYVUX1M
    certif: 5AJhUf7dKcQa0b1hn+JhyZGe5hAgxn8y7g7Hp1PSPRjkgpV8hI37D+genN669dW6
    certif: +tguA/4tn+S4oohPG0uy5IAx19aTLwZ4zjqZdrNzXU3fJvMZeA6nRDxmWv7Gfc72
    certif: UnENHJk/rhMXca7Ix3PQn5abDl5Jd7cliONLuFA2IQoiC0DV4ih4xaD8bWud/0zN
    certif: Bd5BmTkU6O61y09ucJMYGclvKIN5k0/HZD2SH0HcU8srsjuqgrQYVGlhZ28gMSA8
    certif: dGlhZ29AcmlwZS5uZXQ+iFcEExECABcFAj1jYZYFCwcKAwQDFQMCAxYCAQIXgAAK
    certif: CRBchF+OSibHF+lDAJ9eY0qO2eJS7c49U0VgLe4dUwq9NACg2eN9jI+0sjEV39nA
    certif: FrfeGE52rgS5AQ0EPWNhlxAEANqiQyIWuXbmNDmWHZlY9rbyzEl2ztzwdeRFC2Ls
    certif: /y7XNNSmW44oo9vCEugam8SqPIA3wxiLKq/Iiqs/7H+cpXwh2mbb3QAKLiSIF/C/
    certif: BJ+JWqLssGfqdEiSicyLKLT37zMcCWKRHtdHBxcZHqQIDzbGcpDggZNmDloMGVKJ
    certif: aQ5HAAMFBACC0GFetEfC1Fj4dDx0V8rHbSCbsWNbt8uSDugsdVFV5tH6i/JwH28W
    certif: hhNtLu04BBUD7LvK8wcN18gx2kt6YqrkhoKtOZ8ulUHa4o5JYUhtdRbb2U734SFb
    certif: 2p1iomO8WfSY7GLIUlcSCfXY1Oet7fF8eZWAx3fsmu6HjLTTgGYWAohGBBgRAgAG
    certif: BQI9Y2GXAAoJEFyEX45KJscXJtIAn2CBl6tfFAkzebXmua3Z5KLO1tXlAJ49jjYj
    certif: j12sLDENvMEjsp7AdiJi8w==
    certif: =5IS6
    certif: -----END PGP PUBLIC KEY BLOCK-----
    changed:  dbtest@ripe.net 20020101
    mnt-by:   TEST-MNT
    source:   DB-TEST

    password: ola

    mntner:      TEST-MNT
    descr:       test mantainer object
    admin-c:     TP1-DB-TEST
    tech-c:      TP1-DB-TEST
    upd-to:      dbtest@ripe.net
    auth:        CRYPT-PW ppnvBUPxXbrQU # ola
    auth:        PGPKEY-4A26C717
    mnt-by:      TEST-MNT
    referral-by: TEST-DBM-MNT
    changed:     dbtest@ripe.net 20020101
    source:      DB-TEST

## Appendix C
test File Sample

    # route authorisation tests - autnum
    # CASE 4: We try to create 20.13.0.0/16 route object.
    # Will fail because of mnt-lower
    %
    % requires initial objects to be loaded
    #
    # create route object
    #
    #FAILURE
    #
    # RESULT 4: The creation will fail with Hierarchical authorisation failed

    route:   20.13.0.0/16
    descr:   Test route object
    origin:  AS200
    mnt-by:  TEST-MNT
    changed: test@ripe.net 20020101
    source:  DB-TEST

    password: auth

$Id: test_scenarios.html,v 1.2 2003/07/04 08:22:47 engin Exp $