Wiki: All Combinations

# All Combinations

This is the second of a suite of fixtures that make up the AllPairsExample. Like with AllFiles, we expand wildcard characters in path specifications to get lists of tests. But this time we run all combinations of the tests. This is another warmup for AllPairs testing.

A project can expect to have dozens or hundreds of files testing specific features in a stand-alone way. We may wonder if there are any interactions between features that will cause any of these tests to fail. AllCombinations explores this. It is a fixture that enumerates cases by drawing one file at a time from each of a number of lists; running each in turn, and then repeating the drawing and running for all possible combinations.

For the purpose of illustration we've created a number of files that contain small tests of the CalculatorExample. We've categorized the files as magnitudes, signs and functions, which enter values, changes signs, and compute functions, respectivly.

- http:Release/Documents/AllPairs/

This table runs all possible sequences of magnitude, sign and function. Try it with http:run.cgi.

eg.AllCombinations
input/AllPairs/magnitude/*.html |
input/AllPairs/sign/*.html |
input/AllPairs/function/*.html |
1
180+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
change-sign.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
2
180+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
change-sign.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
3
180+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
multiply.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
4
180+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
multiply.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
5
180+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
no-change.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
6
180+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
no-change.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
7
30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
change-sign.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
8
30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
change-sign.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
9
30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
multiply.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
10
30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
multiply.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
11
30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
no-change.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
12
30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
no-change.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
13
360+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
change-sign.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
14
360+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
change-sign.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
15
360+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
multiply.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
16
360+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
multiply.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
17
360+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
no-change.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
18
360+30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
no-change.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
19
90-30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
change-sign.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 0 right, 1 wrong, 0 ignores, 0 exceptions[!]
20
90-30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
change-sign.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 0 right, 1 wrong, 0 ignores, 0 exceptions[!]
21
90-30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
multiply.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 0 right, 1 wrong, 0 ignores, 0 exceptions[!]
22
90-30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
multiply.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 0 right, 1 wrong, 0 ignores, 0 exceptions[!]
23
90-30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
no-change.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
cosine.html | 0 right, 1 wrong, 0 ignores, 0 exceptions[!]
24
90-30.html | 1 right, 0 wrong, 0 ignores, 0 exceptions
no-change.html | 0 right, 0 wrong, 0 ignores, 0 exceptions
sine.html | 0 right, 1 wrong, 0 ignores, 0 exceptions[!]

The number of cases will as large as the product of the size of each list of tests. The searches in the table find four magnitues, three signs and two functions.

- 4 magnitudes X 3 signs X 2 functions = 24 cases

Any file of any case can fail. We save the results from failed tests only. A single file can fail different ways in different cases. Look for differences in the counts or examine each FileFootnote.

If the tests are intended to interact, as ours are here, then some care is required to be sure that checks in the tests are independent of this interaction. No checks at all my be suitable if we are only testing that there are no exceptions thrown. Or the simplest checks may be adaquate so long as features are tested well in isolation. It may be convenient to run a simulation, a so called oracale, along side the application to generate expected results.

Implementation

The fixture inherits path expansion from AllFiles, but captures these by overrinding doRows(Parse, List) and saving the list of files in a list of lists. Then, at the end of doTable, the fixture gets to work computing and running combinations.

See source.

- http:Release/Source/eg/AllCombinations.java

We had a little trouble setting up the test data for this example so we wrote this little Trigonometry Refresher.

eg.ArithmeticColumnFixture
x | sin() | cos()
0 | 0.0000 | 1.0000
30 | 0.5000 | 0.8660
60 | 0.8660 | 0.5000
90 | 1.0000 | 0.0000
120 | 0.8660 | -0.5000
150 | 0.5000 | -.8660
180 | 0.0000 | -1.0000

fit.Summary
counts | 80 right, 6 wrong, 0 ignores, 0 exceptions
input file | input/AllCombinations.html
input update | Sun Apr 30 03:58:36 2006
output file | output/AllCombinations.html
run date | Wed May 3 18:09:51 2006
run elapsed time | 0 wallclock secs ( 0.02 usr + 0.00 sys = 0.02 CPU)

Last edited November 4, 2002