This is an excerpt from SICS Protocol Implementation Measurement
System:User Manual, covering chapters 1 and 2, the Introduction and
How to run SPIMS. There is also an excerpt on the SPIMS installation.

1. Introduction 
	1.1 This document 
	1.2 Note to the reader 
	1.3 Supported protocols 
	1.4 Limitations 
2. How to run SPIMS 
	2.1 The simple way
	2.2 Becoming more advanced
	2.3 Using your own measurement specifications

1. Introduction

SPIMS is a tool for measuring the performance of different protocol
implementations. This is done by providing a portable tool which uses
a protocol independent specification language so that the same
measurement specification can be used for measurement on different
machines and operating systems.

SPIMS uses the standard measurement facilities supplied by the
operating system i.e. it does not rely on special high-resolution
clocks, or other special support by the operating system. So SPIMS is
just a (distributed) application which executes measurement
specifications and presents the measured performance to the user.


1.1 This document

This document describes how to use the UNIX prototype
implementation of SPIMS, although parts of the description like the
specification language will be applicable to other SPIMS'
implementations.

The document describes how to do simple throughput measurements as
well as all the features of the specification language. It also
contains descriptions on how to make extensions to SPIMS such as
adding interfaces towards new protocols, and measuring other parts of
the protocols.

A short installation guide  for the SPIMS software is found in an Appendix A.

The topic of porting SPIMS to other systems is discussed in the "SPIMS
Portability Guide".


1.2 Note to the reader

This manual is structured starting with the simplest ways of using
SPIMS going towards more advanced use of the capabilities and features
of SPIMS. The levels to be presented are:

*	A simple test program (See the next Section)
*	Using supplied communication benchmarks
*	Writing your own measurement specifications in the SPIMS specification
	 language.
*	Extending SPIMS to measure e.g. other services of the protocols or 
	other protocols.

The extensions to SPIMS are made by writing small amounts of C code
i.e. doing them requires some proficiency it the C language. Note that
all the extensions are possible even if you only have a binary
distribution of SPIMS.

1.3 Supported protocols
 
SPIMS version 2.0 is packaged with interfaces to measure the following
protocols: (Footnote 2)

*	Arpa UDP, TCP, and FTP.
*	The ISODE 4.0 OSI protocols TP,  ROSE, and FTAM.
*	Sun MicroSystem Inc.: SunRPC and SunLink 5.2 TP
*	VMTP (available at Stanford University).
Default are the ARPA protocols.


1.4 Limitations

SPIMS version 2.0 requires that a TCP connection can be established
between the machines participating in measurements.


2. How to run SPIMS

2.1 The simple way

First, in order to use SPIMS easily set the environment variable
$SPIMSDIR in the shell, and then add $SPIMSDIR/bin to your command
search path.

Example when using the C-shell:
setenv SPIMSDIR /usr/local/spims if SPIMS is installed in /usr/local/spims
set path=($path $SPIMSDIR/bin)
rehash

edit your .cshrc file so that $SPIMSDIR and your path gets set like
above. Note: .cshrc not .login, since rsh does not run .login.3

Now, run the test program spims-test by typing spims-test followed by
the names of the machines to measure between and the protocols to
measure on.

Example: spims-test khons horus tcp ftp measures some
performance metrics between the machines named khons and horus. This
is done on the TCP and FTP protocols. The output will be presented as:

Measuring between khons and horus: tcp protocol...
Throughput: 215.43 kbytes/s
Two-way delay: 7.7 ms
Connect plus disconnect: 17.0 ms
Measuring between khons and horus: ftp protocol...
Throughput: 209.55 kbytes/s
Connect plus disconnect: 16.0 ms

Note: The measurements done with spims-test are relatively inaccurate
- they serve merely as a fast indication on the performance, and for
demonstration purposes.


2.2 Becoming more advanced

Exactly the same as the above example can be done by running
benchmark demo khons horus tcp ftp 

In addition to the demo measurement a few others are available:

basic 	
	same as demo but it runs longer and produces results with
higher accuracy.

bulksize

	measures the throughput using different sizes on the data
units being given to the protocol for transmission. The sizes range
from 1 kilobyte to 16 kilobytes.

rrsize	

measures the two-way delay for different size messages ranging from 1
byte to several kilobytes.


The internals: 

The spims-test and benchmark programs are UNIX shell-scripts. They use
a measurement specification and a filter program which extracts and
presents the results.

Example: benchmark basic reads a textual measurement specification
from the file $SPIMSDIR/benchmarks/basic and passes it to the program
doing the actual measurements. The output from the measurement is sent
to the (shell-script) program $SPIMSDIR/filters/basic.


2.3 Using your own measurement specifications

In Chapter 4 you will learn how to write your own measurement
specifications, however there exists some measurement specification
that you can use. When using them you have to:

*	Make sure that there are measurement demons executing on all
the machines you wish to measure between. The startdemons shell script
helps you in starting the demon processes. Example startdemons khons
horus tcp ftp is executed automatically in the spims-test example
above. See Section 6.2.

*	Execute the measurement program for the protocol of interest
and supply the specification to it.

*	Interpret the output of the measurement.

We once again look at a throughput measurement on TCP between the two
machines khons and horus as an example.

The specification for this measurement is:
10 bulk_get 1024*1000;

Execute it by typing (e.g. on the khons machine):
startdemons tcp khons horus	

		make sure the demons are running
tcp/bench khons horus
10 bulk_get 1024*1000;
^D

The output will look like:
Initiator 0:
Executing benchmark:
Specification: runs 10
   Basic bulk_get from khons to khons, tcp protocol
      Data: type bytes
         Size spec: iterations 1000
            Message size: number of values 1
               Constant value 1024
            Message size: number of values 1
               Constant value 1024
         Source:MEMORY
         Destination:MEMORY
      No timing specification
Benchmark using the tcp protocol
Started on Wed Sep  7 15:50:40 1988
Ended on Wed Sep  7 15:51:49 1988
Results for 10 runs:
Field	Average	Min	Max	Std Dev
elapsed  	5996.0000   	4920.0000  	9400.0000	1307.1700
cputime 	3648.6000   	3540.0000   	3741.0000 	76.3242
stime    	3554.5000   	3400.0000   	3701.0000 	92.6058
utime      	94.1000 	40.0000    	140.0000 	39.9067
datarcvd	1024000.0000
msgsent     	1.0000
msgrcvd  	1006.8000   	1001.0000   	1013.0000  	3.3928
ivcsw     	168.3000    	142.0000    	216.0000 	24.9847
vcsw      	129.6000 	98.0000    	197.0000 	29.0218
maxrss     	33.9000 	33.0000 	34.0000  	0.3162


The first part of the output is a pretty-printing of the measurement
specification (which can be disabled by giving the -e switch to the
bench program - see Appendix D for more on switches). The second part
contains the actual results. We note that the first two rows are the
elapsed time and CPU time in milliseconds, and leave the more detailed
description to Section 4.4. However, from the results we see that
transferring <datarcvd> bytes took <elapsed> milliseconds i.e. the
throughput is 1024000 bytes / 5.996 seconds = 171 kbyte/s.
(Measurements collected at the Responder can also be obtained by
giving the -r switch to the bench program - see Appendix D.)

_______________Footnotes
1UNIX is a registered trade mark for AT&T Bell Labs.
2Check with your system administrator for local additions and/or deletions to this list.
3Some of the utilities use rsh (remote shell) to start programs on other machines (See Section 6).

