#

# Conformance and Performance Testing

One of the important design goals of the W3C Reference Library is that it must conform to the current specifications of the Web model, for example for HTTP. This page gives an overview of what tests we have available and what we are working on. We have divided the tests the following four categories:

Basic core test suite
  In order to make sure that the basic parts of the Libray conforms to the current specifications, we have developed a set of basic tools.
Stress and Regresion tests
  It is important to test the Library code under stress - that is - in an environment with limited resources. This can for example be the number of available socket descriptors, lack of virtual memory, disk space etc. We have a set of example applications that can serve to do the stress testing:
  - Mini server and proxy
  - Robot
  - Command line tool
  - Line mode browser

  The test should also include testing of a well-known test base where we can perform regression tests on.

Performance testing
  Even though performance is not the most important factor in the W3C reference code, we need to be aware of the problem as demos only work in practice if they run reasonaly smooth. We have some first results available and more will come.
Platform dependent testing
  We should select a small set of main platforms for tesing platform dependencies. In all other cases we should rely on external help for doing the test.

Performance Testing

-------------------

The test run contained GET methods in first generation (181 requests) and HEAD

(121 requests) methods in second generation links from this page. The total

amount of requests issued was 302. The results are shown below with the robot

running on www23 (solaris 2.4)

Without proxy:

Robot started on Sat, 10 Feb 1996 15:48:46

Robot terminated Sat, 10 Feb 1996 15:49:27

total: 41 sec (7.3 request/sec)

With mini proxy running on www20 (solaris 2.3):

Robot started on Sat, 10 Feb 1996 16:13:54

Robot terminated Sat, 10 Feb 1996 16:14:36

total: 42 sec which gives a proxy overhead of 1 sec or 2.4%

With CERN proxy and a minimum configuration on www20:

Robot started on Sat, 10 Feb 1996 16:18:48

Robot terminated Sat, 10 Feb 1996 16:20:00

run 1: total: 72 sec which gives a proxy overhead of 31 sec or 75.6%

run 2: total: 71 sec which gives a proxy overhead of 30 sec or 73.1%

With mini proxy running on zorch (alpha):

Robot started on Sat, 10 Feb 1996 15:52:25

Robot terminated Sat, 10 Feb 1996 15:53:06

total: 41 sec which gives a proxy overhead of 0 sec ???

This of course doesn't show a lot as we need more statistics in order to take

the variance into account. However, the purpose was to get a grib on if it

made sense for me to continue working on integrating ILU into libwww for

making a testbed for HTTP-NG. I think this looks promising and I have done

some first attempts on the merging of ILU and libwww but don't have any

results yet.

Robot Profiling

---------------

It's is interesting to see that more than 66% of all cycles are used in the

SGML/HTML parser alone, and that doesn't include any presentation - only

parsing. The Event loop does not seem to be a problem even though it consumes

a lot of cycles on context swapping and the select call itself. Also, our new

dynamic memory handler does not provide significant overhead. There is some

information about the memory handler at

http://www.w3.org/devel/WWW/Library/User/Using/Memory.html

----------------------------------------------------------------------------

* -p[rocedures] using basic-block counts; *

* sorted in descending order by the number of cycles executed in each *

* procedure; unexecuted procedures are excluded *

----------------------------------------------------------------------------

95982832 cycles

cycles %cycles cum % cycles bytes procedure (file)

/call /line

21668317 22.58 22.58 57 28 SGML_character

(../../Library/Implementation/SGML.c)

16048691 16.72 39.30 26267 31 HTPlain_write

(../../Library/Implementation/HTPlain.c)

12562990 13.09 52.38 42 25 HTML_put_character

(../../Library/Implementation/HTML.c)

7138056 7.44 59.82 6 24 HText_appendCharacter

(../../Robot/Implementation/HTRobot.c)

6506182 6.78 66.60 18859 30 SGML_write

(../../Library/Implementation/SGML.c)

4317494 4.50 71.10 501 19 scan (../../Library/Implementation/

HTParse.c)

3848148 4.01 75.11 83 56 strcasecomp

(../../Library/Implementation/HTString.c)

2613979 2.72 77.83 307 33 HTAtom_for

(../../Library/Implementation/HTAtom.c)

2115564 2.20 80.03 38 35 HTChunk_putc

(../../Library/Implementation/HTChunk.c)

1478274 1.54 81.57 1478274 26 HTEvent_Loop

(../../Library/Implementation/HTEvntrg.c)

1401192 1.46 83.03 207 32 SGMLFindTag

(../../Library/Implementation/SGML.c)

1367270 1.42 84.46 1435 30 HTAnchor_findAddress

(../../Library/Implementation/HTAnchor.c)

1324508 1.38 85.84 1374 29 HTMIME_put_block

(../../Library/Implementation/HTMIME.c)

889459 0.93 86.77 43 29 HTSACopy

(../../Library/Implementation/HTString.c)

822996 0.86 87.62 20 24 HTMemory_free

(../../Library/Implementation/HTMemory.c)

782508 0.82 88.44 867 43 HTSimplify

(../../Library/Implementation/HTParse.c)

759660 0.79 89.23 4063 26 parseheader

(../../Library/Implementation/HTMIME.c)

677414 0.71 89.94 157 21 HTParse

(../../Library/Implementation/HTParse.c)

645633 0.67 90.61 23 30 HTMemory_malloc

(../../Library/Implementation/HTMemory.c)

629904 0.66 91.26 56 37 wild_match

(../../Library/Implementation/HTFormat.c)

Server Profiling

----------------

The server does not include the SGML/HTML parser at all and hence the

profiling test looks very different. Here the event loop takes a significant

amount of time, the reason beeing that the event loop is optimized for user

applications and not server applications. The solution is to provide multiple

event loops that are optimized for various purposes. Not surprising, a

considerable part of the time is spent in the MIME parser.

----------------------------------------------------------------------------

* -p[rocedures] using basic-block counts; *

* sorted in descending order by the number of cycles executed in each *

* procedure; unexecuted procedures are excluded *

----------------------------------------------------------------------------

7059122 cycles

cycles %cycles cum % cycles bytes procedure (file)

/call /line

1385301 19.62 19.62 1385301 26 HTEvent_Loop

(../../Library/Implementation/HTEvntrg.c)

734278 10.40 30.03 458 19 scan (../../Library/Implementation/

HTParse.c)

725385 10.28 40.30 4267 29 HTMIME_put_block

(../../Library/Implementation/HTMIME.c)

357407 5.06 45.37 303 33 HTAtom_for

(../../Library/Implementation/HTAtom.c)

290905 4.12 49.49 787 46 HTNextField

(../../Library/Implementation/HTWWWStr.c)

258050 3.66 53.14 255 31 __ProcessFds

(../../Library/Implementation/HTEvntrg.c)

188749 2.67 55.82 1869 26 parseheader

(../../Library/Implementation/HTMIME.c)

172651 2.45 58.26 19 24 HTMemory_free

(../../Library/Implementation/HTMemory.c)

166670 2.36 60.62 1651 30 HTAnchor_findAddress

(../../Library/Implementation/HTAnchor.c)

161866 2.29 62.92 111 56 strcasecomp

(../../Library/Implementation/HTString.c)

142504 2.02 64.93 44 29 HTSACopy

(../../Library/Implementation/HTString.c)

119604 1.69 66.63 704 26 HTTPReceive_put_block

(../../Library/Implementation/HTTPServ.c)

112702 1.60 68.22 141 21 HTParse

(../../Library/Implementation/HTParse.c)

111908 1.59 69.81 277 55 strncasecomp

(../../Library/Implementation/HTString.c)

103960 1.47 71.28 23 30 HTMemory_malloc

(../../Library/Implementation/HTMemory.c)

95455 1.35 72.64 95 24 HTSocketRead

(../../Library/Implementation/HTSocket.c)

89520 1.27 73.90 48 29 HTAlertCall_find

(../../Library/Implementation/HTAlert.c)

88258 1.25 75.15 874 43 HTSimplify

(../../Library/Implementation/HTParse.c)

86423 1.22 76.38 95 35 HTEvent_Register

(../../Library/Implementation/HTEvntrg.c)

85162 1.21 77.58 52 28 buf_put_block

(../../Library/Implementation/HTConLen.c)