CGI and WWW Server Measurement

Introduction

Lots of people have suggestions for improving World Wide Web performance,
but very few do a careful analysis of what aspects of the system should be
measured and improved.

It is our belief that latency (as observed by Web clients) is the most
important factor in user perceived performance. Current benchmarks
tend to emphasize throughput (Mbit/s, connections/s, etc) and as a result
they are poor predictors of a user's experience. Users are not interested
in the number of connections that can be processed every second. They only
care about the time between when they click on a link and the arrival of the
document on their screen.

In addition no benchmark
we have seen takes into account documents generated through the
Common Gateway Interface (CGI). Some types of scripts
significantly affect the load on the server, and as a result they add latency
to all requests.

We have developed a benchmark, WebStone-SS, that simulates
realistic workloads in a reproducible way. We analysed logs taken
from real web servers to produce a configuration file that drives the
benchmark.

With this tool we were able to measure both the effect of realistic workloads
on CGI performance, as well as the effect of CGI scripts.

WebStone-SS

We decided that to conduct a reasonable analysis of CGI traffic and its
effect on server performance, we first needed a benchmark that could model
real WWW traffic. Statistical analysis of server logs indicated that a
site's behavior can be modeled by a set of simple configuration variables.

Access pattern heuristics

To generate the configuration files that drive the benchmark, we extract
higher level usage patters from WWW logs using simple heuristics. Users
are identified by their IP address. Any set of files that the user downloads
within 15 seconds of each other, is considered to be a single page. A group
of pages downloaded with less than 5 minutes between each download is
considered to be a session.

Reduction Heuristics

To reduce the size of the data set, we used three more heuristics to add
a level of abstraction.

Combine if

Factor of Reduction

Same file sets

2.1

Same number of files

75% same file sets

Each of the remaining 25% of files is about the same size

4.7

Same front (.html) page

2.5

CGI Analysis

Languages and Performance

The advanced string handling capabilities built into the Perl programming
language are very well suited to CGI programming. As a result, most CGI
scripts are written in Perl.

Perl is an interpreted language, which for many operations runs much slower
than compiled languages. In addition the Perl interpreter is a very large
program which must be launched every time a Perl script runs. Obviously
Perl is a poor choice for large, compute-intensive applications. We set out
to answer the following two questions:

Does the start up time for the large interpreter rule out using Perl
for latency sensitive Web applications?

Does the heavy interperter increase load on the server that will add
latency to all requests?

Start-up Cost

To answer the first question we created two CGI programs, one in Perl and one
in C, that return a one byte result. We used (unmodified) WebStone 2.0 to
pound the server with requests for each of these scripts. Although on
average the Perl version had twice the latency of the C version, the noise
present in this measurement was quite a bit larger than the difference between
the two runs.

We conclude that the start-up cost of the Perl interpreter does not add
significant latency to a CGI request.

Interpreter effect on Server Load

We used the WebStone-SS benchmark to measure the effect of running short Perl
scripts on a realistically loaded server. We made two versions of the
software site configuration, one that called the trivial Perl script for
every CGI request, and another that loaded a static document instead. The
execution of the Perl interpreter produced no observable effect on measured
latency of overall traffic on the site.

Effect of Server Load on CGI Performance

We used our WebStone 2.0 micro-benchmark test to compare the behavior of
a variety of scripts under differing server loads. Most of the latencies
went up linearly with the number of clients up to a certain point after
which the server failed to service requests properly.

We see that the counter and the one byte Perl script were equivalent in
terms of latency. Both failed when WebStone reached 30 clients. At this
point the error rates jumped up to about 80%. The server's error log reported
that at this point it was no longer able to fork off new processes.
The Perl script returning 300kb was slower, but also failed at 30 clients.
The Glimpse test however failed at only 14 clients. Since the Glimpse
script takes so long to execute, the WebStone clients timed out while
reading the results.

Effect of CGI on Server Load

To measure the effect of CGI on server load, we made another version of
the configuration that called a counter script.
We compared the results of this test against the static file configuration.
Running the benchmark with these two
configurations we found that the counter had little impact on the system.
In fact the reported latencies were slightly lower when the CGI was running
than when it was not.

Since
some CGI scripts are much more intensive
than counters, we also ran
the same trace with the GlimpseHTTP search engine on all CGI requests.
Each time the engine searched the NetBSD source tree for "mbufs". In this
case more than half of all requests (including non-CGI) took two to three
seconds longer to complete.

We believe this slowdown is caused by the search engine's intense use of
both the CPU and the disk. The index files the engine uses are over four
Mb for the NetBSD source tree. Scanning these huge files will knock Web
documents out of the buffer cache. Normal request latency now depends on
the speed of the disk.

This result shows two things: CGI should not be ignored in
analyses of Web traffic; Web developers should be careful to keep the
resource usage of their scripts to a reasonable level.

Conclusions

Benchmarks which do not attempt to model realistic Web traffic are
not a good measure of performance.

CGI behavior must be a part of the model.

Choice of language is not a concern for short scripts.

Counters and redirection scripts do not, in general, have to be
carefully optimized, but more intensive applications like search
engines have a significant effect on overall server performance.