Summary
I've been working on this since the first post of this example; you can see that the code has been completely rewritten and (I think) greatly improved, both in terms of the generics and the performance tests themselves.

Advertisement

A performance test framework

This is why it takes me so long to write (or rewrite) a book. Examples like this, which I think are done, rear up and gobble lots of time. The good thing is that I end up understanding all the problems in much more depth. But it makes books -- especially one the size of Thinking in Java fourth edition -- take forever. The following was taken directly from the book:

To prevent code duplication and to provide consistency among
tests, Ive put the basic functionality of the test process into a framework.
The following code establishes a base class from which you create a list of anonymous inner classes, one for each different test. Each of these inner classes is
called as part of the testing process. This approach allows you to easily add
and remove new kinds of tests.

This is another example of the Template Method design
pattern. Although you follow the typical approach of overriding the template
method Test.test( ) for each particular test, in this case the core
code (that doesnt change) is in a separate Tester class.[1] The
type of container under test is the generic parameter C:

Each Test object stores the name of that test. When
you call the test() method, it must be given the container to be tested
along with a messenger or data transfer object that holds the various parameters for that particular test. The parameters include size,
indicating the number of elements in the container, and loops, which
control the number of iterations for that test. These parameters may or may not
be used in every test.

Each container will undergo a sequence of calls to test(),
each with a different TestParam, so TestParam also contains staticarray() methods that make it easy to create arrays of TestParam
objects. The first version of array() takes a variable argument list
containing alternating size and loops values, and the second
version takes the same kind of list except that the values are inside Stringsthis
way it can be used to parse command-line arguments:

To use the framework, you pass the container to be tested
along with a List of Test objects to a Tester.run() method
(these are overloaded generic convenience methods which reduce the amount of
typing necessary to use them). Tester.run() calls the appropriate
overloaded constructor, then calls timedTest(), which executes each test
in the list for that container. timedTest() repeats each test for each
of the TestParam objects in paramList. Because paramList
is initialized from the static defaultParams array, you can change the paramList
for all tests by reassigning defaultParams, or you can change the paramList
for one test by passing in a custom paramList for that test:

The stringField() and numberField() methods
produce formatting strings for outputting the results. The standard width for
formatting can be changed by modifying the static fieldWidth value. The displayHeader()
method formats and prints the header information for each test.

If you need to perform special initialization, override the initialize( )
method. This produces an initialized container object of the appropriate
sizeyou can either modify the existing container object or create a new one.
You can see in test( ) that the result is captured in a local
referencecalled kontainer, which allows you to replace the
stored member container with a completely different initialized
container.

The return value of each Test.test() method must be
the number of operations performed by that test, which is used to calculate the
number of nanoseconds required for each operation. You should be aware that System.nanoTime()
typically produces values with a granularity that is greater than one (and this
granularity will vary with machines and operating systems), and this will
produce a certain amount of rattle in the results.

The results may vary from machine to machine; these tests
are only intended to compare the performance of the different containers.

Here is a performance test for the most essential of the List operations. For comparison, it also shows the most important Queue operations. Two separate lists of tests are created for testing each
class of container. In this case, Queue operations only apply to LinkedLists.

Each test requires careful thought to ensure that you are
producing meaningful results. For example, the add test clears the List
and then refills it to the specified list size. The call to clear() is
thus part of the test, and may have an impact on the time, especially for small
tests. Although the results here seem fairly reasonable, you could imagine
rewriting the test framework so that there is a call to a preparation method
(which would, in this case, include the clear() call) outside of
the timing loop.

Note that for each test, you must accurately calculate the
number of operations that occur and return that value from test(), so
the timing is correct.

The get and set tests both use the random
number generator to perform random accesses to the List. In the output,
you can see that, for a List backed by an array and for an ArrayList,
these accesses are fast and very consistent regardless of the list size,
whereas for a LinkedList the access times grow very significantly for
larger lists. Clearly, linked lists are not a good choice if you will be
performing many random accesses.

The iteradd test uses an iterator in the middle of
the list to insert new elements. For an ArrayList this gets expensive as
the list gets bigger, but for a LinkedList it is relatively cheap, and
constant regardless of size. This makes sense because an ArrayList must
create space and copy all its references forward during an insertion. This
becomes expensive as the ArrayList gets bigger. A LinkedList only
needs to link in a new element, and doesnt have to modify the rest of the
list, so you expect the cost to be roughly the same regardless of the list
size.

The insert and remove tests both use
location number 5 as the point of insertion or removal, rather than either end
of the List. A LinkedList treats the end points of the List
speciallythis improves the speed when using a LinkedList as a Queue.
However, if you add or remove elements in the middle of the list, you include
the cost of random access, which weve already seen varies with the different List
implementations. By performing the insertions and removals at location five,
the cost of the random access should be negligible and we should see only the
cost of insertion and removal, but we will not see any specialized optimization
for the end of a LinkedList. You can see from the output that the cost
of insertion and removal in a LinkedList is quite cheap and doesnt vary
with the list size, but with an ArrayList, insertions especially are very
expensive, and the cost increases with list size.

From the Queue tests, you can see how quickly a LinkedList
can insert and remove elements from the endpoints of the list, which is optimal
for Queue behavior.

Normally, you can just call Tester.run(), passing the
container and the tests list. Here, however, we must override the initialize()
method so that the List is cleared and refilled before each
testotherwise the List control over the size of the List would
be lost during the various tests. ListTester inherits from Tester
and performs this initialization using CountingIntegerList. The run()
convenience method is also overridden.

Wed also like to compare array access to container access
(primarily against ArrayList). In the first test in main(), a
special Test object is created using an anonymous inner class. The initialize()
method is overridden to create a new object each time it is called (ignoring
the stored container object, so null is the container
argument for this Tester constructor). The new object is created using Generated.array( )
(which was defined in the Arrays chapter) and Arrays.asList().
Only two of the tests can be performed in this case, because you cannot insert
or remove elements when using a List backed by an array, so the List.subList()
method is used to select the desired tests from the tests list.

For random-access get( ) and set( )
operations, a List backed by an array is slightly faster than an ArrayList,
but the same operations are dramatically more expensive for a LinkedList
because it is not designed for random-access operations.

Vector should be avoided; its only in the library
for legacy code support (the only reason it works in this program is because it
was adapted to be a List for forward compatibility).

The best approach is probably to choose an ArrayList
as your default and to change to a LinkedList if you need its extra
functionality or you discover performance problems due to many insertions and
removals from the middle of the list. If you are working with a fixed-sized
group of elements, either use a List backed by an array (as produced by Arrays.asList( )),
or if necessary, an actual array.

CopyOnWriteArrayList is a special implementation of List
used in concurrent programming, and will be discussed in the Concurrency
chapter.

The performance of HashSet is generally superior to TreeSet,
but especially when adding elements and looking them up, which are the two most
important operations. TreeSet exists because it maintains its elements
in sorted order, so you use it only when you need a sorted Set. Because
of the internal structure necessary to support sorting and because
iteration is something youre more likely to do, iteration is usually faster
with a TreeSet than a HashSet.

Note that LinkedHashSet is more expensive for
insertions than HashSet; this is because of the extra cost of
maintaining the linked list along with the hashed container.

Insertions for all the Map implementations except for
IdentityHashMap get significantly slower as the size of the Map
gets large. In general, however, lookup is much cheaper than insertion, which
is good because youll typically be looking items up much more often than you
insert them.

Hashtable performance is roughly the same as HashMap.
Since HashMap is intended to replace Hashtable, and thus uses the
same underlying storage and lookup mechanism (which you will learn about later)
this is not too surprising.

A TreeMap is generally slower than a HashMap.
As with TreeSet, a TreeMap is a way to create an ordered list.
The behavior of a tree is such that its always in order and doesnt have to be
specially sorted. Once you fill a TreeMap, you can call keySet( ) to get a Set view of the keys, then toArray( ) to produce an array of
those keys. You can then use the static method Arrays.binarySearch( )
to rapidly find objects in your sorted array. Of course, this only makes sense
if the behavior of a HashMap is unacceptable, since HashMap is
designed to rapidly find keys. Also, you can easily create a HashMap from
a TreeMap with a single object creation or call to putAll(). In
the end, when youre using a Map,your first choice should be HashMap,
and only if you need a constantly sorted Map will you need TreeMap.

LinkedHashMap tends to be slower than HashMap
for insertions because it maintains the linked list (to preserve insertion
order) in addition to the hashed data structure. Because of this list,
iteration is faster.

IdentityHashMap has different performance because it
uses == rather than equals( ) for comparisons. WeakHashMap
is described later in this chapter.

[1] Krzysztof
Sobolewski assisted me in figuring out the generics for this example.

Talk Back!

Have an opinion?
Readers have already posted
11
comments
about this weblog entry. Why not
add yours?

RSS Feed

If you'd like to be notified whenever Bruce Eckel adds a new entry to his weblog, subscribe to his RSS feed.