Important Note:
The content in this site is accessible to any browser or Internet device, however, some graphics will display correctly only in the newer versions of Netscape. To get the most out of our site we suggest you upgrade to the latest Netscape. More information

Performance Assessment

Performance Assessment

According to the Principles of PISA, with PISA
PA, one can reliably compare different algorithms on benchmark problems using
appropriate indicators and statistical methods. To this end, the communication
between the Variator
and Selector module is controlled and observed by a Monitor
module. The tool flow is depicted in the following figure:

The performance assessment tool suite

performs multiple runs automatically,

runs different optimizers on different test problems in one pass,

calculates the right performance indicators to compare the algorithms,

The following package contains the complete binaries for Solaris,
Linux and Windows, the source code and some documentation of the individual
tools:
Performance Assessment Package (tar.gz) or
zip (PLEASE NOTE: The R-indicator implementation contained an error in a previous version.).

Tutorial on Performance Assessment

This tutorial reviews the current state of the art in statistical
performance assessment of stochastic multiobjective optimizers, derives general
recommendations, and demonstrates the implementation of the suggested guidelines
on the basis of a case study (pdf):

The tutorial was given at the Third International Conference on Evolutionary
Multi-Criterion Optimization EMO 2005 in Guanajuato, Mexico
(pdf).

Monitor

The part of the performance assessment package which is responsible
for starting multiple runs and for recording the non-dominated front
through the optimization run is called the monitor. In particular, the purposes of the monitor tool are the following,
see also Principles of PISA:

to observe the communication between a Variator
and Selector program in order to store information
about the generated populations in text files.

to automatically invoke several optimization runs
as an input to a subsequent statistical analysis, and

to control the Variator and Selector programs.

Therefore, the monitor is useful also independently of the
performance assessment in order to automatically invoke and report optimization
runs.