Obsolete benchmark (1998/2006)

ADMB vs. Gauss vs. Matlab vs. R/S-Plus

Summary written by Arni Magnusson, 16 June 2010

Schnute et al. (1998) published a benchmark comparing the performance of different optimization software. They tested ADMB, Gauss, Matlab, and S-Plus with a catch-at-age model by Schnute and Richards (1995), using datasets simulated with S-Plus.

The simulated datasets included T years of data and A age classes, resulting in m=T+A+5 estimated parameters. The computer had a 133 MHz processor and 48 MB RAM, running Windows NT 4.0.

Benchmark trials for two cases of a catch-at-age model, in which T years of data are available for A age classes and the total number of parameters is m=T+A+5.

Case

T

A

m

Product

ms/call

Calls

Time

Scale

1

20

12

37

ADMB

29

161

4.6 s

1.0

GAUSS

42

4,041

2.8 min

1.0

MATLAB

178

1,936

5.8 min

1.0

S-PLUS

1429

n/a

n/a

2

80

15

100

ADMB

131

291

38 s

8.3

GAUSS

167

23,365

1.08 hr

23.1

MATLAB

639

18,360

3.25 hr

33.6

S-PLUS

n/a

n/a

n/a

S-Plus exhausted the computer memory before finishing the
calculations, and in smaller examples where evaluations were obtained,
S-Plus was much slower than the other products.

ADMB performed much faster and required fewer function calls than its
competitors in this benchmark. The reduced number of function calls
stems directly from automatic differentiation. One call in ADMB
evaluates both the function and its gradient. The other products
approximate gradients by making a small change to each estimated
parameter, requiring m+1 function calls to achieve the same result.
Furthermore, ADMB obtains gradients accurate to machine precision,
instead of approximations.

Case 2 scales the problem by a factor of 100/37=2.7, but the
computation time increases by a greater factor. ADMB appears more
scalable than the other products, probably from the efficiencies of
reverse automatic differentiation.

---

Schnute, J.T. and L.J. Richards. 1995. The influence of error on
population estimates from catch-age models. Canadian Journal of
Fisheries and Aquatic Sciences 52:2063-2077.

In Nov 2006, Dave Fournier ported the S-Plus model to R and benchmarked it against ADMB on newer hardware. With T=80 years and A=15 ages, the updated benchmark corresponds to Case 2 in the original benchmark:

Product

ms/call

Calls

Time

Function value

ADMB

10.79

278

3 s

3718.693119

R

33.17

56,074

31 min

3717.469

R

33.81

165,000

93 min

3718.693119

In the first run, R was manually interrupted after 31 minutes, to check the function value and parameter estimates. The function value was still far from the optimal value, and one parameter, gamma, was at 0.61, far from the optimal value of 0.42.

In the other run, R was allowed to converge. This took 93 minutes, more than 1000 times longer than ADMB.