Cubic regularization in symmetric rank-1 quasi-Newton methods

Abstract

Quasi-Newton methods based on the symmetric rank-one (SR1) update have been known to be fast and provide better approximations of the true Hessian than popular rank-two approaches, but these properties are guaranteed under certain conditions which frequently do not hold. Additionally, SR1 is plagued by the lack of guarantee of positive definiteness for the Hessian estimate. In this paper, we propose cubic regularization as a remedy to relax the conditions on the proofs of convergence for both speed and accuracy and to provide a positive definite approximation at each step. We show that the n-step convergence property for strictly convex quadratic programs is retained by the proposed approach. Extensive numerical results on unconstrained problems from the CUTEr test set are provided to demonstrate the computational efficiency and robustness of the approach.

Keywords

Mathematics Subject Classification

Notes

Acknowledgements

We would like to thank Daniel Bienstock and Andreas Waechter for their handling of the paper as Editor and Associate Editor, respectively, for MPC. We would also like to thank the two anonymous referees whose feedback and suggestions have greatly improved the paper.

Appendix

Iteration and runtime (in CPU seconds) results for the proposed modified SR1 approach and BFGS on unconstrained NLPs from the CUTEr test set

Problem

n

Modified SR1

BFGS

Iters

Time

\(f(x*)\)

Iters

Time

\(f(x*)\)

aircrftb

5

47

0.000

0.000

44

0.001

0.000

allinitu

4

12

0.000

5.744

12

0.000

5.744

arglina

100

1

0.001

100.000

1

0.004

100.000

arglinb

10

1

0.000

4.634

2

0.000

4.634

arglinc

8

1

0.000

6.135

2

0.000

6.135

arwhead

5000

(IL)

(LINE)

bard

3

15

0.000

0.008

25

0.000

0.008

bdexp

5000

17

5.241

0.001

17

5.257

0.001

bdqrtic

1000

88

0.755

3983.818

(LINE)

beale

2

15

0.000

0.000

16

0.000

0.000

biggs3

3

24

0.000

0.000

20

0.000

0.000

biggs5

5

112

0.001

0.006

76

0.001

0.006

biggs6

6

88

0.001

0.000

118

0.002

0.000

box2

2

10

0.000

0.000

5

0.000

0.000

box3

3

14

0.000

0.000

13

0.000

0.000

bratu1d

1001

(EVAL)

(LINE)

brkmcc

2

9

0.000

0.169

4

0.000

0.169

brownal

10

10

0.000

0.000

11

0.000

0.000

brownbs

2

13

0.000

0.000

10

0.000

0.000

brownden

4

23

0.001

85,822.202

37

0.001

85,822.202

broydn7d

1000

378

3.472

374.526

297

2.248

365.796

brybnd

5000

27

8.788

0.000

34

10.920

0.000

chainwoo

1000

187

1.596

1.000

469

3.348

4.573

chnrosnb

50

459

0.008

0.000

269

0.005

0.000

cliff

2

3

0.001

0.000

(EVAL)

clplatea

4970

5799

1772.120

\(-\)0.013

580

182.254

\(-\)0.013

clplateb

4970

1225

431.320

\(-\)6.988

447

138.601

\(-\)6.988

cosine

10,000

(IL)

14

18.929

\(-\)9999.000

cragglvy

5000

88

31.249

1688.215

(LINE)

cube

2

31

0.000

0.000

29

0.000

0.000

deconvu

51

2647

0.105

0.000

86

0.003

0.000

denschna

2

10

0.000

0.000

12

0.000

0.000

denschnb

2

9

0.000

0.000

9

0.000

0.000

denschnc

2

19

0.000

0.000

22

0.001

0.000

denschnd

3

69

0.000

0.000

92

0.001

0.000

denschne

3

26

0.000

0.000

(EVAL)

denschnf

2

11

0.000

0.000

12

0.000

0.000

dixmaana

3000

9

0.840

1.000

17

1.707

1.000

dixmaanb

3000

13

1.292

1.000

28

2.686

1.000

dixmaanc

3000

16

1.631

1.000

34

3.283

1.000

dixmaand

3000

20

2.110

1.000

45

4.328

1.000

dixmaane

3000

1242

116.429

1.000

1206

109.774

1.000

dixmaanf

3000

368

37.857

1.000

1024

94.188

1.000

dixmaang

3000

1703

150.896

1.000

1217

111.171

1.000

dixmaanh

3000

950

90.330

1.000

1090

100.547

1.000

dixmaani

3000

(IL)

19,146

1830.670

1.000

dixmaanj

3000

3585

318.487

1.000

14,720

1433.890

1.000

dixmaank

3000

15,465

1337.190

1.000

16,719

1609.120

1.000

dixmaanl

3000

2246

199.981

1.000

16,463

1532.540

1.000

dixon3dq

10

18

0.000

0.000

21

0.001

0.000

dqdrtic

5000

8

2.292

0.000

10

3.063

0.000

dqrtic

5000

131

48.048

0.160

427

127.586

0.116

edensch

2000

17

0.702

12,003.285

20

0.761

12,003.285

eg2

1000

6

0.036

\(-\)998.947

(LINE)

engval1

5000

25

8.078

5548.668

(LINE)

engval2

3

41

0.000

0.000

27

0.000

0.000

errinros

50

452

0.010

39.904

397

0.007

39.904

expfit

2

13

0.000

0.241

13

0.000

0.241

extrosnb

10

0

0.000

0.000

0

0.000

0.000

fletcbv2

100

807

0.042

\(-\)0.514

100

0.004

\(-\)0.514

fletchcr

100

749

0.037

0.000

173

0.009

0.000

flosp2hl

650

(IL)

(LINE)

flosp2hm

650

(IL)

(LINE)

flosp2th

650

(IL)

(LINE)

flosp2tm

650

(IL)

(LINE)

fminsrf2

1024

169,458

1232.500

1.000

222

1.753

1.000

fminsurf

1024

1075

8.433

1.000

203

1.590

1.000

freuroth

5000

18

6.300

608,159.189

(LINE)

genrose

500

6055

6.267

1.000

1705

1.236

1.000

growth

3

1

0.000

3542.149

(LINE)

growthls

3

1

0.000

3542.149

(LINE)

gulf

3

47

0.001

0.000

32

0.001

0.000

hairy

2

(EVAL)

54

0.001

20.000

hatfldd

3

26

0.000

0.000

18

0.000

0.000

hatflde

3

34

0.000

0.000

25

0.000

0.000

heart6ls

6

891

0.008

0.000

804

0.008

0.000

heart8ls

8

450

0.004

0.000

2542

0.026

0.000

helix

3

30

0.000

0.000

29

0.001

0.000

hilberta

10

15

0.000

0.000

52

0.001

0.000

hilbertb

50

7

0.000

0.000

7

0.001

0.000

himmelbb

2

11

0.000

0.000

10

0.000

0.000

himmelbf

4

34

0.000

318.572

29

0.001

318.572

himmelbg

2

8

0.000

0.000

9

0.000

0.000

himmelbh

2

8

0.000

\(-\)1.000

7

0.000

\(-\)1.000

humps

2

83

0.001

0.000

111

0.002

0.000

jensmp

2

3

0.002

2010.574

20

0.001

124.362

kowosb

4

41

0.001

0.000

32

0.001

0.000

liarwhd

10,000

18

28.060

0.000

20

27.335

0.000

mancino

100

18

0.101

0.000

17

0.106

0.000

maratosb

2

5

0.001

\(-\)1.000

(LINE)

methanb8

31

468

0.011

0.000

162

0.006

0.000

methanl8

31

2345

0.058

0.002

386

0.012

0.002

mexhat

2

10

0.000

\(-\)0.040

8

0.000

\(-\)0.040

minsurf

36

17

0.000

1.000

15

0.001

1.000

morebv

5000

0

0.043

0.000

0

0.169

0.000

msqrtals

1024

51,339

665.522

0.000

1732

17.272

0.000

msqrtbls

1024

23,372

312.256

0.000

1612

16.802

0.000

nasty

2

(EVAL)

3

0.000

0.000

ncb20

1010

136

1.881

1668.907

(LINE)

ncb20b

1000

77

3.885

1721.840

(LINE)

noncvxu2

1000

3861

29.786

2317.180

1426

9.725

2317.749

noncvxun

1000

25

0.201

2316.808

24

0.163

2316.808

nondia

9999

9

1.092

0.000

8

10.092

0.000

nondquar

10,000

422

697.313

0.000

1442

1913.620

0.000

nonmsqrt

9

412

0.002

0.752

(CONV)

osbornea

5

178

0.003

0.000

87

0.002

0.000

osborneb

11

102

0.006

0.040

75

0.004

0.040

palmer1c

8

11

0.000

0.098

114

0.001

0.098

palmer1d

7

48

0.000

0.653

(LINE)

palmer1e

8

(EVAL)

212

0.004

0.001

palmer2c

8

48

0.000

0.014

150

0.001

0.014

palmer2e

8

105

0.002

0.000

211

0.003

0.000

palmer3c

8

27

0.000

0.020

134

0.001

0.020

palmer3e

8

1419

0.013

0.000

240

0.004

0.000

palmer4c

8

23

0.000

0.050

20

0.000

0.050

palmer4e

8

45

0.001

0.000

148

0.002

0.000

palmer5c

6

11

0.000

2.128

15

0.001

2.128

palmer5d

4

7

0.000

87.339

31

0.001

87.339

palmer6c

8

29

0.000

0.016

87

0.001

0.016

palmer7c

8

68

0.000

0.602

93

0.001

0.602

palmer8c

8

33

0.000

0.160

24

0.000

0.160

penalty1

1000

139

1.253

0.010

82

0.642

0.010

penalty2

100

171

0.022

97,096.084

(LINE)

penalty3

100

(EVAL)

(LINE)

pfit1

3

(EVAL)

333

0.004

0.000

pfit1ls

3

(EVAL)

333

0.003

0.000

pfit2

3

(EVAL)

524

0.004

0.000

pfit2ls

3

(EVAL)

524

0.004

0.000

pfit4

3

(EVAL)

955

0.008

0.000

pfit4ls

3

(EVAL)

955

0.009

0.000

powellsg

4

37

0.000

0.000

58

0.001

0.000

power

1000

5780

47.952

0.000

2342

16.570

0.000

rosenbr

2

44

0.000

0.000

37

0.000

0.000

s201

2

5

0.000

0.000

7

0.000

0.000

s202

2

12

0.000

48.984

18

0.000

48.984

s204

2

7

0.000

0.184

9

0.000

0.184

s205

2

13

0.000

0.000

12

0.000

0.000

s206

2

9

0.000

0.000

8

0.000

0.000

s207

2

13

0.000

0.000

9

0.000

0.000

s208

2

44

0.000

0.000

37

0.000

0.000

s209

2

160

0.001

0.000

123

0.001

0.000

s210

2

257

0.001

0.000

522

0.004

0.000

s211

2

31

0.000

0.000

29

0.001

0.000

s212

2

12

0.000

0.000

14

0.000

0.000

s213

2

49

0.000

0.000

32

0.000

0.000

s240

3

5

0.000

0.000

3

0.000

0.000

s243

3

10

0.000

0.797

13

0.001

0.797

s245

3

14

0.000

0.000

33

0.000

0.000

s246

3

30

0.000

0.000

21

0.000

0.000

s256

4

37

0.000

0.000

58

0.001

0.000

s258

4

26

0.000

0.000

37

0.000

0.000

s260

4

26

0.000

0.000

37

0.001

0.000

s261

4

43

0.000

0.000

48

0.001

0.000

s266

5

14

0.000

1.000

15

0.001

1.000

s267

5

38

0.001

0.000

54

0.001

0.000

s271

6

8

0.000

0.000

8

0.000

0.000

s272

6

48

0.000

0.006

44

0.001

0.006

s272a

6

448

0.006

0.000

46

0.001

0.034

s273

6

22

0.000

0.000

59

0.001

0.000

s274

2

4

0.000

0.000

4

0.000

0.000

s275

4

6

0.000

0.000

7

0.000

0.000

s276

6

6

0.000

0.000

7

0.000

0.000

s281a

10

16

0.000

0.000

43

0.000

0.000

s282

10

362

0.002

0.000

123

0.001

0.000

s283

10

125

0.002

0.000

319

0.003

0.000

s286

20

51

0.000

0.000

35

0.001

0.000

s287

20

26

0.000

0.000

38

0.001

0.000

s288

20

47

0.000

0.000

58

0.001

0.000

s289

30

5

0.000

0.000

3

0.000

0.000

s290

2

5

0.000

0.000

7

0.000

0.000

s291

10

15

0.000

0.000

19

0.000

0.000

s292

30

32

0.000

0.000

39

0.001

0.000

s293

50

44

0.001

0.000

53

0.001

0.000

s294

6

76

0.000

0.000

60

0.001

0.000

s295

10

194

0.001

0.000

87

0.001

0.000

s296

16

231

0.002

0.000

123

0.001

0.000

s297

30

431

0.005

0.000

224

0.002

0.000

s298

50

799

0.016

0.000

352

0.006

0.000

s299

100

1179

0.066

0.000

681

0.032

0.000

s300

20

27

0.000

\(-\)20.000

32

0.000

\(-\)20.000

s301

50

56

0.001

\(-\)50.000

69

0.002

\(-\)50.000

s302

100

106

0.005

\(-\)100.000

131

0.007

\(-\)100.000

s303

20

11

0.000

0.000

14

0.000

0.000

s304

50

20

0.001

0.000

25

0.001

0.000

s305

100

28

0.001

0.000

24

0.001

0.000

s308

2

15

0.000

0.773

15

0.000

0.773

s309

2

22

0.000

\(-\)3.987

12

0.000

\(-\)3.987

s311

2

8

0.000

0.000

7

0.000

0.000

s312

2

40

0.000

5.923

33

0.001

5.923

s314

2

9

0.000

0.169

4

0.000

0.169

s333

3

39

0.001

0.043

(LINE)

s334

3

15

0.000

0.008

25

0.000

0.008

s350

4

41

0.000

0.000

32

0.001

0.000

s351

4

37

0.000

318.572

44

0.001

318.572

s352

4

7

0.000

903.234

20

0.001

903.234

s370

6

74

0.001

0.002

53

0.001

0.002

s371

9

226

0.002

0.000

124

0.002

0.000

s379

11

101

0.006

0.040

75

0.004

0.040

s386

2

5

0.000

0.000

7

0.000

0.000

sineval

2

(IL)

66

0.001

0.000

sinquad

10,000

226

389.181

0.000

231

347.267

0.000

sisser

2

12

0.000

0.000

11

0.000

0.000

snail

2

170

0.001

0.000

92

0.001

0.000

srosenbr

10,000

41

65.329

0.000

37

53.320

0.000

testquad

1000

283

2.409

0.000

3291

25.220

0.000

tointgss

10,000

3

2.072

10.001

4

4.810

10.001

tquartic

10,000

19

30.433

0.000

24

33.719

0.000

vardim

100

37

0.003

0.000

36

0.002

0.000

vibrbeam

8

78

0.012

8.481

(LINE)

watson

31

285

0.008

0.000

163

0.003

0.000

woods

10,000

26

39.082

0.000

40

57.429

0.000

yfitu

3

113

0.001

0.000

65

0.001

0.000

zangwil2

2

2

0.000

\(-\)18.200

2

0.000

\(-\)18.200

(CONV) indicates that BFGS stopped making progress before reaching the required level of convergence, (EVAL) indicates that the code encountered a function evaluation error, (IL) indicates that the code reached its iteration limit, and (LINE) indicates that the line search in BFGS failed