Apply the PSU in Standby with RAC database using the standby-first patch apply method.

In this article, I introduce how to apply the patch using the standby-first patch apply the method in a DataGuard environment with Oracle RAC 12c, where we apply with Rolling Patch.In short, we will first apply the patch to standby and because the database is a 2-node RAC, we apply it to one node first and then the other.

In the scenario that I am, there is a patch 21923026 incompatibility with the patch 24448103 that I will apply, so it is necessary to roll back the patch 21923026 before applying the other.

Check the running instances.

1

2

3

4

[Stb1.oracle@srv-stb1 OPatch]$ps-ef|grep pmon

Oracle19819102016?asm_pmon_+ASM1

Oracle36065102016?ora_pmon_stb1

Oracle417402198018:31pts/100:00:00grep pmon

Connect to the Broker and check the current Standby configuration.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

[Stb1.oracle@srv-stb1 u01]$dgmgrl/

DGMGRL forLinux:Version12.1.0.2.0-64bitProduction

Copyright(c)2000,2013,Oracle.All rights reserved.

Welcome toDGMGRL,type"help"forinformation.

Connected asSYSDG.

DGMGRL>show configuration;

Configuration-dg

Protection Mode:MaxAvailability

Members:

Pmy-Primary database

Stb-Physical standby database

Fast-Start Failover:DISABLED

Configuration Status:

SUCCESS(status updated58seconds ago)

DGMGRL>show database stb;

Database-stb

Role:PHYSICAL STANDBY

Intended State:APPLY-ON

Transport Lag:0seconds(computed0seconds ago)

Apply Lag:0seconds(computed0seconds ago)

Average Apply Rate:626.00KByte/s

Real Time Query:OFF

Instance(s):

Stb1

Stb2(apply instance)

Database Status:

SUCCESS

Thus it was verified that redoapply is occurring in node 2, which allows us to apply the patch in node 1.Stop all Oracle Home services where the patch will be applied.In my case, there is only one instance in this OH.

1

[Stb1.oracle@srv-stb1 OPatch]$srvctl stop instance-istb1-dstb

Do the patch rollback 21923026. The -verbose option brings more logs to the screen.

With the root user set the environment variables for Grid Infrastructure Home.

1

[Stb1.root@srv-stb124448103]$./home/oracle/.profile_infra

Run the opatchauto analyze to check the prerequisites.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

[+ASM1.root@srv-stb1~]$$ORACLE_HOME/OPatch/opatchauto

Apply/u01/patch/24448103-analyze

System initialization log file is/u01/app/12.1.0.2/grid/

Cfgtoollogs/opatchautodb/systemconfig2017-01-11_07-17-30PM.log.

Session log file is/u01/app/12.1.0.2/grid/cfgtoollogs/

Opatchauto/opatchauto2017-01-11_07-17-54PM.log

The id forthissession isG5TU

[Init:init]Executing OPatchAutoBinaryAction action on home

/u01/app/oracle/product/12.1.0.2/dbhome_1

[Init:init]Executing OPatchAutoBinaryAction action on home

/u01/app/12.1.0.2/grid

Executing OPatch prereq operations toverify patch applicability

On RAC Home........

Executing OPatch prereq operations toverify patch applicability

On CRS Home........

[Init:init]OPatchAutoBinaryAction action completed on home/

U01/app/12.1.0.2/grid successfully

[Init:init]OPatchAutoBinaryAction action completed on home/

U01/app/oracle/product/12.1.0.2/dbhome_1 successfully

[Init:init]Executing GIRACPrereqAction action on home/u01/

App/oracle/product/12.1.0.2/dbhome_1

[Init:init]Executing GIRACPrereqAction action on home/u01/

App/12.1.0.2/grid

Executing prereq operations before applying on CRS Home........

Executing prereq operations before applying on RAC Home........

[Init:init]GIRACPrereqAction action completed on home/u01/app/

Oracle/product/12.1.0.2/dbhome_1 successfully

[Init:init]GIRACPrereqAction action completed on home/u01/app/

12.1.0.2/grid successfully

[Init:init]Executing RACDBPrereqAction action on home/u01/app/

Oracle/product/12.1.0.2/dbhome_1

Executing prereq operations before applying on RAC Home........

[Init:init]RACDBPrereqAction action completed on home/u01/app/

Oracle/product/12.1.0.2/dbhome_1 successfully

-------------------------Summary-------------------------

Analysis forapplying patches has completed successfully:

Host:srv-stb1

CRS Home:/u01/app/12.1.0.2/grid

==Following patches were SKIPPED:

Patch:/u01/patch/24448103/21436941

Reason:Thispatch isalready applied,so notgoing

Apply again.

==Following patches were SUCCESSFULLY analyzed tobe applied:

Patch:/u01/patch/24448103/24007012

Log:/u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/

Opatch2017-01-11_19-18-00PM_1.log

Patch:/u01/patch/24448103/24340679

Log:/u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/

Opatch2017-01-11_19-18-00PM_1.log

Patch:/u01/patch/24448103/24846605

Log:/u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/

Opatch2017-01-11_19-18-00PM_1.log

Host:srv-stb1

RAC Home:/u01/app/oracle/product/12.1.0.2/dbhome_1

==Following patches were SKIPPED:

Patch:/u01/patch/24448103/21436941

Reason:Thispatch isnotapplicable tothisspecified target

Type-"rac_database"

Patch:/u01/patch/24448103/24007012

Reason:Thispatch isnotapplicable tothisspecified target

Type-"rac_database"

==Following patches were SUCCESSFULLY analyzed tobe applied:

Patch:/u01/patch/24448103/24340679

Log:/u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/

Opatchauto/core/opatch/opatch2017-01-11_19-18-01PM_1.log

Patch:/u01/patch/24448103/24846605

Log:/u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/

Opatchauto/core/opatch/opatch2017-01-11_19-18-01PM_1.log

OPatchAutosuccessful.

Passing all the prerequisites,follow the24448103patch application.

[+ASM1.root@srv-stb1~]$$ORACLE_HOME/OPatch/opatchauto apply

/U01/patch/24448103

System initialization log file is/u01/app/12.1.0.2/grid/

Cfgtoollogs/opatchautodb/systemconfig2017-01-11_07-24-29PM.log.

Session log file is/u01/app/12.1.0.2/grid/cfgtoollogs/

Opatchauto/opatchauto2017-01-11_07-24-46PM.log

The id forthissession isWM33

[Init:init]Executing OPatchAutoBinaryAction action on home/

U01/app/12.1.0.2/grid

[Init:init]Executing OPatchAutoBinaryAction action on home/

U01/app/oracle/product/12.1.0.2/dbhome_1

Executing OPatch prereq operations toverify patch

Applicability on CRS Home........

Executing OPatch prereq operations toverify patch

Applicability on RAC Home........

[Init:init]OPatchAutoBinaryAction action completed on home/

U01/app/12.1.0.2/grid successfully

[Init:init]OPatchAutoBinaryAction action completed on home/

U01/app/oracle/product/12.1.0.2/dbhome_1 successfully

[Init:init]Executing GIRACPrereqAction action on home/u01/

App/oracle/product/12.1.0.2/dbhome_1

[Init:init]Executing GIRACPrereqAction action on home/u01/

App/12.1.0.2/grid

Executing prereq operations before applying on CRS Home........

Executing prereq operations before applying on RAC Home........

[Init:init]GIRACPrereqAction action completed on home/u01/app/

Oracle/product/12.1.0.2/dbhome_1 successfully

[Init:init]GIRACPrereqAction action completed on home/u01/app/

12.1.0.2/grid successfully

[Init:init]Executing RACDBPrereqAction action on home/u01/app/

Oracle/product/12.1.0.2/dbhome_1

Executing prereq operations before applying on RAC Home........

[Init:init]RACDBPrereqAction action completed on home/u01/app/

Oracle/product/12.1.0.2/dbhome_1 successfully

[Shutdown:prepare-shutdown]Executing RACPrepareShutDownAction

Action on home/u01/app/oracle/product/12.1.0.2/dbhome_1

Preparing RAC Home tobring down database service........

[Shutdown:prepare-shutdown]RACPrepareShutDownAction action

Completed on home/u01/app/oracle/product/12.1.0.2/dbhome_1

Successfully

[Shutdown:shutdown]Executing RACOneAction action on home/

U01/app/oracle/product/12.1.0.2/dbhome_1

/u01/app/oracle/product/12.1.0.2/dbhome_1 isnotaRACOne

data base.No step execution required........

[Shutdown:shutdown]RACOneAction action completed on home/

U01/app/oracle/product/12.1.0.2/dbhome_1 successfully

[Shutdown:shutdown]Executing GIShutDownAction action on

Home/u01/app/12.1.0.2/grid

Performing prepatch operations on CRS Home........

Prepatch operation log file location:/u01/app/12.1.0.2/

Grid/cfgtoollogs/crsconfig/

Crspatch_srv-stb1_2017-01-11_07-27-40PM.log

[Shutdown:shutdown]GIShutDownAction action completed on

Home/u01/app/12.1.0.2/grid successfully

[Offline:prepare]Executing RACPatchingAction action on

Home/u01/app/oracle/product/12.1.0.2/dbhome_1

Performing prepatch operation on RAC Home........

[Offline:prepare]RACPatchingAction action completed on home/

U01/app/oracle/product/12.1.0.2/dbhome_1 successfully

[Offline:binary-patching]Executing OPatchAutoBinaryAction

Action on home/u01/app/oracle/product/12.1.0.2/dbhome_1

Start applying binary patches on RAC Home........

[Offline:binary-patching]OPatchAutoBinaryAction action

Completed on home/u01/app/oracle/product/12.1.0.2/dbhome_1

Successfully

[Offline:finalize]Executing RACPatchingAction action on

Home/u01/app/oracle/product/12.1.0.2/dbhome_1

Performing postpatch operation on RAC Home........

[Offline:finalize]RACPatchingAction action completed on home/

U01/app/oracle/product/12.1.0.2/dbhome_1 successfully

[Offline:binary-patching]Executing OPatchAutoBinaryAction

Action on home/u01/app/12.1.0.2/grid

Start applying binary patches on CRS Home........

[Offline:binary-patching]OPatchAutoBinaryAction action

Completed on home/u01/app/12.1.0.2/grid successfully

[Startup:startup]Executing GIStartupAction action on home/

U01/app/12.1.0.2/grid

Performing postpatch operations on CRS Home........

Postpatch operation log file location:/u01/app/12.1.0.2/

Grid/cfgtoollogs/crsconfig/

Crspatch_srv-stb1_2017-01-11_07-34-12PM.log

[Startup:startup]GIStartupAction action completed on home/

U01/app/12.1.0.2/grid successfully

[Startup:startup]Executing RACOneAction action on home/

U01/app/oracle/product/12.1.0.2/dbhome_1

/u01/app/oracle/product/12.1.0.2/dbhome_1 isnotaRACOne

data base.No step execution required........

[Startup:startup]RACOneAction action completed on home/

U01/app/oracle/product/12.1.0.2/dbhome_1 successfully

[Startup:finalize]Executing RACFinalizeStartAction action

On home/u01/app/oracle/product/12.1.0.2/dbhome_1

No step execution required.........

[Startup:finalize]RACFinalizeStartAction action completed

On home/u01/app/oracle/product/12.1.0.2/dbhome_1 successfully

[Online:product-patching]Executing RACOnlineAction action

On home/u01/app/oracle/product/12.1.0.2/dbhome_1

Trying toapply SQL patches on RAC Home.

[WARNING]The database instance'stb1'from'/ u01 / app / oracle /

Product / 12.1.0.2 / dbhome_1 ',inhost'srv-stb1'isnotrunning.

SQL changes,ifany,will notbe applied.

Toapply.The SQL changes,bring up the database instance

Andrun the command manually from any one node(run asoracle).

Refer tothe readme toget the correct steps forapplying

The sql changes.

[Online:product-patching]RACOnlineAction action completed

On home/u01/app/oracle/product/12.1.0.2/dbhome_1

Successfully

[Finalize:finalize]Executing OracleHomeLSInventoryGrepAction

Action on home/u01/app/12.1.0.2/grid

Verifying patches applied on CRS Home.

[Finalize:finalize]OracleHomeLSInventoryGrepAction action

Completed on home/u01/app/12.1.0.2/grid successfully

[Finalize:finalize]Executing OracleHomeLSInventoryGrepAction

Action on home/u01/app/oracle/product/12.1.0.2/dbhome_1

Verifying patches applied on RAC Home.

[Finalize:finalize]OracleHomeLSInventoryGrepAction action

Completed on home/u01/app/oracle/product/12.1.0.2/dbhome_1

Successfully

---------------------Summary--------------------------------

Patching iscompleted successfully.Please find the summary

asfollows:

Host:srv-stb1

RAC Home:/u01/app/oracle/product/12.1.0.2/dbhome_1

Summary:

==Following patches were SKIPPED:

Patch:/u01/patch/24448103/21436941

Reason:Thispatch isnotapplicable tothisspecified

Target type-"rac_database"

Patch:/u01/patch/24448103/24007012

Reason:Thispatch isnotapplicable tothisspecified

Target type-"rac_database"

==Following patches were SUCCESSFULLY applied:

Patch:/u01/patch/24448103/24340679

Log:/u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/

Opatchauto/core/opatch/opatch2017-01-11_19-28-50PM_1.log

Patch:/u01/patch/24448103/24846605

Log:/u01/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/

Opatchauto/core/opatch/opatch2017-01-11_19-28-50PM_1.log

Host:srv-stb1

CRS Home:/u01/app/12.1.0.2/grid

Summary:

==Following patches were SKIPPED:

Patch:/u01/patch/24448103/21436941

Reason:Thispatch isalready applied,so notgoing

Toapply again.

==Following patches were SUCCESSFULLY applied:

Patch:/u01/patch/24448103/24007012

Log:/u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/

Opatch/opatch2017-01-11_19-30-22PM_1.log

Patch:/u01/patch/24448103/24340679

Log:/u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/

Opatch/opatch2017-01-11_19-30-22PM_1.log

Patch:/u01/patch/24448103/24846605

Log:/u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/

Opatch/opatch2017-01-11_19-30-22PM_1.log

OPatchAutosuccessful.

Re-apply patch 21923026 that we had rolled back.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

[Stb1.oracle@srv-stb1 patch]$unzip

P21923026_12102161018ProactiveBP_Linux-x86-64.zip

Archive:p21923026_12102161018ProactiveBP_Linux-x86-64.zip

Creating:21923026/

Inflating:21923026/README.txt

Creating:21923026/files/

Creating:21923026/files/lib/

Creating:21923026/files/lib/libserver12.a/

Inflating:21923026/files/lib/libserver12.a/kdo.o

Inflating:21923026/files/lib/libserver12.a/kdblc.o

Inflating:21923026/files/lib/libserver12.a/kd9ir2c.o

Creating:21923026/etc/

Creating:21923026/etc/config/

Inflating:21923026/etc/config/actions.xml

Inflating:21923026/etc/config/inventory.xml

Replace PatchSearch.xml?[Y]is,[n]o,[A]ll,[N]one,[r]ename:A

Inflating:PatchSearch.xml

Before applying first check to see if there is any conflict.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

[Stb1.oracle@srv-stb121923026]$opatch prereq

CheckConflictAgainstOHWithDetail-ph./

Oracle Interim Patch Installer version12.1.0.1.12

Copyright(c)2017,Oracle Corporation.All rights reserved.

PREREQ session

Oracle Home:/u01/app/oracle/product/12.1.0.2/

Dbhome_1

Central Inventory:/u01/app/oraInventory

From:/u01/app/oracle/product/12.1.0.2/

Dbhome_1/oraInst.loc

OPatch version:12.1.0.1.12

OUI version:12.1.0.2.0

Log file location:/u01/app/oracle/product/12.1.0.2/

Dbhome_1/cfgtoollogs/opatch/opatch2017-01-11_21-01-27PM_1.log

Invoking prereq"checkconflictagainstohwithdetail"

Prereq"checkConflictAgainstOHWithDetail"passed.

OPatch succeeded.

Since we are not going to use the opatchauto for this patch, apply the patch in question only to the local node and when we finish this node do the same procedure on the other node.
Apply the patch with the -local option.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

[Stb1.oracle@srv-stb121923026]$opatch apply-local

Oracle Interim Patch Installer version12.1.0.1.12

Copyright(c)2017,Oracle Corporation.All rights

Reserved.

Oracle Home:/u01/app/oracle/product/12.1.0.2/

Dbhome_1

Central Inventory:/u01/app/oraInventory

From:/u01/app/oracle/product/12.1.0.2/

Dbhome_1/oraInst.loc

OPatch version:12.1.0.1.12

OUI version:12.1.0.2.0

Log file location:/u01/app/oracle/product/12.1.0.2/

Dbhome_1/cfgtoollogs/opatch/opatch2017-01-11_21-06-57PM_1.log

Verifying environment andperforming prerequisite checks...

OPatch continues with these patches:21923026

Doyou want toproceed?[Y|n]

And

User Responded with:Y

All checks passed.

Isthe local system ready forpatching?[Y|n]

And

User Responded with:Y

Backing up files...

Applying interim patch'21923026'toOH'/ u01 / app / oracle /

Product / 12.1.0.2 / dbhome_1 '

Patching component oracle.rdbms,12.1.0.2.0...

Patch21923026successfully applied.

Log file location:/u01/app/oracle/product/12.1.0.2/

Dbhome_1/cfgtoollogs/opatch/

Opatch2017-01-11_21-06-57PM_1.log

OPatch succeeded.

Now check the applied patches again. Note that the patch we apply is a Bundle Patch that is composed of 4 patches. The patch applied was BP 12.1.0.2.161018 which has the number 24340679.

1

2

3

4

5

6

7

8

[Stb1.oracle@srv-stb121923026]$opatch lspatches

21923026;

24846605;OCW Interim patch for24846605

24340679;DATABASE BUNDLE PATCH:12.1.0.2.161018(24340679)

23177536;Database PSU12.1.0.2.160719,Oracle JavaVM Component(JUL2016)

23520664;OCW Interim patch for23520664

OPatch succeeded.

Now go up the instance again.

1

[Stb1.oracle@srv-stb1 OPatch]$srvctl start instance-istb1-dstb

Log in to the Broker and change the instance that is running redoapply.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

[Stb1.oracle@srv-stb1 u01]$dgmgrl/

DGMGRL>show database stb;

Database-stb

Role:PHYSICAL STANDBY

Intended State:APPLY-ON

Transport Lag:0seconds(computed0seconds ago)

Apply Lag:0seconds(computed0seconds ago)

Average Apply Rate:1.23MByte/s

Real Time Query:OFF

Instance(s):

Stb1

Stb2(apply instance)

Database Status:

SUCCESS

DGMGRL>edit database stb set state=apply-off;

Succeeded.

DGMGRL>edit database stb set state=apply-on with apply

Instance=stb1;

Succeeded.

DGMGRL>show database stb;

Database-stb

Role:PHYSICAL STANDBY

Intended State:APPLY-ON

Transport Lag:0seconds(computed0seconds ago)

Apply Lag:0seconds(computed0seconds ago)

Average Apply Rate:8.51MByte/s

Real Time Query:OFF

Instance(s):

Stb1(apply instance)

Stb2

Database Status:

SUCCESS

Done that is just to repeat the whole procedure in node 2. I will not present here to not be repeated, but the step by step is the same.