Oracle RAC 19c RU 打补丁踩到的那些坑

科技 2024-10-17 08:35 中国

作者 | JiekeXu

来源 |公众号 JiekeXu DBA之路（ID: JiekeXu_IT）

如需转载请联系授权 | (个人微信 ID：JiekeXu_DBA)

大家好，我是 JiekeXu,江湖人称“强哥”，荣获 Oracle ACE Pro 称号，墨天轮 MVP，墨天轮年度“墨力之星”，拥有 Oracle OCP/OCM 认证,MySQL 5.7/8.0 OCP 认证以及 PCA、PCTA、OBCA、OGCA、KCP 等众多国产数据库认证证书,今天和大家一起来看看 Oracle RAC 19c RU 打补丁踩到的那些坑，欢迎点击最上方蓝字“JiekeXu DBA之路”关注我的微信公众号，然后点击右上方三个点“设为星标”置顶，更多干货文章才能第一时间推送，谢谢！

目录前言打补丁步骤使用 root 分析补丁问题 1 Invalid current directory 问题 2 OPATCHAUTO-72132 问题 3 CLSRSC-180 手动打补丁问题 4 CRS-6706 检查补丁节点 2 自动打补丁 GI 自动打补丁 DB 自动打补丁失败开库，验证补丁执行 datapatch 编译无效对象 SQL 查看补丁信息总结

前言

Oracle 数据库是一款世界级的优秀产品，业界地位无可动摇，但这么优秀的数据库也存在很多 BUG 缺陷，不像国产数据库，要是有一个 BUG 就会被友商抓住不放，大肆宣扬这个数据库怎么怎么样，有什么问题不能用，自家的数据库多好多好，听到这样的话语实在是让人无语。。。。。。除非遇到致命的 BUG 导致不可用，不然就算是 BUG 也没啥事呀，而在 O 记官方每个季度均会发布补丁包来修复一些已知的问题，而且最近两年从 19.17 开始取消 RUR 补丁，提供每月的 MRP(Monthly Recommended Patches) 月度推荐补丁，本年第三季度的补丁这周二也已经发布了 RU25，顺道说一句 MySQL 9.1，8.4.3 LTS，8.0.40 新版本也发布了。

打补丁步骤

其实这个系列的文章已经写了很多篇，包含 11g、19c 单机 RAC 以及 AIX、Linux 平台的多篇文章，本来想着不在写了，但架不住国庆前一天，手贱非得找点活干，一测试环境 RAC 打补丁从 RU15 升级至 RU23,记得这个环境之前是从 RU19.4 升级到 RU19.15 的，升级过程一切顺利，想着本次升级应该也没问题，谁能想到，这一次几乎遇到了打补丁过程中遇到的所有问题，且从 30 号白天一直到国庆凌晨 2 点都没有完成。。。

Oracle 19c RAC RU15 补丁更新详细步骤
Oracle 19c 19.10DBRU 最新补丁升级看这一篇就够了
 Oracle 11gR2 RAC 190416 补丁安装指导

其他补丁相关文章链接：https://mp.weixin.qq.com/s/2DoBV-8PsoYENgwM3BAIaQhttps://mp.weixin.qq.com/s/zS4aE3TfhbMwDk37nIEGfghttps://mp.weixin.qq.com/s/C0YbdGs2SVxqDuTm1g83fwhttps://www.modb.pro/db/27255https://www.modb.pro/doc/1602https://www.modb.pro/doc/4638

下面正文开始，简要步骤如下：

上传补丁到 /u01/RU23 目录下，然后补丁包使用 grid 解压并给与 777 权限，然后替换各自家目录 OPatch 工具包。

jieke-rac1:/u01/app/oracle/product/19.0.0.0/dbhome_1/rdbms/admin(JiekeXu1)$ opatch versionOPatch Version: 12.2.0.1.42
OPatch succeeded.

A new feature has been added to OPatch to increase performance by deleting inactive patches. See My Oracle Support document 2942102.1 OPatch 12.2.0.1.37+ Introduces a New Feature to Delete Inactive Patches in the ORACLE_HOME/.patch_storage directory.OPatch 添加了一项新功能，可通过删除非活动补丁提高性能。请参见 Oracle 支持文档 2942102.1: OPatch 12.2.0.1.37+ 引入了一项新功能，可删除 ORACLE_HOME/.patch_storage 目录中的非活动补丁。

空间检测、补丁冲突略。 下一步，进行补丁分析。

使用 root 分析补丁

[root@jieke-rac1 RU23]# /u01/app/19.0.0.0/grid/OPatch/opatchauto apply /u01/RU23/36209493/36233126 -analyze
OPatchauto session is initiated at Mon Sep 30 11:15:50 2024
System initialization log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2024-09-30_11-16-09AM.log.
Clusterware is either not running or not configured. You have the following 2 options: 1. Configure and start the Clusterware on this node and re-run the tool 2. Run the tool with '-oh <GI_HOME>' to first patch the Grid Home, then invoke tool with '-database <oracle database name>' or '-oh <RAC_HOME>' to patch the RAC home
OPATCHAUTO-72029: CLusterware home not configured.OPATCHAUTO-72029: Clusterware is either not running or not configured or cluster is software only GIOPATCHAUTO-72029: If only Grid Infrastructure software is installed, please run opatchauto with '-oh' option . Alternatively configure and start the Clusterware before running opatchauto.
OPatchauto session completed at Mon Sep 30 11:16:16 2024Time taken to complete the session 0 minute, 9 seconds
Topology creation failed.

注意：grid 全路径以及 36233126（GI 和 DB 均可使用的补丁）目录，由于集群运行不正常，还是其他未知问题，这里需要 -oh 参数指定 GI 家目录来运行补丁分析，平时是不需要 -oh 参数就能分析的。

[root@jieke-rac1 RU23]# /u01/app/19.0.0.0/grid/OPatch/opatchauto apply /u01/RU23/36209493/36233126 -analyze -oh /u01/app/19.0.0.0/grid
OPatchauto session is initiated at Mon Sep 30 11:24:24 2024System initialization log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2024-09-30_11-24-46AM.log.
Session log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/opatchauto2024-09-30_11-29-57AM.logThe id for this session is 3LMP
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0.0/grid
Patch applicability verified successfully on home /u01/app/19.0.0.0/grid
Executing patch validation checks on home /u01/app/19.0.0.0/gridPatch validation checks successfully completed on home /u01/app/19.0.0.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Analysis for applying patches has completed successfully:
Host:jieke-rac1CRS Home:/u01/app/19.0.0.0/gridVersion:19.0.0.0.0
==Following patches were SUCCESSFULLY analyzed to be applied:
Patch: /u01/RU23/36209493/36233126/36240578Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_11-31-33AM_1.log
Patch: /u01/RU23/36209493/36233126/36233343Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_11-31-33AM_1.log
Patch: /u01/RU23/36209493/36233126/36383196Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_11-31-33AM_1.log
Patch: /u01/RU23/36209493/36233126/36460248Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_11-31-33AM_1.log
Patch: /u01/RU23/36209493/36233126/36233263Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_11-31-33AM_1.log
OPatchauto session completed at Mon Sep 30 11:40:24 2024Time taken to complete the session 15 minutes, 39 seconds
--节点 2 分析时间Time taken to complete the session 13 minutes, 6 seconds

如上所示，分析补丁时，没有遇到任何问题。

下面正式开始打补丁。

问题 1 Invalid current directory

如下所示，不能在 root 家/根目录下执行打补丁的命令，否则会报如下错误：

[root@jieke-rac1 ~]# /u01/app/19.0.0.0/grid/OPatch/opatchauto apply /u01/RU23/36209493/36233126 -oh /u01/app/19.0.0.0/grid
Invalid current directory.  Please run opatchauto from other than '/root' or '/' directory.And check if the home owner user has write permission set for the current directory.opatchauto returns with error code = 2

问题 2 OPATCHAUTO-72132

换个目录到补丁解压目录下，自动打补丁报错 Grid 没有运行。

[root@jieke-rac1 RU23]# /u01/app/19.0.0.0/grid/OPatch/opatchauto apply /u01/RU23/36209493/36233126 -oh /u01/app/19.0.0.0/grid
OPatchauto session is initiated at Mon Sep 30 13:38:33 2024
System initialization log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2024-09-30_01-39-01PM.log.
Session log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/opatchauto2024-09-30_01-40-42PM.logThe id for this session is PRL7OPATCHAUTO-72132: Grid is not running on the local host.OPATCHAUTO-72132: Cannot start a new apply or rollback session when the local grid is not running.OPATCHAUTO-72132: Please start grid service on the local host to start patching.OPatchAuto failed.
OPatchauto session completed at Mon Sep 30 13:42:37 2024Time taken to complete the session 3 minutes, 37 seconds
 opatchauto failed with error code 42

问题 2 解决过程

经查看，原来是节点 1 的 evmd 不知什么时候没有运行，查看节点 2 正常。

jieke-rac1:/u01/RU23(+ASM1)$ crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4534: Cannot communicate with Event Manager--节点 2 正常jieke-rac2:/home/grid(+ASM2)$ crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online

使用 Grid 用户启动 evmd 资源，再次检查资源正常。

jieke-rac1:/u01/RU23(+ASM1)$ crsctl start res ora.evmd -init CRS-2672: Attempting to start 'ora.evmd' on 'jieke-rac1'CRS-2676: Start of 'ora.evmd' on 'jieke-rac1' succeeded
jieke-rac1:/u01/RU23(+ASM1)$ crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online

问题 3 CLSRSC-180

重新使用 opatchauto apply 指定 GI 家目录打补丁。

jieke-rac1:/u01/RU23(+ASM1)$ /u01/app/19.0.0.0/grid/OPatch/opatchauto apply /u01/RU23/36209493/36233126 -oh /u01/app/19.0.0.0/grid
OPatchauto session is initiated at Mon Sep 30 14:04:57 2024
System initialization log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2024-09-30_02-05-20PM.log.
Session log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/opatchauto2024-09-30_02-07-01PM.logThe id for this session is VAZP
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0.0/gridPatch applicability verified successfully on home /u01/app/19.0.0.0/grid

Executing patch validation checks on home /u01/app/19.0.0.0/gridPatch validation checks successfully completed on home /u01/app/19.0.0.0/grid

Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.0.0.0/gridPrepatch operation log file location: /u01/app/grid/crsdata/jieke-rac1/crsconfig/crs_prepatch_apply_inplace_ops-edwrac-86_2024-09-30_02-13-43PM.logCRS service brought down successfully on home /u01/app/19.0.0.0/grid

Start applying binary patch on home /u01/app/19.0.0.0/gridBinary patch applied successfully on home /u01/app/19.0.0.0/grid

Running rootadd_rdbms.sh on home /u01/app/19.0.0.0/gridSuccessfully executed rootadd_rdbms.sh on home /u01/app/19.0.0.0/grid
Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.0.0.0/gridPostpatch operation log file location: /u01/app/grid/crsdata/jieke-rac1/crsconfig/crs_postpatch_apply_inplace_jieke-rac1_2024-09-30_02-40-18PM.logFailed to start CRS service on home /u01/app/19.0.0.0/grid
Execution of [GIStartupAction] patch action failed, check log for more details. Failures:Patch Target : jieke-rac1->/u01/app/19.0.0.0/grid Type[crs]Details: [---------------------------Patching Failed---------------------------------Command execution failed during patching in home: /u01/app/19.0.0.0/grid, host: jieke-rac1.Command failed:  bash -c "LD_LIBRARY_PATH=/u01/app/19.0.0.0/grid/lib:/u01/app/19.0.0.0/grid/srvm/lib:/u01/RU23/36209493/36233126/36240578/files/lib:/u01/app/19.0.0.0/grid/lib:/u01/app/19.0.0.0/grid/lib:/u01/app/19.0.0.0/grid/srvm/lib:/u01/app/19.0.0.0/grid/lib:/u01/app/19.0.0.0/grid/srvm/lib:/u01/app/19.0.0.0/grid/lib:/lib:/usr/lib /u01/app/19.0.0.0/grid/perl/bin/perl -I/u01/app/19.0.0.0/grid/perl/lib -I/u01/app/19.0.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_jieke-rac1/patchwork/crs/install -I/u01/app/19.0.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_jieke-rac1/patchwork/xag /u01/app/19.0.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_jieke-rac1/patchwork/crs/install/rootcrs.pl -postpatch"Command failure output: Using configuration parameter file: /u01/app/19.0.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_jieke-rac1/patchwork/crs/install/crsconfig_paramsThe log of current session can be found at:  /u01/app/grid/crsdata/jieke-rac1/crsconfig/crs_postpatch_apply_inplace_jieke-rac1_2024-09-30_02-40-18PM.log2024/09/30 14:40:46 CLSRSC-180: An error occurred while executing the command '/bin/rpm -qf /sbin/init' 
After fixing the cause of failure Run opatchauto resume
]OPATCHAUTO-68061: The orchestration engine failed.OPATCHAUTO-68061: The orchestration engine failed with return code 1OPATCHAUTO-68061: Check the log for more details.OPatchAuto failed.
OPatchauto session completed at Mon Sep 30 14:40:50 2024Time taken to complete the session 35 minutes, 31 seconds
 opatchauto failed with error code 42jieke-rac1:/u01/RU23(+ASM1)$

通过上面日志可以看到，补丁已经执行成功，只是在启动 CRS 服务 start CRS service 时出错了，’/bin/rpm -qf /sbin/init’ 命令没有执行成功而报错。

问题 3 解决过程

root 直接执行此命令也是报错。

[root@jieke-rac1 RU23]# /bin/rpm -qf /sbin/initerror: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB libraryerror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recoveryerror: cannot open Packages index using db5 -  (-30973)error: cannot open Packages database in /var/lib/rpmerror: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB libraryerror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recoveryerror: cannot open Packages database in /var/lib/rpmerror: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB libraryerror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recoveryerror: cannot open Packages database in /var/lib/rpmfile /sbin/init is not owned by any package

那么需要查看日志解决相关问题，

more /u01/app/grid/crsdata/jieke-rac1/crsconfig/crs_postpatch_apply_inplace_jieke-rac1_2024-09-30_02-40-18PM.log
2024-09-30 14:40:45: Executing cmd: /bin/rpm -q sles-release2024-09-30 14:40:46: Command output:>  error: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB library>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery>  error: cannot open Packages index using db5 -  (-30973)>  error: cannot open Packages database in /var/lib/rpm>  error: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB library>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery>  error: cannot open Packages database in /var/lib/rpm>  package sles-release is not installed >End Command output2024-09-30 14:40:46: Check if the startup mechanism systemd is being used2024-09-30 14:40:46: Executing cmd: /bin/rpm -qf /sbin/init2024-09-30 14:40:46: Command output:>  error: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB library>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery>  error: cannot open Packages index using db5 -  (-30973)>  error: cannot open Packages database in /var/lib/rpm>  error: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB library>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery>  error: cannot open Packages database in /var/lib/rpm>  error: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB library>  error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery>  error: cannot open Packages database in /var/lib/rpm>  file /sbin/init is not owned by any package >End Command output2024-09-30 14:40:46: Executing cmd: /u01/app/19.0.0.0/grid/bin/clsecho -p has -f clsrsc -m 180 '/bin/rpm -qf /sbin/init' '1'2024-09-30 14:40:46: Executing cmd: /u01/app/19.0.0.0/grid/bin/clsecho -p has -f clsrsc -m 180 '/bin/rpm -qf /sbin/init' '1'2024-09-30 14:40:46: Command output:>  CLSRSC-180: An error occurred while executing the command '/bin/rpm -qf /sbin/init' >End Command output2024-09-30 14:40:46: CLSRSC-180: An error occurred while executing the command '/bin/rpm -qf /sbin/init'

通过上面日志分析由于 rpm 不能打开 /var/lib/rpm 的数据库，导致 /bin/rpm -qf/sbin/init 命令执行失败，进而导致 ROOTCRS_POSTPATCH_UPDATE_OHASD_SERVICE checkpoint has failed。

注意：如果说上面出现的问题都解决了，可以使用 resume 参数接着执行打补丁的命令。

[root@jieke-rac1 RU23]# /u01/app/19.0.0.0/grid/OPatch/opatchauto resume

晚上仔细去看日志，在执行上面命令前，停止 CRS 也报错了，但不影响后续执行。如下所示：

2024-09-30 14:40:42: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl stop crs  -f2024-09-30 14:40:43: Command output:>  CRS-4639: Could not contact Oracle High Availability Services>  CRS-4000: Command Stop failed, or completed with errors. >End Command output2024-09-30 14:40:43: The return value of stop of CRS: 12024-09-30 14:40:43: Executing cmd: /u01/app/19.0.0.0/grid/bin/crsctl check crs2024-09-30 14:40:43: Command output:>  CRS-4639: Could not contact Oracle High Availability Services >End Command output2024-09-30 14:40:43: Oracle CRS stack has been shut down2024-09-30 14:40:43: update system files

当然，上面的分析是晚上进行的，白天由于时间紧急，采用了手动打补丁的方式。

我们已经自动打完了 GI 的补丁，只是没法启动 CRS，那么我们先手动打完 DB 的补丁，然后再来解决 CRS 无法启动的问题。

手动打补丁

手动打 GI 的补丁，由于 36233263 补丁前面已经自动打上了，这里提示直接跳过了。

jieke-rac1:/u01/RU23/36209493/36233126(+ASM1)$ opatch apply -oh $ORACLE_HOME -local /u01/RU23/36209493/36233126/36233263Oracle Interim Patch Installer version 12.2.0.1.42Copyright (c) 2024, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/19.0.0.0/gridCentral Inventory : /u01/app/oraInventory   from           : /u01/app/19.0.0.0/grid/oraInst.locOPatch version    : 12.2.0.1.42OUI version       : 12.2.0.7.0Log file location : /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2024-09-30_15-16-45PM_1.log
Verifying environment and performing prerequisite checks...
The following patch(es) are duplicate patches with patches installed in the Oracle Home. [ 36233263]You have already installed same patch(es) with same UPI(s) or same version(s).These patch(es) will be skipped.

OPatch Session completed with warnings.Log file location: /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2024-09-30_15-16-45PM_1.log
OPatch completed with warnings.

手动打节点 1 DB 补丁

注意：手动打补丁时需要关库，停止监听，OGG 等其他应用

jieke-rac1:/u01/RU23/36209493/36233126/36240578/custom/scripts(JiekeXu1)$ ./prepatch.sh -dbhome /u01/app/oracle/product/19.0.0.0/dbhome_1./prepatch.sh completed successfully.
jieke-rac1:/u01/RU23/36209493/36233126(JiekeXu1)$ opatch apply -oh /u01/app/oracle/product/19.0.0.0/dbhome_1 /u01/RU23/36209493/36233126/36233263
jieke-rac1:/u01/RU23/36209493/36233126(JiekeXu1)$ opatch apply -oh /u01/app/oracle/product/19.0.0.0/dbhome_1 /u01/RU23/36209493/36233126/36240578
jieke-rac1:/u01/RU23/36209493/36233126(JiekeXu1)$ cd /u01/RU23/36209493/36233126/36240578jieke-rac1:/u01/RU23/36209493/36233126/36240578/custom/scripts(JiekeXu1)$ ./postpatch.sh -dbhome /u01/app/oracle/product/19.0.0.0/dbhome_1

上面已经手动打完了 DB 的补丁，接下来我们得去解决 GI 打补丁失败的问题，这里尝试执行 rootadd_rdbms.sh postpatch 失败。

[root@jieke-rac1 36233126]# /u01/app/19.0.0.0/grid/rdbms/install/rootadd_rdbms.sh[root@jieke-rac1 36233126]# /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -postpatchSmartmatch is deprecated at /u01/app/19.0.0.0/grid/crs/install/crsupgrade.pm line 6512.Using configuration parameter file: /u01/app/19.0.0.0/grid/crs/install/crsconfig_paramsThe log of current session can be found at:  /u01/app/grid/crsdata/jieke-rac1/crsconfig/crs_postpatch_apply_inplace_ops-edwrac-86_2024-09-30_04-05-54PM.log2024/09/30 16:06:18 CLSRSC-180: An error occurred while executing the command '/bin/rpm -qf /sbin/init'Died at /u01/app/19.0.0.0/grid/crs/install/s_crsutils.pm line 2729.The command '/u01/app/19.0.0.0/grid/perl/bin/perl -I/u01/app/19.0.0.0/grid/perl/lib -I/u01/app/19.0.0.0/grid/crs/install -I/u01/app/19.0.0.0/grid/xag /u01/app/19.0.0.0/grid/crs/install/rootcrs.pl -postpatch' execution failed[root@jieke-rac1 36233126]#

执行 postpatch 也是一样的报错，那么我们需要解决此问题了，不解决 CRS 看来不能正常启动了。然后通过 MOS ID 2365433.1 找到了相关问题，操作系统级 RPM 包所使用的数据库已损坏

Postpatch Command Fails on Grid Infrastructure Home Due To rpm Database Corruption At The OS Level (Doc ID 2365433.1) To BottomTo Bottom

那么处理方案也比较简单，删除 /var/lib/rpm/__ 的目录，重新创建 RPM 包的数据库即可。

[root@jieke-rac1 RU23]# /bin/rpm -qa | moreerror: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB libraryerror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recoveryerror: cannot open Packages index using db5 -  (-30973)error: cannot open Packages database in /var/lib/rpmerror: rpmdb: BDB0113 Thread/process 54749/140110208747584 failed: BDB1507 Thread died in Berkeley DB libraryerror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recoveryerror: cannot open Packages database in /var/lib/rpm[root@jieke-rac1 RU23]# [root@jieke-rac1 RU23]# ll /var/lib/rpm/__*-rw-r--r-- 1 root root  311296 Sep 30 16:24 /var/lib/rpm/__db.001-rw-r--r-- 1 root root   90112 Sep 30 16:24 /var/lib/rpm/__db.002-rw-r--r-- 1 root root 1318912 Sep 30 16:24 /var/lib/rpm/__db.003[root@jieke-rac1 RU23]# rm -rf /var/lib/rpm/__*[root@jieke-rac1 RU23]# [root@jieke-rac1 RU23]# /bin/rpm --rebuilddb[root@jieke-rac1 RU23]# echo $?0[root@jieke-rac1 RU23]# /bin/rpm -qa | morelibblockdev-mdraid-2.18-3.el7.x86_64perl-Filter-1.49-3.el7.x86_64joda-time-2.2-3.tzdata2013c.el7.noarchlibmspack-0.5-0.6.alpha.el7.x86_64control-center-filesystem-3.28.1-4.el7.x86_64glusterfs-3.12.2-18.el7.x86_64perl-File-Path-2.09-2.el7.noarchslf4j-1.7.4-4.el7_4.noarch

问题 4 CRS-6706

当解决掉上面的问题后我们继续执行 postpatch 提示由于 prepatch 没有执行成功导致的。prepatch 也没法执行，提示集群版本不一致，直接启动 CRS 集群也是一样的报错。

[root@jieke-rac1 36233126]# /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -postpatchSmartmatch is deprecated at /u01/app/19.0.0.0/grid/crs/install/crsupgrade.pm line 6512.Using configuration parameter file: /u01/app/19.0.0.0/grid/crs/install/crsconfig_paramsThe log of current session can be found at:  /u01/app/grid/crsdata/jieke-rac1/crsconfig/crs_postpatch_apply_inplace_jieke-rac1_2024-09-30_04-25-46PM.log2024/09/30 16:25:49 CLSRSC-739: cannot run the postpatch operation because the prepatch operation did not complete successfullyDied at /u01/app/19.0.0.0/grid/crs/install/crspatch.pm line 3019.The command '/u01/app/19.0.0.0/grid/perl/bin/perl -I/u01/app/19.0.0.0/grid/perl/lib -I/u01/app/19.0.0.0/grid/crs/install -I/u01/app/19.0.0.0/grid/xag /u01/app/19.0.0.0/grid/crs/install/rootcrs.pl -postpatch' execution failed
[root@jieke-rac1 36233126]# /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -prepatchSmartmatch is deprecated at /u01/app/19.0.0.0/grid/crs/install/crsupgrade.pm line 6512.Using configuration parameter file: /u01/app/19.0.0.0/grid/crs/install/crsconfig_paramsThe log of current session can be found at:  /u01/app/grid/crsdata/jieke-rac1/crsconfig/crs_prepatch_apply_inplace_ops-edwrac-86_2024-09-30_04-26-37PM.logCRS-6706: Oracle Clusterware Release patch level ('1545812015') does not match Software patch level ('3509829075'). Oracle Clusterware cannot be started.CRS-4000: Command Start failed, or completed with errors.2024/09/30 16:26:51 CLSRSC-117: Failed to start Oracle Clusterware stack from the Grid Infrastructure home /u01/app/19.0.0.0/gridDied at /u01/app/19.0.0.0/grid/crs/install/crspatch.pm line 1773.The command '/u01/app/19.0.0.0/grid/perl/bin/perl -I/u01/app/19.0.0.0/grid/perl/lib -I/u01/app/19.0.0.0/grid/crs/install -I/u01/app/19.0.0.0/grid/xag /u01/app/19.0.0.0/grid/crs/install/rootcrs.pl -prepatch' execution failed

问题 4 解决过程

那么我们通过 MOS 的 ID 2884975.1 文档处理，执行 clscfg -localpatch 后启动 CRS。

After apply the Patch 30094077, Patching another Grid Infrastructure gives error CRS-6706: Oracle Clusterware Release Patch Level (‘202851471’) Does Not Match Software Patch Level (Doc ID 2884975.1)

[root@jieke-rac1 36233126]# /u01/app/19.0.0.0/grid/bin/crsctl start crs -waitCRS-6706: Oracle Clusterware Release patch level ('1545812015') does not match Software patch level ('3509829075'). Oracle Clusterware cannot be started.CRS-4000: Command Start failed, or completed with errors.
[root@jieke-rac1 36233126]# /u01/app/19.0.0.0/grid/bin/clscfg -localpatchclscfg: EXISTING configuration version 0 detected.Creating OCR keys for user 'root', privgrp 'root'..Operation successful.[root@jieke-rac1 36233126]# /u01/app/19.0.0.0/grid/bin/crsctl start crs -waitCRS-4123: Starting Oracle High Availability Services-managed resourcesCRS-2672: 尝试启动 'ora.mdnsd' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.evmd' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.mdnsd' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.evmd' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.gpnpd' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.gpnpd' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.gipcd' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.gipcd' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.crf' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.cssdmonitor' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.cssdmonitor' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.cssd' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.diskmon' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.diskmon' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.crf' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.cssd' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.cluster_interconnect.haip' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.ctssd' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.ctssd' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.cluster_interconnect.haip' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.asm' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.asm' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.storage' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.storage' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.crsd' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.crsd' (在 'jieke-rac1' 上)CRS-6017: 正在处理以下服务器的资源自动启动: jieke-rac1CRS-2673: 尝试停止 'ora.LISTENER_SCAN1.lsnr' (在 'jieke-rac2' 上)CRS-2672: 尝试启动 'ora.chad' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.ons' (在 'jieke-rac1' 上)CRS-2677: 成功停止 'ora.LISTENER_SCAN1.lsnr' (在 'jieke-rac2' 上)CRS-2673: 尝试停止 'ora.scan1.vip' (在 'jieke-rac2' 上)CRS-2677: 成功停止 'ora.scan1.vip' (在 'jieke-rac2' 上)CRS-2672: 尝试启动 'ora.scan1.vip' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.chad' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.scan1.vip' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.LISTENER_SCAN1.lsnr' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.ons' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.LISTENER_SCAN1.lsnr' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.DATA.dg' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.FRA.dg' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.DATA.dg' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.FRA.dg' (在 'jieke-rac1' 上)CRS-2672: 尝试启动 'ora.edwstb.db' (在 'jieke-rac1' 上)CRS-2676: 成功启动 'ora.edwstb.db' (在 'jieke-rac1' 上)CRS-6016: 服务器 jieke-rac1 的资源自动启动已完成CRS-6024: 已完成启动 Oracle 集群就绪服务管理的资源CRS-4123: Oracle High Availability Services has been started.

启动 CRS 完成，查看 GI 和 DB 补丁版本均正常，reboot 主机也能正常启动 CRS，到这里算是彻底解决了。

检查补丁

[root@jieke-rac1 36233126]# su - grid Last login: Mon Sep 30 16:26:53 CST 2024 on pts/0jieke-rac1:/home/grid(+ASM1)$ jieke-rac1:/home/grid(+ASM1)$ opatch lspatches36460248;TOMCAT RELEASE UPDATE 19.0.0.0.0 (36460248)36383196;DBWLM RELEASE UPDATE 19.0.0.0.0 (36383196)36240578;OCW RELEASE UPDATE 19.23.0.0.0 (36240578)36233343;ACFS RELEASE UPDATE 19.23.0.0.0 (36233343)36233263;Database Release Update : 19.23.0.0.240416 (36233263)
OPatch succeeded.jieke-rac1:/home/grid(+ASM1)$ exitlogout[root@jieke-rac1 36233126]# su - oracle Last login: Mon Sep 30 15:32:21 CST 2024 on pts/0jieke-rac1:/home/oracle(JiekeXu1)$ opatch lspatches36240578;OCW RELEASE UPDATE 19.23.0.0.0 (36240578)36233263;Database Release Update : 19.23.0.0.240416 (36233263)33808367;OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367)
OPatch succeeded.

节点 2 自动打补丁

GI 自动打补丁

[root@jieke-rac2 RU23]# /u01/app/19.0.0.0/grid/OPatch/opatchauto apply /u01/RU23/36209493/36233126 -oh /u01/app/19.0.0.0/grid
OPatchauto session is initiated at Mon Sep 30 18:02:13 2024
System initialization log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2024-09-30_06-04-07PM.log.
Session log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/opatchauto2024-09-30_06-05-37PM.logThe id for this session is LXZ6
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0.0/gridPatch applicability verified successfully on home /u01/app/19.0.0.0/grid
Executing patch validation checks on home /u01/app/19.0.0.0/gridPatch validation checks successfully completed on home /u01/app/19.0.0.0/grid
Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/19.0.0.0/gridPrepatch operation log file location: /u01/app/grid/crsdata/jieke-rac2/crsconfig/crs_prepatch_apply_inplace_jieke-rac2_2024-09-30_06-14-01PM.logCRS service brought down successfully on home /u01/app/19.0.0.0/grid
Start applying binary patch on home /u01/app/19.0.0.0/gridBinary patch applied successfully on home /u01/app/19.0.0.0/grid
Running rootadd_rdbms.sh on home /u01/app/19.0.0.0/gridSuccessfully executed rootadd_rdbms.sh on home /u01/app/19.0.0.0/grid
Performing postpatch operations on CRS - starting CRS service on home /u01/app/19.0.0.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/jieke-rac2/crsconfig/crs_postpatch_apply_inplace_jieke-rac2_2024-09-30_06-37-43PM.logCRS service started successfully on home /u01/app/19.0.0.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:jieke-rac2CRS Home:/u01/app/19.0.0.0/gridVersion:19.0.0.0.0Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /u01/RU23/36209493/36233126/36233263Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_18-19-01PM_1.log
Patch: /u01/RU23/36209493/36233126/36233343Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_18-19-01PM_1.log
Patch: /u01/RU23/36209493/36233126/36240578Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_18-19-01PM_1.log
Patch: /u01/RU23/36209493/36233126/36383196Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_18-19-01PM_1.log
Patch: /u01/RU23/36209493/36233126/36460248Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_18-19-01PM_1.log
OPatchauto session completed at Mon Sep 30 18:56:58 2024Time taken to complete the session 52 minutes, 59 seconds

折腾了一整天，从早上十一点开始到晚上 19 点，居然没有打完补丁，问题不断，好在节点 2 GI 自动打补丁比较顺利，虽然花费时间有点长，但总算成功没有出错，本想可以回家过国庆了，还是太轴，想着要不打完 DB 补丁再走，谁成想，DB 自动打补丁失败了。

DB 自动打补丁失败

[root@jieke-rac2 RU23]# /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatchauto apply /u01/RU23/36209493/36233126 -oh /u01/app/oracle/product/19.0.0.0/dbhome_1
OPatchauto session is initiated at Mon Sep 30 19:07:26 2024
System initialization log file is /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2024-09-30_07-08-02PM.log.
Session log file is /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2024-09-30_07-13-43PM.logThe id for this session is ZLAT
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0.0/dbhome_1Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0.0/dbhome_1
Executing patch validation checks on home /u01/app/oracle/product/19.0.0.0/dbhome_1Patch validation checks successfully completed on home /u01/app/oracle/product/19.0.0.0/dbhome_1
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0.0/dbhome_1Skipping SQL patch step execution on standby database : edwstbSQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0.0/dbhome_1
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0.0/dbhome_1Successfully prepared home /u01/app/oracle/product/19.0.0.0/dbhome_1 to bring down database service
Bringing down database service on home /u01/app/oracle/product/19.0.0.0/dbhome_1Following database(s) and/or service(s) are stopped and will be restarted later during the session: edwstbDatabase service successfully brought down on home /u01/app/oracle/product/19.0.0.0/dbhome_1
Performing prepatch operation on home /u01/app/oracle/product/19.0.0.0/dbhome_1Prepatch operation completed successfully on home /u01/app/oracle/product/19.0.0.0/dbhome_1
Start applying binary patch on home /u01/app/oracle/product/19.0.0.0/dbhome_1
Failed while applying binary patches on home /u01/app/oracle/product/19.0.0.0/dbhome_1
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:Patch Target : jieke-rac2->/u01/app/oracle/product/19.0.0.0/dbhome_1 Type[rac]Details: [---------------------------Patching Failed---------------------------------Command execution failed during patching in home: /u01/app/oracle/product/19.0.0.0/dbhome_1, host: jieke-rac2.Command failed:  /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatchauto  apply /u01/RU23/36209493/36233126 -oh /u01/app/oracle/product/19.0.0.0/dbhome_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.0.0.0/dbhome_1/oraInst.loc -jre /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jre -persistresult /u01/app/oracle/product/19.0.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_ops-edwrac-87_rac_2.ser -analyzedresult /u01/app/oracle/product/19.0.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_ops-edwrac-87_rac_2.serCommand failure output: ==Following patches FAILED in apply:
Patch: /u01/RU23/36209493/36233126/36233263Log: /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_19-28-47PM_1.logReason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.
Patch: /u01/RU23/36209493/36233126/36240578Log: /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_19-28-47PM_1.logReason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed. 
After fixing the cause of failure Run opatchauto resume
]OPATCHAUTO-68061: The orchestration engine failed.OPATCHAUTO-68061: The orchestration engine failed with return code 1OPATCHAUTO-68061: Check the log for more details.OPatchAuto failed.
OPatchauto session completed at Mon Sep 30 19:32:30 2024Time taken to complete the session 24 minutes, 33 seconds
 opatchauto failed with error code 42

如上所示，花费 24 分钟去打 DB 补丁，竟然全都在 check，而且 check 失败，照实让人很生气，焦躁不安。如果上面的 check 失败问题解决了，可以接着前面的操作继续使用 resume 接着打补丁，这里由于没解决 check 失败问题就继续了，进一步也是失败。

[root@jieke-rac2 RU23]# /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatchauto resume                                                               Session log file is /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2024-09-30_07-36-27PM.logResuming existing session with id ZLAT
Start applying binary patch on home /u01/app/oracle/product/19.0.0.0/dbhome_1 Failed while applying binary patches on home /u01/app/oracle/product/19.0.0.0/dbhome_1
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:Patch Target : jieke-rac2->/u01/app/oracle/product/19.0.0.0/dbhome_1 Type[rac]Details: [---------------------------Patching Failed---------------------------------Command execution failed during patching in home: /u01/app/oracle/product/19.0.0.0/dbhome_1, host: jieke-rac2.Command failed:  /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatchauto  apply /u01/RU23/36209493/36233126 -oh /u01/app/oracle/product/19.0.0.0/dbhome_1 -target_type rac_database -binary -invPtrLoc /u01/app/oracle/product/19.0.0.0/dbhome_1/oraInst.loc -jre /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jre -persistresult /u01/app/oracle/product/19.0.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_ops-edwrac-87_rac_2.ser -analyzedresult /u01/app/oracle/product/19.0.0.0/dbhome_1/opatchautocfg/db/sessioninfo/sessionresult_analyze_ops-edwrac-87_rac_2.serCommand failure output: ==Following patches FAILED in apply:
Patch: /u01/RU23/36209493/36233126/36233263Log: /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_19-37-11PM_1.logReason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.
Patch: /u01/RU23/36209493/36233126/36240578Log: /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2024-09-30_19-37-11PM_1.logReason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed. 
After fixing the cause of failure Run opatchauto resume
]OPATCHAUTO-68061: The orchestration engine failed.OPATCHAUTO-68061: The orchestration engine failed with return code 1OPATCHAUTO-68061: Check the log for more details.OPatchAuto failed.
OPatchauto session completed at Mon Sep 30 19:40:58 2024Time taken to complete the session 4 minutes, 34 seconds
 opatchauto failed with error code 42

解决问题就得去查看详细日志解决问题，然后继续。如果通过日志排查不出来具体问题，实在也看不懂日志，那么你只有放弃这种办法了。我这里就是，焦躁不安，直接放弃，回家！！！

[root@jieke-rac2 ~]# grep "does not exi" /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto_2024-09-30_19-37-05_binary.log2024-09-30 19:38:44,012 INFO  [1] oracle.glcm.opatch.common.impl.SingletonPatch - /u01/RU23/36209493/36233126/36233263/etc/config/checksum.xml does not exist2024-09-30 19:38:44,013 INFO  [1] oracle.glcm.opatch.common.impl.SingletonPatch - /u01/RU23/36209493/36233126/36233263/etc/config/automation.xml does not exist2024-09-30 19:38:44,013 INFO  [1] oracle.glcm.opatch.common.impl.SingletonPatch - Patch Deploy artifact does not exist2024-09-30 19:38:44,013 INFO  [1] oracle.glcm.opatch.common.impl.SingletonPatch - /u01/RU23/36209493/36233126/36233263/etc/config/maintenance_alias.xml does not exist2024-09-30 19:38:45,137 INFO  [1] oracle.glcm.opatch.common.impl.SingletonPatch - /u01/RU23/36209493/36233126/36240578/etc/config/checksum.xml does not exist2024-09-30 19:38:45,138 INFO  [1] oracle.glcm.opatch.common.impl.SingletonPatch - /u01/RU23/36209493/36233126/36240578/etc/config/automation.xml does not exist2024-09-30 19:38:45,138 INFO  [1] oracle.glcm.opatch.common.impl.SingletonPatch - Patch Deploy artifact does not exist2024-09-30 19:38:45,138 INFO  [1] oracle.glcm.opatch.common.impl.SingletonPatch - /u01/RU23/36209493/36233126/36240578/etc/config/maintenance_alias.xml does not exist [root@jieke-rac2 RU23]# chown -R oracle:oinstall 36209493/
/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatchauto resume

放弃自动打补丁，我们可以尝试手动打补丁，如下，则是回家之后手动打补丁，没有出现任何报错。

jieke-rac2:/home/oracle(JiekeXu2)$ cd /home/oracle/36209493/36233126/36240578/custom/scriptsjieke-rac2:/u01/RU23/36209493/36233126/36240578/custom/scripts(JiekeXu2)$ ./prepatch.sh -dbhome /u01/app/oracle/product/19.0.0.0/dbhome_1
jieke-rac2:/u01/RU23/36209493/36233126(JiekeXu2)$ opatch apply -oh /u01/app/oracle/product/19.0.0.0/dbhome_1 /home/oracle/36209493/36233126/36233263jieke-rac2:/u01/RU23/36209493/36233126(JiekeXu2)$ opatch apply -oh /u01/app/oracle/product/19.0.0.0/dbhome_1 /home/oracle/36209493/36233126/36240578
jieke-rac2:/u01/RU23/36209493/36233126(JiekeXu2)$ cd /home/oracle/36209493/36233126/36240578/custom/scripts/jieke-rac2:/u01/RU23/36209493/36233126/36240578/custom/scripts(JiekeXu2)$ ./postpatch.sh -dbhome /u01/app/oracle/product/19.0.0.0/dbhome_1

如果要打 OJVM 的话，关库直接 opatch apply

jieke-rac2:/home/oracle(JiekeXu2)$ cd /home/oracle/36209493/36199232jieke-rac2:/home/oracle/36209493/36199232(JiekeXu2)$ opatch apply

开库，验证补丁

jieke-rac2:/home/oracle/36209493/36199232(JiekeXu2)$ opatch lspatches36199232;OJVM RELEASE UPDATE: 19.23.0.0.240416 (36199232)36240578;OCW RELEASE UPDATE 19.23.0.0.0 (36240578)36233263;Database Release Update : 19.23.0.0.240416 (36233263)
OPatch succeeded.
jieke-rac2:/home/grid(+ASM2)$ opatch lspatches36460248;TOMCAT RELEASE UPDATE 19.0.0.0.0 (36460248)36383196;DBWLM RELEASE UPDATE 19.0.0.0.0 (36383196)36240578;OCW RELEASE UPDATE 19.23.0.0.0 (36240578)36233343;ACFS RELEASE UPDATE 19.23.0.0.0 (36233343)36233263;Database Release Update : 19.23.0.0.240416 (36233263)
OPatch succeeded.

执行 datapatch 编译无效对象

因为是手动打的补丁，datapatch 也没有自动执行，所以我们要开库手动执行，注意了，这一步执行时间过长，长的让你怀疑人生。

cd $ORACLE_HOME/OPatch./datapatch -sanity_checks./datapatch -verbose

如下是日志，可以忽略不看。sanity_checks 就花了很长时间，verbose 直接导致窗口超时断开了，注意看此时已经是国庆 Oct 1 01:24:43 2024

jieke-rac1:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch(JiekeXu1)$ ./datapatch -sanity_checksSQL Patching sanity checks version 19.23.0.0.0 on Tue Oct  1 01:24:43 2024Copyright (c) 2021, 2024, Oracle.  All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sanity_checks_20241001_012443_75210/sanity_checks_20241001_012443_75210.log
Running checksUse of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.
Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 942.Use of uninitialized value $pdb in hash element at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 946.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.Use of uninitialized value $pdb in concatenation (.) or string at /u01/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 932.
##################################################################                        Welcome to LINUX                       ##                                                               ##   All  connections  are  monitored  and  recorded.            ##   Disconnect IMMEDIATELY if you are  not an authorized user!  ####################################################################################################################################                        Welcome to LINUX                       ##                                                               ##   All  connections  are  monitored  and  recorded.            ##   Disconnect IMMEDIATELY if you are  not an authorized user!  ####################################################################################################################################                        Welcome to LINUX                       ##                                                               ##   All  connections  are  monitored  and  recorded.            ##   Disconnect IMMEDIATELY if you are  not an authorized user!  ##################################################################JSON report generated in /u01/app/oracle/cfgtoollogs/sqlpatch/sanity_checks_20241001_012443_75210/sqlpatch_sanity_checks_summary.json fileChecks completed. Printing report:
Check: Database component status - OKCheck: PDB Violations - OKCheck: Invalid System Objects - OKCheck: Tablespace Status - OKCheck: Backup jobs - OKCheck: Temp file exists - OKCheck: Temp file online - OKCheck: Data Pump running - OKCheck: Container status - OKCheck: Oracle Database Keystore - OKCheck: Dictionary statistics gathering - OKCheck: Scheduled Jobs - WARNING  Execution of scheduler jobs while database patching is running may lead to failures and/or performance issues.  There are jobs currently running or scheduled to be executed during next hour.  Consider patching the database when jobs are not running and will not be scheduled to run during patching.  To check for jobs that are running or scheduled to run:    SELECT owner as schema_name, job_name, state, next_run_date    FROM sys.all_scheduler_jobs    WHERE state = 'RUNNING'    UNION      SELECT owner as schema_name, job_name, state, next_run_date      FROM sys.all_scheduler_jobs      WHERE state = 'SCHEDULED'      and cast(next_run_date as date) > sysdate      and cast(next_run_date as date) < sysdate + 1/24;  :    |          JOB_NAME           |            NEXT_RUN_DATE            | SCHEMA_NAME |   STATE   |    |-----------------------------+-------------------------------------+-------------+-----------|    |       DBMS_JOB$_4002        | 01-OCT-24 02.01.27.000000 AM +08:00 | APEX_030200 | SCHEDULED |    |-----------------------------+-------------------------------------+-------------+-----------|    |         DBMS_JOB$_4         | 01-OCT-24 01.52.26.000000 AM +08:00 |     DWD     | SCHEDULED |    |-----------------------------+-------------------------------------+-------------+-----------|    |        DBMS_JOB$_63         | 01-OCT-24 01.19.53.000000 AM +08:00 |    ODS_R    |  RUNNING  |    |-----------------------------+-------------------------------------+-------------+-----------|    |  CLEANUP_ONLINE_IND_BUILD   | 01-OCT-24 02.14.00.781550 AM +08:00 |     SYS     | SCHEDULED |    |-----------------------------+-------------------------------------+-------------+-----------|    |     CLEANUP_ONLINE_PMO      | 01-OCT-24 02.18.57.263366 AM +08:00 |     SYS     | SCHEDULED |    |-----------------------------+-------------------------------------+-------------+-----------|    |     CLEANUP_TAB_IOT_PMO     | 01-OCT-24 02.14.10.081980 AM +08:00 |     SYS     | SCHEDULED |    |-----------------------------+-------------------------------------+-------------+-----------|    |    CLEANUP_TRANSIENT_PKG    | 01-OCT-24 02.18.47.000000 AM +08:00 |     SYS     | SCHEDULED |    |-----------------------------+-------------------------------------+-------------+-----------|    | PMO_DEFERRED_GIDX_MAINT_JOB |  01-OCT-24 02.00.00.989190 AM PRC   |     SYS     | SCHEDULED |    |-----------------------------+-------------------------------------+-------------+-----------|Check: GoldenGate triggers - OKCheck: Logminer DDL triggers - OKCheck: Check sys public grants - OKCheck: Statistics gathering running - OKCheck: Optim dictionary upgrade parameter - OKCheck: Symlinks on oracle home path - OKCheck: Central Inventory - OKCheck: Queryable Inventory dba directories - ERROR  Datapatch uses Queryable Patch Inventory to query for Patch information from within the database. This feature uses Oracle directory objects to reference physical directories on the server. Hence, these directory objects and paths must exist and be defined correctly.  There are inconsistencies with the Queryable Inventory directories.  Verify that the directories used by Queryable Patch Inventory are defined correctly and are relative to the ORACLE HOME. Check both physical paths and directory objects; and create or correct as needed. Refer to MOS NOTE 1602089.1 for more details.  jieke-rac2:    Could not determine if directory OPATCH_SCRIPT_DIR /u01/app/oracle/product/19.0.0.0/dbhome_1/QOpatch exists.    Could not determine if directory OPATCH_LOG_DIR /u01/app/oracle/product/19.0.0.0/dbhome_1/rdbms/log exists.    Could not determine if directory OPATCH_INST_DIR /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch exists.Check: Queryable Inventory locks - SKIPPED (ERROR)  Message: Skipped as previous check wasn't executed successfully.Check: Queryable Inventory package - SKIPPED (ERROR)  Message: Skipped as previous check wasn't executed successfully.Check: Queryable Inventory external table - SKIPPED (ERROR)  Message: Skipped as previous check wasn't executed successfully.Check: Imperva processes - OKCheck: Guardium processes - OKCheck: Locale - OK
Refer to MOS Note 2680521.1 and debug log/u01/app/oracle/cfgtoollogs/sqlpatch/sanity_checks_20241001_012443_75210/sanity_checks_debug_20241001_012443_75210.log
SQL Patching sanity checks completed on Tue Oct  1 01:53:16 2024jieke-rac1:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch(JiekeXu1)$ 
./datapatch -verboseSQL Patching tool version 19.23.0.0.0 Production on Tue Oct  1 01:55:13 2024Copyright (c) 2012, 2024, Oracle.  All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_97060_2024_10_01_01_55_13/sqlpatch_invocation.log
Connecting to database...OKGathering database info...doneBootstrapping registry and package to current versions...doneDetermining current state...done
Current state of interim SQL patches:Interim patch 29774421 (OJVM RELEASE UPDATE: 19.4.0.0.190716 (29774421)):  Binary registry: Not installed  SQL registry: Rolled back successfully on 23-FEB-23 12.44.59.058809 AMInterim patch 33808367 (OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367)):  Binary registry: Not installed  SQL registry: Applied successfully on 23-FEB-23 12.44.59.069589 AMInterim patch 36199232 (OJVM RELEASE UPDATE: 19.23.0.0.240416 (36199232)):  Binary registry: Installed  SQL registry: Not installed
Current state of release update SQL patches:  Binary registry:    19.23.0.0.0 Release_Update 240406004238: Installed  SQL registry:    Applied 19.15.0.0.0 Release_Update 220331125408 successfully on 23-FEB-23 12.13.27.404863 AM
Adding patches to installation queue and performing prereq checks...doneInstallation queue:  The following interim patches will be rolled back:    33808367 (OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367))  Patch 36233263 (Database Release Update : 19.23.0.0.240416 (36233263)):    Apply from 19.15.0.0.0 Release_Update 220331125408 to 19.23.0.0.0 Release_Update 240406004238  The following interim patches will be applied:    36199232 (OJVM RELEASE UPDATE: 19.23.0.0.240416 (36199232))

由于会话中断，第二天下午接着执行,14:43:07 开始执行，16:14:16 结束，大约花费 90 分钟，真是有点出乎意料，也是一波三折，所以才想着去记录这个曲折的过程。

jieke-rac1:/home/oracle(JiekeXu1)$ cd /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatchjieke-rac1:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch(JiekeXu1)$ jieke-rac1:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch(JiekeXu1)$ ./datapatch -verboseSQL Patching tool version 19.23.0.0.0 Production on Tue Oct  1 14:43:07 2024Copyright (c) 2012, 2024, Oracle.  All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_17115_2024_10_01_14_43_07/sqlpatch_invocation.log
Connecting to database...OKGathering database info...Unable to acquire sqlpatch global lock in EXCLUSIVE mode because another datapatch session is currently running.Waiting for that session to complete before continuing...doneBootstrapping registry and package to current versions...done


Determining current state...done
Current state of interim SQL patches:Interim patch 29774421 (OJVM RELEASE UPDATE: 19.4.0.0.190716 (29774421)):  Binary registry: Not installed  SQL registry: Rolled back successfully on 23-FEB-23 12.44.59.058809 AMInterim patch 33808367 (OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367)):  Binary registry: Not installed  SQL registry: Applied successfully on 23-FEB-23 12.44.59.069589 AMInterim patch 36199232 (OJVM RELEASE UPDATE: 19.23.0.0.240416 (36199232)):  Binary registry: Installed  SQL registry: Not installed
Current state of release update SQL patches:  Binary registry:    19.23.0.0.0 Release_Update 240406004238: Installed  SQL registry:    Applied 19.15.0.0.0 Release_Update 220331125408 successfully on 23-FEB-23 12.13.27.404863 AM
Adding patches to installation queue and performing prereq checks...doneInstallation queue:  The following interim patches will be rolled back:    33808367 (OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367))  Patch 36233263 (Database Release Update : 19.23.0.0.240416 (36233263)):    Apply from 19.15.0.0.0 Release_Update 220331125408 to 19.23.0.0.0 Release_Update 240406004238  The following interim patches will be applied:    36199232 (OJVM RELEASE UPDATE: 19.23.0.0.240416 (36199232))
Installing patches...Patch installation complete.  Total patches installed: 3
Validating logfiles...donePatch 33808367 rollback: SUCCESS  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/33808367/24680225/33808367_rollback_JiekeXu_2024Oct01_14_59_37.log (no errors)Patch 36233263 apply: SUCCESS  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/36233263/25638263/36233263_apply_JiekeXu_2024Oct01_14_59_40.log (no errors)Patch 36199232 apply: SUCCESS  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/36199232/25601966/36199232_apply_JiekeXu_2024Oct01_14_59_39.log (no errors)SQL Patching tool complete on Tue Oct  1 16:14:16 2024

编译无效对象

export PATH=$PATH:$ORACLE_HOME/bincd $ORACLE_HOME/rdbms/admin$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catcon.pl -n 1 -e -b utlrp -d $ORACLE_HOME/rdbms/admin utlrp.sql

SQL 查看补丁信息

SYS@JiekeXu1> col ACTION_TIME  for a30SYS@JiekeXu1> col DESCRIPTION for a58SYS@JiekeXu1> select patch_id,action,status,action_time,description from dba_registry_sqlpatch;
  PATCH_ID ACTION          STATUS                    ACTION_TIME                    DESCRIPTION---------- --------------- ------------------------- ------------------------------ ----------------------------------------------------------  29774421 APPLY           SUCCESS                   07-MAR-20 10.46.17.632365 PM   OJVM RELEASE UPDATE: 19.4.0.0.190716 (29774421)  29834717 APPLY           SUCCESS                   07-MAR-20 10.46.17.626228 PM   Database Release Update : 19.4.0.0.190716 (29834717)  29774421 ROLLBACK        WITH ERRORS               23-FEB-23 12.13.14.147500 AM   OJVM RELEASE UPDATE: 19.4.0.0.190716 (29774421)  33806152 APPLY           SUCCESS                   23-FEB-23 12.13.27.404863 AM   Database Release Update : 19.15.0.0.220419 (33806152)  29774421 ROLLBACK        SUCCESS                   23-FEB-23 12.44.59.058809 AM   OJVM RELEASE UPDATE: 19.4.0.0.190716 (29774421)  33808367 APPLY           SUCCESS                   23-FEB-23 12.44.59.069589 AM   OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367)  33808367 ROLLBACK        SUCCESS                   01-OCT-24 02.59.39.844404 PM   OJVM RELEASE UPDATE: 19.15.0.0.220419 (33808367)  36199232 APPLY           SUCCESS                   01-OCT-24 02.59.40.071968 PM   OJVM RELEASE UPDATE: 19.23.0.0.240416 (36199232)  36233263 APPLY           SUCCESS                   01-OCT-24 04.09.51.103043 PM   Database Release Update : 19.23.0.0.240416 (36233263)
9 rows selected.

总结

本环境虽是一套测试环境，但机器负载一直很高，几乎 CPU 在 90% 以上，打补丁时又没有停止应用、也没有先停止 OGG 以及 crontab、mtail 等，可能在自动打补丁时出现一些不曾遇到的错误,所以才会一波三折，当然也比较有趣，一天中遇到了多个错误，也是值得学习与记录的，加油，国庆快乐！加油，少年！

参考链接

https://forums.oracle.com/ords/apexds/post/opatchauto-72132-grid-is-not-running-on-the-local-host-1094
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=565017.1

全文完，希望可以帮到正在阅读的你，如果觉得有帮助，可以分享给你身边的朋友，同事，你关心谁就分享给谁，一起学习共同进步~~~

❤️ 欢迎关注我的公众号【JiekeXu DBA之路】，一起学习新知识！

分享几个数据库备份脚本

一文搞懂 Oracle 统计信息

我的 Oracle ACE 心路历程
MOP 系列|MOP 三种主流数据库索引简介
Oracle 主流版本不同架构下的静默安装指南
关机重启导致 ASM 磁盘丢失数据库无法启动
Oracle SQL 性能分析(SPA)原理与实战演练

Oracle 11g 升级到 19c 需要关注的几个问题
Windows 10 环境下 MySQL 8.0.33 安装指南
SQL 大全（四）|数据库迁移升级时常用 SQL 语句
OGG|使用 OGG19c 迁移 Oracle11g 到 19C(第二版)
Oracle 大数据量导出工具——sqluldr2 的安装与使用
从国产数据库调研报告中你都能了解哪些信息及我的总结建议
使用数据泵利用 rowid 分片导出导入 lob 大表及最佳实践
在归档模式下直接 rm dbf 数据文件并重启数据库还有救吗？

欢迎关注我的公众号【JiekeXu DBA之路】，一起学习新知识！
——————————————————————————
公众号：JiekeXu DBA之路
墨天轮：https://www.modb.pro/u/4347
CSDN ：https://blog.csdn.net/JiekeXu
ITPUB：https://blog.itpub.net/69968215
腾讯云：https://cloud.tencent.com/developer/user/5645107
——————————————————————————

JiekeXu DBA之路

JiekeXu:Oracle ACE-Pro，获 Oracle OCP/OCM 及 MySQL OCP 认证，墨天轮 MVP，利用闲时间记录菜鸟 DBA 学习成长之路，所发布文字属于个人观点和学习笔记，如有错误及不当之处，敬请批评指正！