linux loopback interface 导致Oracle RAC实例crash
本站文章除注明转载外,均为本站原创: 转载自love wife love life —Roger的Oracle/MySQL/PostgreSQL数据恢复博客
很久没有写Oracle方面的技术文章了,今天同事反馈某客户的一体机其中一个计算节点报错,咨询是否需要重启网络服务;
因为担心重启整个网络服务之后,该计算节点无法启动,接下来我们看看是什么报错:
1 2 3 4 5 6 |
ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:sendmsg failed with status: 105 ORA-27301: OS failure message: No buffer space available ORA-27302: failure occurred at: sskgxpsnd2 kgefec: fatal error 0 |
该环境为3节点Oracle Rac集群,存储是zdata分布式架构。其实上述错误非常简单,而且十分常见,这里不做过多解释。
Oracle Mos也提供了不少参考文档:
Troubleshooting ORA-27300 ORA-27301 ORA-27302 Errors (Doc ID 579365.1)
Oracle Linux: ORA-27301:OS Failure Message: No Buffer Space Available (Doc ID 2041723.1)
With NUMA Enabled, Database Fails To Open With ORA-600[ksbmoveme4], ORA-27300, ORA-27301, ORA-27302 Errors Reported
对于这里这个问题而言,结合相关文档和实际分析来看,
主要有2方面点需要关注:
1、其他2个计算节点关闭了NUMA,而报错这个计算节点没有关闭NUMA
2、3个节点lo 网卡MTU 设置均为64k。
从现场同时反馈来看,报错所在节点numa 为4,因此对于操作系统对于vm.min_free_kbytes = n * 0.4% 的要求更高;其中n为numa node数量。由于操作系统内存为1TB,因此这个参数设置就需要比较大了。
其次从Oracle MOS文档来看,在Linux 7+版本中,默认把lo MTU从16k改到了64k,这会导致一些问题。
因此必须将该值调整为16436. 实际上通过ifconfig lo mtu 16436调整后即可生效,同时修改配置文件即可。不需要重启整个网络服务。
搜索了一下相关文档,发现类似的有不少问题,这里简单记录一下这个问题。非常容易忽视。
其他参考:
BUG:10010310 – ORA-27300: INVALID_PROCESS_ID FAILED WITH STATUS: 0; ORA-27302: SKGPALIVE1
NOTE:6441119.8 – Bug 6441119 – Instance crash due to ORA-27300 / ORA-27152
NOTE:580552.1 – Database Crashes With ORA-04030 ORA-07445 ORA-27300 ORA-27301 ORA-27302
NOTE:1229767.1 – ORA-27300 OS system dependent operation fork failed with status 11, ORA-27301 and ORA-27302 After Increasing Processes Parameter
NOTE:557153.1 – ORA-27370 ORA-27301 (Permission denied) When Running Job Of Type EXECUTABLE
BUG:3411021 – LMD PROCESS CRASH DUE TO ORA-27300/ORA-27301/ORA-27302 AT SKGXPPOST1
NOTE:848387.1 – With NUMA Enabled, Database Fails To Open With ORA-600[ksbmoveme4], ORA-27300, ORA-27301, ORA-27302 Errors Reported
NOTE:300956.1 – Ora-27302: sskgxpsnd1 – Starting Instance
NOTE:466370.1 – ORA-7445 [ACCESS_VIOLATION] [unable_to_trans_pc] [UNABLE_TO_WRITE] ORA-27301: OS failure message: Not enough storage ORA-27300 ORA-27302
NOTE:438205.1 – ORA-27300 ORA-27301 ORA-27302 ORA-27157 Database Crash
NOTE:453959.1 – Cannot Connect As “/ as sysdba” ORA-27140 ORA-27300 ORA-27301 ORA-27302 And ORA-27303 In Trace File
NOTE:949468.1 – Database Startup Fails with ORA-27300: OS system dependent operation:semget failed with status: 28
NOTE:465002.1 – Database Crash With Error ORA-00490
NOTE:6629265.8 – Bug 6629265 – Intermittent ORA-27504 / ORA-27300 ioctl error 11
NOTE:314179.1 – Instance Startup Fails With Error ORA-27154,ORA-27300,ORA-27301,ORA-27302
NOTE:392006.1 – ORA-27300 OS system dependent operation:fork failed with status: 11
NOTE:364353.1 – ORA-00603 ORA-27504 ORA-27300 ORA-27504 in the Alert Log
BUG:7620133 – HIGH NUMBER OF DBW* PROCESSES SPAWNED IN 11.1.0.7
NOTE:458442.1 – 10.2.0.3 PMON Crashes on Startup on AIX 5L 5.3 ML05 — Works on ML06
NOTE:297498.1 – Resolving Instance Evictions on Windows Platforms
NOTE:277399.1 – DBMS_SCHEDULER Fails To Execute Jobs With Program_type EXECUTABLE On HP-UX
BUG:7232946 – ORA-600[KSKRECONFIGNUMA2] CAUSES INSTANCE CRASH
BUG:6991131 – ORA-27300, ORA-27301 AND ORA-27302 HAPPENED DURING SHUTDOWN USING SRVCTL
NOTE:225349.1 – Implementing Address Windowing Extensions (AWE) or VLM on 32-bit Windows Platforms
NOTE:371074.1 – ORA-27300 ORA-27301 ORA-27302 in alert log. Cannot connect to database.
NOTE:1038058.6 – LIST OF UNIX ERRORS AND EXPLANATIONS
NOTE:1274030.1 – Startup Instance Failed with ORA-27140 ORA-27300 ORA-27301 ORA-27302 and ORA-27303 on skgpwinit6
BUG:9059053 – ORA-00603 ORA-27504 ORA-27300 ORA-27301 ORA-27302 IN ALERT.LOG
NOTE:356640.1 – ORA-27300, ORA-27301, ORA-27302: Failure Occurred At: Skgpalive1 During ‘Shutdown’
NOTE:560309.1 – Database Cannot Start Due to Lack of Memory
NOTE:1252265.1 – ORA-27300 ORA-27301 ORA-27302: failure occurred at: skgpalive1
NOTE:3411021.8 – Bug 3411021 – Oracle process may die due to ORA-27300/ORA-27301/ORA-27302 at skgxppost1
NOTE:746888.1 – ORA-27302: Failure Occurred at: skgxpvfymmtu Signalled in the Alert.log
NOTE:295832.1 – This document has been replaced with more recent information on this topic. Please refer to more recent documentation.
NOTE:2171341.1 – ORA-27302: skgxpmap1: ORA-27303: maximum(512) number of regions/context exceeded 512
Leave a Reply
You must be logged in to post a comment.