Linux文件系统异常导致数据库文件无法访问
本站文章除注明转载外，均为本站原创： 转载自love wife love life —Roger的Oracle/MySQL/PostgreSQL数据恢复博客
本文链接地址: Linux文件系统异常导致数据库文件无法访问
51节前某政府单位客户数据库出现异常；大致现象是：数据库状态是open的；但是其中一个数据文件无法访问；通过ls 查看文件都报错，如下所示：
|
1 2 |
[oracle@oracledata01 oracle]$ ls ls: 无法访问zf4.dbf: 输入/输出错误 |
同时dd命令也无法读取该文件：
|
1 2 3 |
[oracle@oracledata01 fd]$ dd if=zf4.dbf bs=8192 count=1 skip=1|od -x|header -1 bash: header: command not found dd: 正在打开"zf4.dbf": 没有那个文件或目录 |
此时在从数据库层面看到的信息类似如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
Tue Apr 28 16:49:51 2020 Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_9146.trc: ORA-01157: 无法标识/锁定数据文件 11 - 请参阅 DBWR 跟踪文件 ORA-01110: 数据文件 11: '/oracle_data/oracle/zf4.dbf' ORA-1157 signalled during: ALTER DATABASE OPEN... Tue Apr 28 16:49:51 2020 Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m000_9152.trc: ORA-27037: 无法获得文件状态 Linux-x86_64 Error: 5: Input/output error Additional information: 3 ORA-01122: 数据库文件 11 验证失败 ORA-01110: 数据文件 11: '/oracle_data/oracle/zf4.dbf' ORA-01565: 标识文件 '/oracle_data/oracle/zf4.dbf' 时出错 ORA-27037: 无法获得文件状态 Linux-x86_64 Error: 5: Input/output error Additional information: 3 Tue Apr 28 16:50:19 2020 USER (ospid: 9309): terminating the instance |
从上述信息来看，数据库第11号文件无法读取；其中在我们进行恢复之前，客户进行了多次尝试，还遭遇了一些数据库坏块，也遇到了不少ora-00600 错误：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 |
Wed Apr 29 03:10:12 2020 Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_j009_65816.trc: ORA-12012: 自动执行作业 "ZX_XT"."JS_XT_XZGLXX_XZXK_FR" 出错 ORA-06502: PL/SQL: 数字或值错误 : 字符串缓冲区太小 ORA-06512: 在 "ZX_XT.PKG_双公示接收_邢_2019", line 1071 ORA-00942: 表或视图不存在 ORA-02063: 紧接着 line (起自 BM_SJJH) alter database datafile 11 resize 20g Completed: alter database datafile 11 resize 20g Wed Apr 29 03:10:15 2020 Corrupt Block Found CONT = 0, TSN = 1, TSNAME = SYSAUX RFN = 3, BLK = 3, RDBA = 12582915 OBJN = 1, OBJD = -1, OBJECT = _NEXT_OBJECT, SUBOBJECT = SEGMENT OWNER = SYS, SEGMENT TYPE = Invalid Type Wed Apr 29 03:10:15 2020 Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m006_66106.trc (incident=246250): ORA-00600: 内部错误代码, 参数: [kdBlkCheckError], [3], [3], [18018], [], [], [], [], [], [], [], [] Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_246250/orcl_m006_66106_i246250.trc Wed Apr 29 03:10:15 2020 ...... Wed Apr 29 03:11:00 2020 Doing block recovery for file 3 block 3 Resuming block recovery (PMON) for file 3 block 3 Block recovery from logseq 50078, block 15827 to scn 14514058257813 Wed Apr 29 03:11:00 2020 Recovery of Online Redo Log: Thread 1 Group 3 Seq 50078 Reading mem 0 Mem# 0: /oracle_data/oradata/orcl/redo03.log Block recovery completed at rba 50078.23957.16, scn 3379.1363764630 Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m004_66271.trc (incident=245043): ORA-00600: 内部错误代码, 参数: [kdBlkCheckError], [3], [3], [18018], [], [], [], [], [], [], [], [] Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_245043/orcl_m004_66271_i245043.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Apr 29 03:11:01 2020 Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m004_66271.trc: ORA-00600: 内部错误代码, 参数: [kdBlkCheckError], [3], [3], [18018], [], [], [], [], [], [], [], [] Wed Apr 29 03:11:02 2020 ...... Wed Apr 29 03:15:49 2020 Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_j000_65779.trc (incident=245026): ORA-00600: internal error code, arguments: [13011], [7907], [12713924], [44], [12687314], [0], [], [], [], [], [], [] ORA-01403: no data found ORA-06512: at "SYS.DBMS_AQ_INV", line 1248 ORA-06512: at line 1 Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_245026/orcl_j000_65779_i245026.trc Wed Apr 29 03:15:52 2020 Dumping diagnostic data in directory=[cdmp_20200429031552], requested by (instance=1, osid=65779 (J000)), summary=[incident=245026]. Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_j000_65779.trc (incident=245027): ORA-00600: internal error code, arguments: [13011], [7907], [12713924], [44], [12687314], [0], [], [], [], [], [], [] ORA-01403: no data found ORA-06512: at "SYS.DBMS_AQ_INV", line 1248 ORA-06512: at line 1 Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_245027/orcl_j000_65779_i245027.trc Use ADRCI or Support Workbench to package the incident. ...... Wed Apr 29 04:05:07 2020 Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m003_76294.trc: ORA-00600: 内部错误代码, 参数: [kdBlkCheckError], [3], [3], [18018], [], [], [], [], [], [], [], [] Wed Apr 29 04:05:08 2020 Dumping diagnostic data in directory=[cdmp_20200429040508], requested by (instance=1, osid=76263), summary=[incident=282250]. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_76263.trc (incident=282251): ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_282251/orcl_ora_76263_i282251.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Apr 29 04:05:10 2020 Sweep [inc][282251]: completed Sweep [inc2][282265]: completed Sweep [inc2][282251]: completed Wed Apr 29 04:05:10 2020 Dumping diagnostic data in directory=[cdmp_20200429040510], requested by (instance=1, osid=76263), summary=[incident=282251]. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_76263.trc (incident=282252): ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_282252/orcl_ora_76263_i282252.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Apr 29 04:07:26 2020 ...... Wed Apr 29 04:24:01 2020 Sweep [inc][279857]: completed Sweep [inc2][279857]: completed Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_78503.trc (incident=279858): ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_279858/orcl_ora_78503_i279858.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Apr 29 04:24:03 2020 Sweep [inc][279858]: completed Sweep [inc2][279858]: completed Wed Apr 29 04:24:04 2020 Dumping diagnostic data in directory=[cdmp_20200429042404], requested by (instance=1, osid=78503), summary=[incident=279858]. Wed Apr 29 04:25:57 2020 Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_78738.trc (incident=282222): ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_282222/orcl_ora_78738_i282222.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Apr 29 04:25:58 2020 Sweep [inc][282222]: completed Sweep [inc2][282222]: completed Wed Apr 29 04:25:58 2020 Dumping diagnostic data in directory=[cdmp_20200429042558], requested by (instance=1, osid=78738), summary=[incident=282222]. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_78738.trc (incident=282223): ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_282223/orcl_ora_78738_i282223.trc |
上述ora-00600 错误，总的来都是相对常见的；几乎都可以定性为数据坏块；对于file 3的坏块，处理相对简单，直接drop或者truncate相关对象即可；然后重建awr。
但是对于第11号数据文件，怎么办呢？这里我们的处理方法是就当文件丢失处理（很久之前处理过Windows环境数据文件大小为 0 kb的问题）；这几种情况都类似。
这里我说一下简单的处理思路：
1、通过odu 扫盘；分别以不同offset方式来扫（0和4096）；
2、对比两次扫描结果；组合extent信息；抽取文件。
由于这里文件不多，只有1个；大小20g左右；通过dd 拼接即可。
最终文件拼接完成后，由于这里是非归档环境，无法进行正常recover，因此还需要通过bbed来修改数据文件头的checkpoint信息。
对于文件系统来讲，目前已经不是Oracle存储的最佳选择了，建议使用ASM。当然；其他数据库比如MySQL、PostgreSQL除外。
题外话：昨天遇到某客户通过400电话求助，PostgreSQL 被加密勒索（大概是每个文件前面2Mb左右）；这种处理是比较麻烦的；目前还没有发现很好的扫盘或者扫描文件的工具。后面有时间再研究一下。
Leave a Reply
You must be logged in to post a comment.