ASM 加盘导致数据库无法Open的案例（add disk很危险）

本站文章除注明转载外，均为本站原创： 转载自love wife love life —Roger的Oracle/MySQL/PostgreSQL数据恢复博客

本文链接地址: ASM 加盘导致数据库无法Open的案例（add disk很危险）

周5我司某客户在进行存储扩容时；当add disk完成后，发现数据库很快crash。

Fri Apr 10 00:32:17 2020
SQL> alter diskgroup DATA_NEW add disk '/dev/asm_lun06_rac_alis' 
NOTE: GroupBlock outside rolling migration privileged region
NOTE: Assigning number (2,2) to disk (/dev/asm_lun06_rac_alis)
NOTE: requesting all-instance membership refresh for group=2
NOTE: initializing header on grp 2 disk DATA_NEW_0002
NOTE: requesting all-instance disk validation for group=2
Thu Apr 09 08:32:18 2020
NOTE: skipping rediscovery for group 2/0x2c084b53 (DATA_NEW) on local instance.
NOTE: requesting all-instance disk validation for group=2
NOTE: skipping rediscovery for group 2/0x2c084b53 (DATA_NEW) on local instance.
NOTE: initiating PST update: grp = 2
Thu Apr 09 08:32:18 2020
GMON updating group 2 at 9 for pid 27, osid 14325
NOTE: PST update grp = 2 completed successfully 
NOTE: membership refresh pending for group 2/0x2c084b53 (DATA_NEW)
GMON querying group 2 at 10 for pid 18, osid 10740
NOTE: cache opening disk 2 of grp 2: DATA_NEW_0002 path:/dev/asm_lun06_rac_alis
GMON querying group 2 at 11 for pid 18, osid 10740
SUCCESS: refreshed membership for 2/0x2c084b53 (DATA_NEW)
NOTE: starting rebalance of group 2/0x2c084b53 (DATA_NEW) at power 1
SUCCESS: alter diskgroup DATA_NEW add disk '/dev/asm_lun06_rac_alis'
Starting background process ARB0
Thu Apr 09 08:32:21 2020
ARB0 started with pid=31, OS id=14725 
NOTE: assigning ARB0 to group 2/0x2c084b53 (DATA_NEW) with 1 parallel I/O
WARNING: cache read  a corrupt block: group=2(DATA_NEW) dsk=0 blk=2 disk=0 (DATA_NEW_0000) incarn=3915955114 au=0 blk=2 count=1
Errors in file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_14725.trc:
ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]
NOTE: a corrupted block from group DATA_NEW was dumped to /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_14725.trc
WARNING: cache read (retry) a corrupt block: group=2(DATA_NEW) dsk=0 blk=2 disk=0 (DATA_NEW_0000) incarn=3915955114 au=0 blk=2 count=1
Errors in file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_14725.trc:
ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]
ERROR: cache failed to read group=2(DATA_NEW) dsk=0 blk=2 from disk(s): 0(DATA_NEW_0000)
ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]
NOTE: cache initiating offline of disk 0 group DATA_NEW

Fri Apr 10 00:32:17 2020

SQL> alter diskgroup DATA_NEW add disk '/dev/asm_lun06_rac_alis'

NOTE: GroupBlock outside rolling migration privileged region

NOTE: Assigning number (2,2) to disk (/dev/asm_lun06_rac_alis)

NOTE: requesting all-instance membership refresh for group=2

NOTE: initializing header on grp 2 disk DATA_NEW_0002

NOTE: requesting all-instance disk validation for group=2

Thu Apr 09 08:32:18 2020

NOTE: skipping rediscovery for group 2/0x2c084b53 (DATA_NEW) on local instance.

NOTE: requesting all-instance disk validation for group=2

NOTE: skipping rediscovery for group 2/0x2c084b53 (DATA_NEW) on local instance.

NOTE: initiating PST update: grp = 2

Thu Apr 09 08:32:18 2020

GMON updating group 2 at 9 for pid 27, osid 14325

NOTE: PST update grp = 2 completed successfully

NOTE: membership refresh pending for group 2/0x2c084b53 (DATA_NEW)

GMON querying group 2 at 10 for pid 18, osid 10740

NOTE: cache opening disk 2 of grp 2: DATA_NEW_0002 path:/dev/asm_lun06_rac_alis

GMON querying group 2 at 11 for pid 18, osid 10740

SUCCESS: refreshed membership for 2/0x2c084b53 (DATA_NEW)

NOTE: starting rebalance of group 2/0x2c084b53 (DATA_NEW) at power 1

SUCCESS: alter diskgroup DATA_NEW add disk '/dev/asm_lun06_rac_alis'

Starting background process ARB0

Thu Apr 09 08:32:21 2020

ARB0 started with pid=31, OS id=14725

NOTE: assigning ARB0 to group 2/0x2c084b53 (DATA_NEW) with 1 parallel I/O

WARNING: cache read a corrupt block: group=2(DATA_NEW) dsk=0 blk=2 disk=0 (DATA_NEW_0000) incarn=3915955114 au=0 blk=2 count=1

Errors in file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_14725.trc:

ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]

NOTE: a corrupted block from group DATA_NEW was dumped to /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_14725.trc

WARNING: cache read (retry) a corrupt block: group=2(DATA_NEW) dsk=0 blk=2 disk=0 (DATA_NEW_0000) incarn=3915955114 au=0 blk=2 count=1

Errors in file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_14725.trc:

ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]

ERROR: cache failed to read group=2(DATA_NEW) dsk=0 blk=2 from disk(s): 0(DATA_NEW_0000)

ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]

NOTE: cache initiating offline of disk 0 group DATA_NEW

可以看add disk 是成功了，然后开始reblance时，无法进行，进而报错；导致diskgroup dismount。进一步分析arb进程的trace发现asm 元数据确实存在坏块：

*** 2020-04-09 08:32:21.593
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=0, mask=0x0)
----- Error Stack Dump -----
ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]
Hex dump of disk block image:
Dump of memory from 0x0000000062135000 to 0x0000000062136000
062135000 00000000 00000000 00000000 00000000  [................]
Repeat 255 times
OSM metadata block dump:
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfbtTraverseBlock:  Invalid OSM block type 0
WARNING: cache read (retry) a corrupt block: group=2(DATA_NEW) dsk=0 blk=2 disk=0 (DATA_NEW_0000) incarn=3915955114 au=0 blk=2 count=1

*** 2020-04-09 08:32:21.593

dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=0, mask=0x0)

----- Error Stack Dump -----

ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]

Hex dump of disk block image:

Dump of memory from 0x0000000062135000 to 0x0000000062136000

062135000 00000000 00000000 00000000 00000000 [................]

Repeat 255 times

OSM metadata block dump:

kfbh.endian: 0 ; 0x000: 0x00

kfbh.hard: 0 ; 0x001: 0x00

kfbh.type: 0 ; 0x002: KFBTYP_INVALID

kfbh.datfmt: 0 ; 0x003: 0x00

kfbh.block.blk: 0 ; 0x004: blk=0

kfbh.block.obj: 0 ; 0x008: file=0

kfbh.check: 0 ; 0x00c: 0x00000000

kfbh.fcn.base: 0 ; 0x010: 0x00000000

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

kfbh.spare2: 0 ; 0x01c: 0x00000000

kfbtTraverseBlock: Invalid OSM block type 0

WARNING: cache read (retry) a corrupt block: group=2(DATA_NEW) dsk=0 blk=2 disk=0 (DATA_NEW_0000) incarn=3915955114 au=0 blk=2 count=1

*** 2020-04-09 08:32:21.603
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=0, mask=0x0)
----- Error Stack Dump -----
ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]
ERROR: cache failed to read group=2(DATA_NEW) dsk=0 blk=2 from disk(s): 0(DATA_NEW_0000)
CE: (0x0x623c59c8) group=2 (DATA_NEW) dsk=0 blk=2
hashFlags=0x0000 lid=0x0002 lruFlags=0x0000 bastCount=1
mirror=0
flags_kfcpba=0x49 copies=1 blockIndex=2 AUindex=0 AUcount=1 loctr fcn=0.0
copy #0:  disk=0  au=0 flags=01
BH: (0x0x62394808) bnum=310 type=reading state=reading chgSt=not modifying pageIn=current
flags=0x00000000 pinmode=excl lockmode=excl bf=0x62135000
kfbh_kfcbh.fcn_kfbh = 0.0 lowAba=0.0 highAba=0.0
last kfcbInitSlot return code=null chgCount=0 cpkt lnk is null ralFlags=0x00000000
PINS:
(kfcbps) pin=2789 get by kfd.c line 7131 mode=excl
dsk=0 blk=2 status=pinned
flags=0x80000000 flags2=0x00000000
class=1400 type=ALLOCTBL stateWanted=current
bastCount=1 waitStatus=0x00000000 relocCount=0
scanBastCount=0 scanBxid=0 scanSkipCode=0
last released by kfc.c 18274
LE: (0x6b6e0768) le=308 group=2 dsk=0 blk=2
open=T kjStat=0 mode=EX closing=0 lop=(nil)
flags=00000000 astFlags=00000000 rlsFlags=00000000
rcvFlags=00000000 id=0xa000000.2 bucket=6346
lastScanWaiterMode=0 fcn=0.0

*** 2020-04-09 08:32:21.603

dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=0, mask=0x0)

----- Error Stack Dump -----

ORA-15196: invalid ASM block header [kfc.c:26086] [endian_kfbh] [2147483648] [2] [0 != 1]

ERROR: cache failed to read group=2(DATA_NEW) dsk=0 blk=2 from disk(s): 0(DATA_NEW_0000)

CE: (0x0x623c59c8) group=2 (DATA_NEW) dsk=0 blk=2

hashFlags=0x0000 lid=0x0002 lruFlags=0x0000 bastCount=1

mirror=0

flags_kfcpba=0x49 copies=1 blockIndex=2 AUindex=0 AUcount=1 loctr fcn=0.0

copy #0: disk=0 au=0 flags=01

BH: (0x0x62394808) bnum=310 type=reading state=reading chgSt=not modifying pageIn=current

flags=0x00000000 pinmode=excl lockmode=excl bf=0x62135000

kfbh_kfcbh.fcn_kfbh = 0.0 lowAba=0.0 highAba=0.0

last kfcbInitSlot return code=null chgCount=0 cpkt lnk is null ralFlags=0x00000000

PINS:

(kfcbps) pin=2789 get by kfd.c line 7131 mode=excl

dsk=0 blk=2 status=pinned

flags=0x80000000 flags2=0x00000000

class=1400 type=ALLOCTBL stateWanted=current

bastCount=1 waitStatus=0x00000000 relocCount=0

scanBastCount=0 scanBxid=0 scanSkipCode=0

last released by kfc.c 18274

LE: (0x6b6e0768) le=308 group=2 dsk=0 blk=2

open=T kjStat=0 mode=EX closing=0 lop=(nil)

flags=00000000 astFlags=00000000 rlsFlags=00000000

rcvFlags=00000000 id=0xa000000.2 bucket=6346

lastScanWaiterMode=0 fcn=0.0

如果数据库asm 元数据的朋友肯定都知道，AU 0除了disk header block之外，后面的block都是allocate table。如果不了解这一点，查看上面的trace 报错也能看出来，有明确的提示：这应该是一个allocate block；但是数据均为空。。。

当天凌晨恢复时没有去进行深入分析，开始以为只有个别block问题，尝试kfed merge去修改pst、disk directory、cod和acd等元数据，尝试把新加的盘踢掉；然后强制拉磁盘组；发现都不行；而且还遇到了一个新错误:

Thu Apr 09 17:37:08 2020
Dumping diagnostic data in directory=[cdmp_20200409173708], requested by (instance=2, osid=18443 (RBAL)), summary=[incident=72145].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ERROR: ORA-600 in COD recovery for diskgroup 2/0xda5d0748 (DATA_NEW)
ERROR: ORA-600 thrown in RBAL for group number 2
Errors in file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_18443.trc:
ORA-00600: internal error code, arguments: [kfrcRecover_4], [0], [], [], [], [], [], [], [], [], [], []
Errors in file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_18443.trc:
ORA-00600: internal error code, arguments: [kfrcRecover_4], [0], [], [], [], [], [], [], [], [], [], []
RBAL (ospid: 18443): terminating the instance due to error 488
System state dump requested by (instance=2, osid=18443 (RBAL)), summary=[abnormal instance termination].
System State dumped to trace file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM2/trace/+ASM2_diag_18417.trc
Instance terminated by RBAL, pid = 18443

Thu Apr 09 17:37:08 2020

Dumping diagnostic data in directory=[cdmp_20200409173708], requested by (instance=2, osid=18443 (RBAL)), summary=[incident=72145].

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

ERROR: ORA-600 in COD recovery for diskgroup 2/0xda5d0748 (DATA_NEW)

ERROR: ORA-600 thrown in RBAL for group number 2

Errors in file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_18443.trc:

ORA-00600: internal error code, arguments: [kfrcRecover_4], [0], [], [], [], [], [], [], [], [], [], []

Errors in file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_18443.trc:

ORA-00600: internal error code, arguments: [kfrcRecover_4], [0], [], [], [], [], [], [], [], [], [], []

RBAL (ospid: 18443): terminating the instance due to error 488

System state dump requested by (instance=2, osid=18443 (RBAL)), summary=[abnormal instance termination].

System State dumped to trace file /home/u01/app/11.2.0/grid/log/diag/asm/+asm/+ASM2/trace/+ASM2_diag_18417.trc

Instance terminated by RBAL, pid = 18443

最后为了尽快恢复业务，我们选择了两条腿走路；一遍kfed修复，一遍进行amdu+odu进行恢复。。。。。

那么为什么会出现这种问题呢？都add disk成功了，然后磁盘组还会挂掉？

本质上是因为Oracle 在 add disk操作的时候并不会去检查allocate table是否完整或者有异常；主要是更新disk directory、pst等元数据。当加盘成功之后，开始进行reblance时才会去更新allocate table；如果此时发现at block有损坏；则会报错。

如下一个简单的测试：

+++add disk后破坏at block（第一个at）
GMON querying group 1 at 9 for pid 13, osid 2610
SUCCESS: refreshed membership for 1/0xf1d9ce85 (DATA)
SUCCESS: ALTER DISKGROUP data add disk '/dev/asm-diskh' REBALANCE POWER 2
NOTE: starting rebalance of group 1/0xf1d9ce85 (DATA) at power 2
Starting background process ARB0
Sun Apr 12 02:09:12 2020
ARB0 started with pid=21, OS id=2672 
NOTE: assigning ARB0 to group 1/0xf1d9ce85 (DATA) with 2 parallel I/Os
cellip.ora not found.
WARNING: cache read  a corrupt block: group=1(DATA) dsk=0 blk=2 disk=0 (DATA_0000) incarn=3915988604 au=0 blk=2 count=1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM/trace/+ASM_arb0_2672.trc:
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [2] [0 != 1]
NOTE: a corrupted block from group DATA was dumped to /u01/app/grid/diag/asm/+asm/+ASM/trace/+ASM_arb0_2672.trc
WARNING: cache read (retry) a corrupt block: group=1(DATA) dsk=0 blk=2 disk=0 (DATA_0000) incarn=3915988604 au=0 blk=2 count=1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM/trace/+ASM_arb0_2672.trc:
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [2] [0 != 1]
ERROR: cache failed to read group=1(DATA) dsk=0 blk=2 from disk(s): 0(DATA_0000)
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [2] [0 != 1]
+++还原at block 备份之后
SQL> ALTER DISKGROUP data add disk '/dev/asm-diskh' REBALANCE POWER 2; 
Diskgroup altered.
+++reblance操作
Sun Apr 12 02:13:03 2020
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 1/0xda69ce8f (DATA)

+++add disk后破坏at block（第一个at）

GMON querying group 1 at 9 for pid 13, osid 2610

SUCCESS: refreshed membership for 1/0xf1d9ce85 (DATA)

SUCCESS: ALTER DISKGROUP data add disk '/dev/asm-diskh' REBALANCE POWER 2

NOTE: starting rebalance of group 1/0xf1d9ce85 (DATA) at power 2

Starting background process ARB0

Sun Apr 12 02:09:12 2020

ARB0 started with pid=21, OS id=2672

NOTE: assigning ARB0 to group 1/0xf1d9ce85 (DATA) with 2 parallel I/Os

cellip.ora not found.

WARNING: cache read a corrupt block: group=1(DATA) dsk=0 blk=2 disk=0 (DATA_0000) incarn=3915988604 au=0 blk=2 count=1

Errors in file /u01/app/grid/diag/asm/+asm/+ASM/trace/+ASM_arb0_2672.trc:

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [2] [0 != 1]

NOTE: a corrupted block from group DATA was dumped to /u01/app/grid/diag/asm/+asm/+ASM/trace/+ASM_arb0_2672.trc

WARNING: cache read (retry) a corrupt block: group=1(DATA) dsk=0 blk=2 disk=0 (DATA_0000) incarn=3915988604 au=0 blk=2 count=1

Errors in file /u01/app/grid/diag/asm/+asm/+ASM/trace/+ASM_arb0_2672.trc:

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [2] [0 != 1]

ERROR: cache failed to read group=1(DATA) dsk=0 blk=2 from disk(s): 0(DATA_0000)

ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [2] [0 != 1]

+++还原at block 备份之后

SQL> ALTER DISKGROUP data add disk '/dev/asm-diskh' REBALANCE POWER 2;

Diskgroup altered.

+++reblance操作

Sun Apr 12 02:13:03 2020

NOTE: stopping process ARB0

SUCCESS: rebalance completed for group 1/0xda69ce8f (DATA)

由此可见；这是一个比较大的隐患。在10g和11g对asm 元数据方向的管理和维护并不是非常完善。曾经遇到过add disk对于磁盘实际大小校验不严格的问题。

我想，这也是为什么Oracle 从12c版本开始，不仅仅是对disk header进行备份；而是会将整个AU 0都进行备份。这或许是Oracle发现了这个缺陷吧。

Tags: amdu, endian_kfbh, kfed merge, kfrcRecover_4, odu, ora-00600, ORA-15066, ORA-15130: diskgroup "DATA_NEW" is being dismounted, ORA-15196: invalid ASM block header
Posted in ASM, backup & rcovery, Oracle rdbms on 4月 12, 2020

One Response to “ASM 加盘导致数据库无法Open的案例（add disk很危险）”

admin Says:

4月 12, 2020 at 上午 8:30

实际上后面我kfed读了一下之前同事的备份；发现确实大量的坏块。
[oracle@mysqldb1 tmp]$ for (( i=0; i<256; i++ )) do kfed read /tmp/lun1.bak aun=0 blkn=$i | grep 'kfbh.type' ; done
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.type: 2 ; 0x002: KFBTYP_FREESPC
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.type: 0 ; 0x002: KFBTYP_INVALID

You must be logged in to post a comment.

love wife love life —Roger的Oracle/MySQL/PostgreSQL数据恢复博客

Categories

Archives

最新评论

国内圈子

oracle security

ASM 加盘导致数据库无法Open的案例（add disk很危险）

One Response to “ASM 加盘导致数据库无法Open的案例（add disk很危险）”

Leave a Reply