look twice before adding devices to ASM

February 4, 2015 — 2 Comments

lets say you have a diskgroup called “dummy”:

SYS@+ASM> select GROUP_NUMBER,NAME,STATE from v$asm_diskgroup where name = 'DUMMY';

GROUP_NUMBER NAME			    STATE
------------ ------------------------------ -----------
	   3 DUMMY			    MOUNTED

currently the diskgroup contains once device:

SYS@+ASM> select name,path,header_status from v$asm_disk where group_number = 3;

NAME	   PATH 		HEADER_STATU
---------- -------------------- ------------
DUMMY_0000 /dev/sdg1		MEMBER

some time in the future the diskgroup runs out of space and you request another device from the storage or os team. once the device is ready you check if you can see it in ASM:

SYS@+ASM> select name,path from v$asm_disk where header_status = 'CANDIDATE';

NAME	   PATH
---------- --------------------
	   /dev/sdh1
	   /dev/sdh

perfect, a new device is available to extend the diskgroup:

SYS@+ASM> alter diskgroup dummy add disk '/dev/sdh1';

Diskgroup altered.

time goes on and the diskgroup runs out of space again. another dba checks if there are decives available to add:

SYS@+ASM> select name,path from v$asm_disk where header_status = 'CANDIDATE';

NAME	   PATH
---------- --------------------
	   /dev/sdh

cool, no need to request another device:

SYS@+ASM> alter diskgroup dummy add disk '/dev/sdh';

Diskgroup altered.

and bumm:

Errors in file .../admin/diag/asm/+asm/+ASM/trace/+ASM_arb0_2432.trc:
ORA-15130: diskgroup "" is being dismounted
ORA-15335: ASM metadata corruption detected in disk group 'DUMMY'
ORA-15130: diskgroup "DUMMY" is being dismounted
ORA-15066: offlining disk "DUMMY_0002" in group "DUMMY" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]

What happened?: Two different people did not look exactly what they are doing

Mistake one: the system in question used udev rules for device persistence. when the os people adjusted the udev rules for the new device they did not exclude the whole disk:

KERNEL=="sdh*",  OWNER="asmadmin", GROUP="asmdba", MODE="0660"

When this rule is applied ASM will see all the partitions of the disk as well as the disk itself:

/dev/sdh1
/dev/sdh

Mistake two: the dba which added the last disk did not recognize that the candidate:

SYS@+ASM> select name,path from v$asm_disk where header_status = 'CANDIDATE';

NAME	   PATH
---------- --------------------
	   /dev/sdh

..actually is the whole disk. so the whole disk was added to ASM after the partition (which spanned the whole disk) was added.

so, better look twice …

Advertisements

2 responses to look twice before adding devices to ASM

  1. 

    Hello.. Daniel,, do you have reference or solution to fix: ORA-15196: invalid ASM block header ?

  2. 

    Hi Olie,

    what are the permissions on your disk devices? Can you use kfed to read the headers?

    Cheers,
    Daniel

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s