look twice before adding devices to ASM

lets say you have a diskgroup called “dummy”:

SYS@+ASM> select GROUP_NUMBER,NAME,STATE from v$asm_diskgroup where name = 'DUMMY';

GROUP_NUMBER NAME			    STATE
------------ ------------------------------ -----------
	   3 DUMMY			    MOUNTED

currently the diskgroup contains once device:

SYS@+ASM> select name,path,header_status from v$asm_disk where group_number = 3;

NAME	   PATH 		HEADER_STATU
---------- -------------------- ------------
DUMMY_0000 /dev/sdg1		MEMBER

some time in the future the diskgroup runs out of space and you request another device from the storage or os team. once the device is ready you check if you can see it in ASM:

SYS@+ASM> select name,path from v$asm_disk where header_status = 'CANDIDATE';

NAME	   PATH
---------- --------------------
	   /dev/sdh1
	   /dev/sdh

perfect, a new device is available to extend the diskgroup:

SYS@+ASM> alter diskgroup dummy add disk '/dev/sdh1';

Diskgroup altered.

time goes on and the diskgroup runs out of space again. another dba checks if there are decives available to add:

SYS@+ASM> select name,path from v$asm_disk where header_status = 'CANDIDATE';

NAME	   PATH
---------- --------------------
	   /dev/sdh

cool, no need to request another device:

SYS@+ASM> alter diskgroup dummy add disk '/dev/sdh';

Diskgroup altered.

and bumm:

Errors in file .../admin/diag/asm/+asm/+ASM/trace/+ASM_arb0_2432.trc:
ORA-15130: diskgroup "" is being dismounted
ORA-15335: ASM metadata corruption detected in disk group 'DUMMY'
ORA-15130: diskgroup "DUMMY" is being dismounted
ORA-15066: offlining disk "DUMMY_0002" in group "DUMMY" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]

What happened?: Two different people did not look exactly what they are doing

Mistake one: the system in question used udev rules for device persistence. when the os people adjusted the udev rules for the new device they did not exclude the whole disk:

KERNEL=="sdh*",  OWNER="asmadmin", GROUP="asmdba", MODE="0660"

When this rule is applied ASM will see all the partitions of the disk as well as the disk itself:

/dev/sdh1
/dev/sdh

Mistake two: the dba which added the last disk did not recognize that the candidate:

SYS@+ASM> select name,path from v$asm_disk where header_status = 'CANDIDATE';

NAME	   PATH
---------- --------------------
	   /dev/sdh

..actually is the whole disk. so the whole disk was added to ASM after the partition (which spanned the whole disk) was added.

so, better look twice …