Archives For linux

head over to the blog of dbi services to read the full article:

No journal messages available before the last reboot of your CentOS/RHEL system?

Advertisements

head over to the blog of dbi services to read the full article:

Going from SLES12 SP2 to SLES12 SP3, online

head over to the blog of dbi services to read the full article:

Linux quick tip – What is the local time in Kolkata?

head over to the blog of dbi services to read the full article:

Upgrading the Grid Infrastructure from 12.1.0.1 to 12.1.0.2 on the command line

head over to the blog of dbi services to read the full article:

Linux Magic System Request Key Hacks

head over to the blog of dbi services to read the full article:

tmux – an alternative to screen

lets say you have a diskgroup called “dummy”:

SYS@+ASM> select GROUP_NUMBER,NAME,STATE from v$asm_diskgroup where name = 'DUMMY';

GROUP_NUMBER NAME			    STATE
------------ ------------------------------ -----------
	   3 DUMMY			    MOUNTED

currently the diskgroup contains once device:

SYS@+ASM> select name,path,header_status from v$asm_disk where group_number = 3;

NAME	   PATH 		HEADER_STATU
---------- -------------------- ------------
DUMMY_0000 /dev/sdg1		MEMBER

some time in the future the diskgroup runs out of space and you request another device from the storage or os team. once the device is ready you check if you can see it in ASM:

SYS@+ASM> select name,path from v$asm_disk where header_status = 'CANDIDATE';

NAME	   PATH
---------- --------------------
	   /dev/sdh1
	   /dev/sdh

perfect, a new device is available to extend the diskgroup:

SYS@+ASM> alter diskgroup dummy add disk '/dev/sdh1';

Diskgroup altered.

time goes on and the diskgroup runs out of space again. another dba checks if there are decives available to add:

SYS@+ASM> select name,path from v$asm_disk where header_status = 'CANDIDATE';

NAME	   PATH
---------- --------------------
	   /dev/sdh

cool, no need to request another device:

SYS@+ASM> alter diskgroup dummy add disk '/dev/sdh';

Diskgroup altered.

and bumm:

Errors in file .../admin/diag/asm/+asm/+ASM/trace/+ASM_arb0_2432.trc:
ORA-15130: diskgroup "" is being dismounted
ORA-15335: ASM metadata corruption detected in disk group 'DUMMY'
ORA-15130: diskgroup "DUMMY" is being dismounted
ORA-15066: offlining disk "DUMMY_0002" in group "DUMMY" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:29297] [endian_kfbh] [2147483650] [10] [0 != 1]

What happened?: Two different people did not look exactly what they are doing

Mistake one: the system in question used udev rules for device persistence. when the os people adjusted the udev rules for the new device they did not exclude the whole disk:

KERNEL=="sdh*",  OWNER="asmadmin", GROUP="asmdba", MODE="0660"

When this rule is applied ASM will see all the partitions of the disk as well as the disk itself:

/dev/sdh1
/dev/sdh

Mistake two: the dba which added the last disk did not recognize that the candidate:

SYS@+ASM> select name,path from v$asm_disk where header_status = 'CANDIDATE';

NAME	   PATH
---------- --------------------
	   /dev/sdh

..actually is the whole disk. so the whole disk was added to ASM after the partition (which spanned the whole disk) was added.

so, better look twice …