Archives For December 2012

tricky situation: planned to apply a patch to an oracle home and opatch failes with:

OPatch failed to locate Central Inventory.
Possible causes are: 
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
LsInventorySession failed: OPatch failed to locate Central Inventory.
Possible causes are: 
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.

obviously the first thing to check is if the pointer ( oraInst.loc ) points to the correct location and the inventory group is listed correctly:

cat /etc/oraInst.loc 
inventory_loc=/opt/oracle/oraInventory
inst_group=oinstall

this is what it should look like assuming oinstall is really the inventory group and your inventory location is really /opt/oracle/oraInventory. so this part is fine but somehow the oraInventory got lost:

ls -la /opt/oracle/
total 312
drwxrwx--- 4 grid   oinstall   4096 Sep 19 21:54 .
drwxr-xr-x 4 root   root       4096 Mar 23 08:52 ..
drwxrwx--- 4 grid   oinstall   4096 Mar 23 08:43 product
drwxrwx--- 3 grid   oinstall   4096 May  3 23:52 stage

what to do ? restore it from backup would be one solution ( you do regular backups of your inventory, don’t you ? ). let’s assume no backup is available. you still may restore the inventory using the oracle installer available in the oracle home you want to register:

ls -la $ORACLE_HOME/oui
total 60
drwxr-xr-x  8 oracle oinstall 4096 Mar 26 14:23 .
drwxrwx--- 76 oracle oinstall 4096 May  3 23:17 ..
-rwxr-xr-x  1 oracle oinstall  323 Feb 17  2007 admin_langs.xml
drwxr-xr-x  3 oracle oinstall 4096 Mar 26 14:23 bin
-rw-r-----  1 oracle oinstall 6680 Mar 26 14:20 clusterparam.ini
drwxr-xr-x  2 oracle oinstall 4096 Mar 26 14:20 instImages
drwxr-xr-x  4 oracle oinstall 4096 Mar 26 14:20 jlib
drwxr-xr-x  3 oracle oinstall 4096 Mar 26 14:23 lib
-rwxr-xr-x  1 oracle oinstall 2110 Feb 17  2007 nlsrtlmap.xml
-rw-r-----  1 oracle oinstall 6782 Mar 26 14:20 oraparam.ini
drwxr-xr-x  3 oracle oinstall 4096 Mar 26 14:16 prov
-rwxr-xr-x  1 oracle oinstall  748 Feb 17  2007 runtime_langs.xml
drwxr-xr-x  2 oracle oinstall 4096 Mar 26 14:20 schema

even if the inventory is completely lost for recreating it a simple call to the oracle installer will recreate it and register the ORACLE_HOME:

$ORACLE_HOME/oui/bin/runInstaller -silent -attachHome ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=H11203

if successful you should see the following output:

Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 2000 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /opt/oracle/oraInventory
'AttachHome' was successful.

… and opatch will succeed from now on.

something to smile about …

enterprise manager 12c download

another ouch with GI 11.2.0.3 on solaris 10 sparc 64bit: if one of your cluster nodes restarts and you can not find any evident reason for it despite some of these entries in the logs:

[cssd(1084)]CRS-1612:Network communication with node1 node (1) missing for 50% of timeout interval. Removal of this node from cluster in 14
.258 seconds
[cssd(1084)]CRS-1625:Node node1, number 1, was manually shut down
[cssd(1084)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node2 .
[ctssd(1117)]CRS-2407:The new Cluster Time Synchronization Service reference node is host node2.
[crsd(1522)]CRS-5504:Node down event reported for node 'node1'.
[cssd(1084)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 node2 .

… and:

[ CSSD][20](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
[ CSSD][20]###################################
[ CSSD][20]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
[ CSSD][20]###################################
[ CSSD][20](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
[ CSSD][20]

you probably hit bug 13869978. this seems only to happen if you are on external redundancy for the cluster diskgroup and therefore only one voting disk was created.

two solutions are available:

  • migrate the votings disk to an asm mirrored diskgroup ( normal or high redundancy )
  • or apply PSU4 on top of 11.2.0.3

there seems to be the same issue on linux.

jumping from one host to another because it is not possible to reach the destination host directly can be annoying and time consuming. if you use ssh to connect to your hosts ( and you probably should ) there is an easy way to simplify this.

let’s say you want to jump to host1, from host1 to host2 and from host2 to host3.

here’s the trick ( thanks to a colleague of mine ):

ssh -t -l [USER_ID] -A [HOST1] ssh -t -l [USER_ID] -A [HOST2] ssh -t -l [USER_ID] -A [HOST3]

define an alias for this:

alias my_jump='ssh -t -l [USER_ID] -A [HOST1] ssh -t -l [USER_ID] -A [HOST2] ssh -t -l [USER_ID] -A [HOST3]'

… and it’s easy going:

my_jump