Archives For patching

lost your oraInventory ? or somehow corrupted it ?

December 12, 2012 — 1 Comment

tricky situation: planned to apply a patch to an oracle home and opatch failes with:

OPatch failed to locate Central Inventory.
Possible causes are: 
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
LsInventorySession failed: OPatch failed to locate Central Inventory.
Possible causes are: 
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.

obviously the first thing to check is if the pointer ( oraInst.loc ) points to the correct location and the inventory group is listed correctly:

cat /etc/oraInst.loc 
inventory_loc=/opt/oracle/oraInventory
inst_group=oinstall

this is what it should look like assuming oinstall is really the inventory group and your inventory location is really /opt/oracle/oraInventory. so this part is fine but somehow the oraInventory got lost:

ls -la /opt/oracle/
total 312
drwxrwx--- 4 grid   oinstall   4096 Sep 19 21:54 .
drwxr-xr-x 4 root   root       4096 Mar 23 08:52 ..
drwxrwx--- 4 grid   oinstall   4096 Mar 23 08:43 product
drwxrwx--- 3 grid   oinstall   4096 May  3 23:52 stage

what to do ? restore it from backup would be one solution ( you do regular backups of your inventory, don’t you ? ). let’s assume no backup is available. you still may restore the inventory using the oracle installer available in the oracle home you want to register:

ls -la $ORACLE_HOME/oui
total 60
drwxr-xr-x  8 oracle oinstall 4096 Mar 26 14:23 .
drwxrwx--- 76 oracle oinstall 4096 May  3 23:17 ..
-rwxr-xr-x  1 oracle oinstall  323 Feb 17  2007 admin_langs.xml
drwxr-xr-x  3 oracle oinstall 4096 Mar 26 14:23 bin
-rw-r-----  1 oracle oinstall 6680 Mar 26 14:20 clusterparam.ini
drwxr-xr-x  2 oracle oinstall 4096 Mar 26 14:20 instImages
drwxr-xr-x  4 oracle oinstall 4096 Mar 26 14:20 jlib
drwxr-xr-x  3 oracle oinstall 4096 Mar 26 14:23 lib
-rwxr-xr-x  1 oracle oinstall 2110 Feb 17  2007 nlsrtlmap.xml
-rw-r-----  1 oracle oinstall 6782 Mar 26 14:20 oraparam.ini
drwxr-xr-x  3 oracle oinstall 4096 Mar 26 14:16 prov
-rwxr-xr-x  1 oracle oinstall  748 Feb 17  2007 runtime_langs.xml
drwxr-xr-x  2 oracle oinstall 4096 Mar 26 14:20 schema

even if the inventory is completely lost for recreating it a simple call to the oracle installer will recreate it and register the ORACLE_HOME:

$ORACLE_HOME/oui/bin/runInstaller -silent -attachHome ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=H11203

if successful you should see the following output:

Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 2000 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /opt/oracle/oraInventory
'AttachHome' was successful.

… and opatch will succeed from now on.

In main, patching
permalink

Ouch! ORA-01031: insufficient privileges ?

June 20, 2012 — Leave a comment

yesterday we hit a bug in 11.2.0.3:

if you grant various dbms_* packages to another user “with grant” option and try to pass through this grant with that user you may hit bug 13036331

simple test-case to confirm that you are affected:

create user blubb identified by "blubb";
create user blabb identified by "blabb";
grant connect,resource to blubb;
grant execute on dbms_lob to blubb with grant option;
connect blubb/blubb
grant execute on dbms_lob to blabb;
*
ERROR at line 1:
ORA-01031: insufficient privileges

it seems that only 11.2.0.3 is affected. we already had a psu applied ( 11.2.0.3.1 ), so this does not help…

In patching
permalink

Ouch! tns poison

April 29, 2012 — Leave a comment

there is a very old critical security hole regarding the oracle database. be aware of it, as there is no fix for it currently:

german

english

update: official workaround available now

In patching ,
permalink

last but not least: apply the PSU

March 28, 2012 — Leave a comment

I could have applied the PSU earlier ( before creating the database ), but in real life this is the more likely scenario. As the patch sources are already in place:

su - oracle
cd /opt/oracle/stage
ls -la
6880880_112000_Linux-x86-64.zip
p13348650_112030_Linux-x86-64.zip
opatch_response.rsp

start with updating opatch:

unzip p6880880_112000_Linux-x86-64.zip -d $ORACLE_HOME/

It is safe to answer “All” if the unzip utility ask to replace some files.

Extract and check the readme:

unzip p13348650_112030_LINUX.zip

The structure of the oracle patches is always the same. You will get subdirectories named with the patch number and the readme:

ls -la
drwxr-xr-x 4 oracle oinstall 4096 Jan 14 17:08 13343438
drwxr-xr-x 5 oracle oinstall 4096 Jan 14 17:08 13348650
-rw-r--r-- 1 oracle oinstall 422 Jan 14 17:08 bundle.xml
-rwxrwx--- 1 grid oinstall 621 Mar 23 11:31 opatch_response.rsp
-rwxrwx--- 1 grid oinstall 32510817 Mar 23 10:54 p6880880_112000_Linux-x86-64.zip
-rw-rw-r-- 1 oracle oinstall 107674 Jan 16 14:49 README.html
-rw-r--r-- 1 oracle oinstall 21 Jan 14 17:08 README.txt

Because this patch contains both, the database and the grid patches, change to the database patch directory ( which is 13343438 ) and apply the patchset:

cd /opt/oracle/stage/13343438
srvctl stop listener -l listener_DB112
srvctl stop database -d DB112
export PATH=$ORACLE_HOME/OPatch:$PATH
opatch lsinventory -detail -oh $ORACLE_HOME
opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /opt/oracle/stage/13343438 -oh $ORACLE_HOME
$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -output /opt/oracle/stage/opatch_response.rsp -no_banner
opatch apply -ocmrf /opt/oracle/stage/opatch_response.rsp

The output should be very similar to this:

Oracle Home : /opt/oracle/product/base/11.2.0.3
Central Inventory : /opt/oracle/oraInventory
from : /opt/oracle/product/base/11.2.0.3/oraInst.loc
OPatch version : 11.2.0.3.0
OUI version : 11.2.0.3.0
Log file location : /opt/oracle/product/base/11.2.0.3/cfgtoollogs/opatch/13343438_Mar_28_2012_15_35_47/apply2012-03-28_15-35-47PM_1.log
Applying interim patch '13343438' to OH '/opt/oracle/product/base/11.2.0.3'
Verifying environment and performing prerequisite checks...
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/opt/oracle/product/base/11.2.0.3')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Patching component oracle.rdbms.rsf, 11.2.0.3.0...
Patching component oracle.rdbms, 11.2.0.3.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.3.0...
Verifying the update...
Patch 13343438 successfully applied
Log file location: /opt/oracle/product/base/11.2.0.3/cfgtoollogs/opatch/13343438_Mar_28_2012_15_35_47/apply2012-03-28_15-35-47PM_1.log
OPatch succeeded.

Patch ( almost ) applied.

Perhaps you noticed that opatch creates a new directory under you ORACLE_HOME which is called .patch_storage:

ls -la /opt/oracle/product/base/11.2.0.3/.patch_storage/
total 28
drwxr-xr-x 4 oracle oinstall 4096 Mar 28 15:38 .
drwxrwx--- 75 oracle oinstall 4096 Mar 28 15:36 ..
drwxr-xr-x 4 oracle oinstall 4096 Mar 28 15:38 13343438_Jan_3_2012_02_50_27
-rw-r--r-- 1 oracle oinstall 336 Mar 28 15:36 interim_inventory.txt
-rw-r--r-- 1 oracle oinstall 92 Mar 28 15:36 LatestOPatchSession.properties
-rw-r--r-- 1 oracle oinstall 0 Mar 28 15:38 patch_free
-rw-r--r-- 1 oracle oinstall 51 Mar 28 15:36 record_inventory.txt
drwxr-xr-x 2 oracle oinstall 4096 Mar 28 15:38 verify

Under no circumstances delete this files. You will not be able apply any futures patches if you do this. If you really experience space issues, you can do a cleanup of the files which are not needed:

opatch util cleanup

One step is still missing. It it very likely that oracle modified some SQL files with this patch, so one needs to apply this, too:

srvctl start listener -l listener_DB112
srvctl start database -d DB112
sqlplus / as sysdba
@?/psu/11.2.0.3.1/catpsu.sql
select * from registry$history;

Work done.

Now there is a working Grid Infrastructure and database each patched to the ( currently ) latest release. If you did not export your Virtual Machine and did not create any snapshots I recommend to do an export now ( you need to shutdown your Virtual Machine for this ):

vboxmanage export "oracleplayground" --output oracleplayground.ovf

If you destroy the database or any parts of the grid infrastructure you are always able to bring it back in a few minutes.

Where to go from now ? This depends on you. If you are new to oracle there is a lot of literature out there, but the most important ( and at no cost ): Oracle Database Concepts . This should be your starting point.

In demonstration, patching
permalink

how to apply patchset updates to your grid infrastructure

March 23, 2012 — Leave a comment

As time goes by people report bugs for the oracle software. Some of them are critical, others not. Sooner or later you will come to the situation where you’ll have to patch your oracle software. The utility oracle uses to patch it’s software is called opatch. To avoid any issues with opatch always download the latest release from oracle support ( search for patch number: 6880880 and download ).

For this little demonstration we will apply the Patch Set Update one for the Grid Infratrucure. Note that this patch includes Patch Set Update 1 for the database software, too ( we will patch the database software in a future post ). If you did not transer the patchset and opatch to your Virtual Machine, now is the time to do so ( see here for a description on how to do it ).

As mentioned before, always bring your opatch to the latest release:

su - grid
cd /opt/oracle/stage
unzip p6880880_112000_Linux-x86-64.zip -d $ORACLE_HOME/

It is safe to answer “All” if the unzip utility ask to replace some files.

Extract the patch:

su - grid
cd /opt/oracle/stage
unzip p13348650_112030_LINUX.zip

The structure of the oracle patches is always the same. You will get subdirectories named with the patch number and the readme:

ls -la
drwxr-xr-x 4 grid oinstall      4096 Jan 14 17:08 13343438
drwxr-xr-x 5 grid oinstall      4096 Jan 14 17:08 13348650
-rw-r--r-- 1 grid oinstall       422 Jan 14 17:08 bundle.xml
-rw------- 1 grid oinstall 335597984 Mar 23 11:04 p13348650_112030_Linux-x86-64.zip
drwxr-xr-x 8 grid oinstall      4096 Sep 22  2011 grid
-rw-r--r-- 1 grid oinstall  32510817 Mar 23 10:54 p6880880_112000_Linux-x86-64.zip
-rw-rw-r-- 1 grid oinstall    107674 Jan 16 14:49 README.html
-rw-r--r-- 1 grid oinstall        21 Jan 14 17:08 README.txt

Make sure you carefully read the readme.

Applying the patch is straight forward ( Note that Patches to the Grid Infrastructure need to be applied as root ). The first thing to do is to check if the current oracle inventory is consistent:

su - grid
cd /opt/oracle/stage
export PATH=$ORACLE_HOME/OPatch:$PATH
opatch lsinventory -detail -oh $ORACLE_HOME

If your inventory is fine, you should see the following output:

Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.
Oracle Home       : /opt/oracle/product/crs/11.2.0.3
Central Inventory : /opt/oracle/oraInventory
   from           : /opt/oracle/product/crs/11.2.0.3/oraInst.loc
OPatch version    : 11.2.0.3.0
OUI version       : 11.2.0.3.0
Log file location : /opt/oracle/product/crs/11.2.0.3/cfgtoollogs/opatch/opatch2012-03-23_11-14-23AM_1.log
Lsinventory Output file location : /opt/oracle/product/crs/11.2.0.3/cfgtoollogs/opatch/lsinv/lsinventory2012-03-23_11-14-23AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1): 
Oracle Grid Infrastructure                                           11.2.0.3.0
There are 1 products installed in this Oracle Home.
Installed Products (88): 
Agent Required Support Files                                         10.2.0.4.3
Assistant Common Files                                               11.2.0.3.0
Automatic Storage Management Assistant                               11.2.0.3.0
Bali Share                                                           1.1.18.0.0
Buildtools Common Files                                              11.2.0.3.0
Character Set Migration Utility                                      11.2.0.3.0
Cluster Ready Services Files                                         11.2.0.3.0
Cluster Verification Utility Common Files                            11.2.0.3.0
Cluster Verification Utility Files                                   11.2.0.3.0
Database SQL Scripts                                                 11.2.0.3.0
Deinstallation Tool                                                  11.2.0.3.0
Enterprise Manager Common Core Files                                 10.2.0.4.4
Enterprise Manager Common Files                                      10.2.0.4.3
Enterprise Manager plugin Common Files                               11.2.0.3.0
Expat libraries                                                       2.0.1.0.1
HAS Common Files                                                     11.2.0.3.0
HAS Files for DB                                                     11.2.0.3.0
Installation Common Files                                            11.2.0.3.0
Installation Plugin Files                                            11.2.0.3.0
Installer SDK Component                                              11.2.0.3.0
LDAP Required Support Files                                          11.2.0.3.0
OLAP SQL Scripts                                                     11.2.0.3.0
Oracle Clusterware RDBMS Files                                       11.2.0.3.0
Oracle Configuration Manager Deconfiguration                         10.3.1.0.0
Oracle Containers for Java                                           11.2.0.3.0
Oracle Core Required Support Files                                   11.2.0.3.0
Oracle Database 11g                                                  11.2.0.3.0
Oracle Database 11g Multimedia Files                                 11.2.0.3.0
Oracle Database Deconfiguration                                      11.2.0.3.0
Oracle Database User Interface                                       2.2.13.0.0
Oracle Database Utilities                                            11.2.0.3.0
Oracle DBCA Deconfiguration                                          11.2.0.3.0
Oracle Extended Windowing Toolkit                                    3.4.47.0.0
Oracle Globalization Support                                         11.2.0.3.0
Oracle Globalization Support                                         11.2.0.3.0
Oracle Grid Infrastructure                                           11.2.0.3.0
Oracle Help For Java                                                  4.2.9.0.0
Oracle Ice Browser                                                    5.2.3.6.0
Oracle Internet Directory Client                                     11.2.0.3.0
Oracle Java Client                                                   11.2.0.3.0
Oracle JDBC/OCI Instant Client                                       11.2.0.3.0
Oracle JDBC/THIN Interfaces                                          11.2.0.3.0
Oracle JFC Extended Windowing Toolkit                                4.2.36.0.0
Oracle JVM                                                           11.2.0.3.0
Oracle LDAP administration                                           11.2.0.3.0
Oracle Locale Builder                                                11.2.0.3.0
Oracle Multimedia                                                    11.2.0.3.0
Oracle Multimedia Client Option                                      11.2.0.3.0
Oracle Multimedia Java Advanced Imaging                              11.2.0.3.0
Oracle Multimedia Locator                                            11.2.0.3.0
Oracle Multimedia Locator RDBMS Files                                11.2.0.3.0
Oracle Net                                                           11.2.0.3.0
Oracle Net Listener                                                  11.2.0.3.0
Oracle Net Required Support Files                                    11.2.0.3.0
Oracle Netca Client                                                  11.2.0.3.0
Oracle Notification Service                                          11.2.0.3.0
Oracle Notification Service (eONS)                                   11.2.0.3.0
Oracle One-Off Patch Installer                                       11.2.0.1.7
Oracle Quality of Service Management (Client)                        11.2.0.3.0
Oracle Quality of Service Management (Server)                        11.2.0.3.0
Oracle RAC Deconfiguration                                           11.2.0.3.0
Oracle RAC Required Support Files-HAS                                11.2.0.3.0
Oracle Recovery Manager                                              11.2.0.3.0
Oracle Security Developer Tools                                      11.2.0.3.0
Oracle Text Required Support Files                                   11.2.0.3.0
Oracle Universal Installer                                           11.2.0.3.0
Oracle USM Deconfiguration                                           11.2.0.3.0
Oracle Wallet Manager                                                11.2.0.3.0
Parser Generator Required Support Files                              11.2.0.3.0
Perl Interpreter                                                     5.10.0.0.2
Perl Modules                                                         5.10.0.0.1
PL/SQL                                                               11.2.0.3.0
PL/SQL Embedded Gateway                                              11.2.0.3.0
Platform Required Support Files                                      11.2.0.3.0
Precompiler Required Support Files                                   11.2.0.3.0
RDBMS Required Support Files                                         11.2.0.3.0
RDBMS Required Support Files for Instant Client                      11.2.0.3.0
RDBMS Required Support Files Runtime                                 11.2.0.3.0
Required Support Files                                               11.2.0.3.0
Secure Socket Layer                                                  11.2.0.3.0
SQL*Plus                                                             11.2.0.3.0
SQL*Plus Files for Instant Client                                    11.2.0.3.0
SQL*Plus Required Support Files                                      11.2.0.3.0
SSL Required Support Files for InstantClient                         11.2.0.3.0
Sun JDK                                                             1.5.0.30.03
Universal Storage Manager Files                                      11.2.0.3.0
XDK Required Support Files                                           11.2.0.3.0
XML Parser for Java                                                  11.2.0.3.0
There are 88 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.

Make sure there are no conflicts with the patch you want to apply:

su - grid
cd /opt/oracle/stage
export PATH=$ORACLE_HOME/OPatch:$PATH
opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /opt/oracle/stage/13343438 -oh $ORACLE_HOME

As this is a fresh installation with no patches applied until now, you should see the following output:

Oracle Interim Patch Installer version 11.2.0.3.0
Copyright (c) 2012, Oracle Corporation.  All rights reserved.
PREREQ session
Oracle Home       : /opt/oracle/product/crs/11.2.0.3
Central Inventory : /opt/oracle/oraInventory
   from           : /opt/oracle/product/crs/11.2.0.3/oraInst.loc
OPatch version    : 11.2.0.3.0
OUI version       : 11.2.0.3.0
Log file location : /opt/oracle/product/crs/11.2.0.3/cfgtoollogs/opatch/opatch2012-03-23_11-24-51AM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

Because we want an automated patch installation without repsonding to the questions we create a response file for opatch now:

su - grid
export PATH=$ORACLE_HOME/OPatch:$PATH
$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -output /opt/oracle/stage/opatch_response.rsp -no_banner

You do not need to provide an email address, just respond “Y” to the question that follows.

Time to apply the patch:

su -
cd /opt/oracle/stage
export PATH=/opt/oracle/product/crs/11.2.0.3/OPatch:$PATH
opatch auto /opt/oracle/stage -oh /opt/oracle/product/crs/11.2.0.3 -ocmrf /opt/oracle/stage/opatch_response.rsp

The output, if successful, looks like:

Executing /usr/bin/perl /opt/oracle/product/crs/11.2.0.3/OPatch/crs/patch112.pl -patchdir /opt/oracle -patchn stage -oh /opt/oracle/product/crs/11.2.0.3 -ocmrf /opt/oracle/stage/opatch_response.rsp -paramfile /opt/oracle/product/crs/11.2.0.3/crs/install/crsconfig_params
opatch auto log file location is /opt/oracle/product/crs/11.2.0.3/OPatch/crs/../../cfgtoollogs/opatchauto2012-03-23_11-34-03.log
Detected Oracle Restart install
Using configuration parameter file: /opt/oracle/product/crs/11.2.0.3/crs/install/crsconfig_params
Successfully unlock /opt/oracle/product/crs/11.2.0.3
patch /opt/oracle/stage/13348650  apply successful for home  /opt/oracle/product/crs/11.2.0.3
patch /opt/oracle/stage/13343438  apply successful for home  /opt/oracle/product/crs/11.2.0.3
ACFS-9459: ADVM/ACFS is not supported on this OS version: 'centos-release-5-8.el5.centos'
CRS-4123: Oracle High Availability Services has been started.

As CentOS is not a supported platform, you can ignore the ACFS-9459 message. Be aware that the Cluster Stack will get restarted.

Can can recheck your inventory now to be sure the patch has been applied:

su - grid
export PATH=$ORACLE_HOME/OPatch:$PATH
opatch lsinventory -details

Make a reboot test, to be sure everything is working:

su -
reboot

Login again and check the cluster stack:

su - grid
crsstat

Again, remember to take a snapshot of your Virtual Machine if you want to save your work or if you want to revert to this point. Delete the source files if you do not want to become your Virtual Machine too big before:

su - grid
cd /opt/oracle/stage/
rm -rf 13343438/ 13348650/ bundle.xml README.html README.txt

We will keep the zips and the response file for database patching in one of the next posts.

On your workstation create the snapshot:

vboxmanage snapshot oracleplayground take "after gi patch"
In patching
permalink