when working with oracle there is exactly one numeric data type one may use for storing numbers ( and that’s number ). in postgresql there are ten. what happened when we migrated an oracle database to postgresql was, that a source table in oracle took much less space than the same table in postgresql. thanks to some people on the pgsql-performance mailing list one of the reasons for this was, that the wrong numeric data type was chosen for storing the integers in postgresql.

a simple test-case:

drop table datatype1;
drop table datatype2;
create table datatype1
( a numeric ( 8,0 )
, b numeric ( 8,0 )
);
insert into datatype1 ( a, b )
       values ( generate_series ( 1, 10000000 )
              , generate_series ( 1, 10000000 )
              );
create index idatatype1_1 on datatype1 ( a );
create index idatatype1_2 on datatype1 ( b );
create table datatype2
( a int
, b int
);
insert into datatype2 ( a, b )
       values ( generate_series ( 1, 10000000 )
              , generate_series ( 1, 10000000 )
             );
create index idatatype2_1 on datatype2 ( a );
create index idatatype2_2 on datatype2 ( b );
analyze verbose datatype1;
analyze verbose datatype2;
select pg_size_pretty ( pg_relation_size ( 'datatype1' ) );
 pg_size_pretty 
----------------
 422 MB
(1 row)
select pg_size_pretty ( pg_relation_size ( 'datatype2' ) );
 pg_size_pretty 
----------------
 346 MB
(1 row)

for these two little tables the difference is about 76mb. depending on the statements this is 76mb more that needs to be scanned and this can have great impacts on performance. surprisingly, at least for me, this is not true for the indexes:

select pg_size_pretty ( pg_relation_size ( 'idatatype1_1' ) );
 pg_size_pretty 
----------------
 214 MB
(1 row)
select pg_size_pretty ( pg_relation_size ( 'idatatype2_1' ) );
 pg_size_pretty 
----------------
 214 MB
(1 row)
select pg_size_pretty ( pg_relation_size ( 'idatatype2_1' ) );
 pg_size_pretty 
----------------
 214 MB
(1 row)
select pg_size_pretty ( pg_relation_size ( 'idatatype2_2' ) );
 pg_size_pretty 
----------------
 214 MB
(1 row)

so, it’s worth to keep an eye on which data types to use for numbers …

tricky situation: planned to apply a patch to an oracle home and opatch failes with:

OPatch failed to locate Central Inventory.
Possible causes are: 
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.
LsInventorySession failed: OPatch failed to locate Central Inventory.
Possible causes are: 
    The Central Inventory is corrupted
    The oraInst.loc file specified is not valid.

obviously the first thing to check is if the pointer ( oraInst.loc ) points to the correct location and the inventory group is listed correctly:

cat /etc/oraInst.loc 
inventory_loc=/opt/oracle/oraInventory
inst_group=oinstall

this is what it should look like assuming oinstall is really the inventory group and your inventory location is really /opt/oracle/oraInventory. so this part is fine but somehow the oraInventory got lost:

ls -la /opt/oracle/
total 312
drwxrwx--- 4 grid   oinstall   4096 Sep 19 21:54 .
drwxr-xr-x 4 root   root       4096 Mar 23 08:52 ..
drwxrwx--- 4 grid   oinstall   4096 Mar 23 08:43 product
drwxrwx--- 3 grid   oinstall   4096 May  3 23:52 stage

what to do ? restore it from backup would be one solution ( you do regular backups of your inventory, don’t you ? ). let’s assume no backup is available. you still may restore the inventory using the oracle installer available in the oracle home you want to register:

ls -la $ORACLE_HOME/oui
total 60
drwxr-xr-x  8 oracle oinstall 4096 Mar 26 14:23 .
drwxrwx--- 76 oracle oinstall 4096 May  3 23:17 ..
-rwxr-xr-x  1 oracle oinstall  323 Feb 17  2007 admin_langs.xml
drwxr-xr-x  3 oracle oinstall 4096 Mar 26 14:23 bin
-rw-r-----  1 oracle oinstall 6680 Mar 26 14:20 clusterparam.ini
drwxr-xr-x  2 oracle oinstall 4096 Mar 26 14:20 instImages
drwxr-xr-x  4 oracle oinstall 4096 Mar 26 14:20 jlib
drwxr-xr-x  3 oracle oinstall 4096 Mar 26 14:23 lib
-rwxr-xr-x  1 oracle oinstall 2110 Feb 17  2007 nlsrtlmap.xml
-rw-r-----  1 oracle oinstall 6782 Mar 26 14:20 oraparam.ini
drwxr-xr-x  3 oracle oinstall 4096 Mar 26 14:16 prov
-rwxr-xr-x  1 oracle oinstall  748 Feb 17  2007 runtime_langs.xml
drwxr-xr-x  2 oracle oinstall 4096 Mar 26 14:20 schema

even if the inventory is completely lost for recreating it a simple call to the oracle installer will recreate it and register the ORACLE_HOME:

$ORACLE_HOME/oui/bin/runInstaller -silent -attachHome ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=H11203

if successful you should see the following output:

Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 2000 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /opt/oracle/oraInventory
'AttachHome' was successful.

… and opatch will succeed from now on.

something to smile about …

enterprise manager 12c download

another ouch with GI 11.2.0.3 on solaris 10 sparc 64bit: if one of your cluster nodes restarts and you can not find any evident reason for it despite some of these entries in the logs:

[cssd(1084)]CRS-1612:Network communication with node1 node (1) missing for 50% of timeout interval. Removal of this node from cluster in 14
.258 seconds
[cssd(1084)]CRS-1625:Node node1, number 1, was manually shut down
[cssd(1084)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node2 .
[ctssd(1117)]CRS-2407:The new Cluster Time Synchronization Service reference node is host node2.
[crsd(1522)]CRS-5504:Node down event reported for node 'node1'.
[cssd(1084)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 node2 .

… and:

[ CSSD][20](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
[ CSSD][20]###################################
[ CSSD][20]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
[ CSSD][20]###################################
[ CSSD][20](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
[ CSSD][20]

you probably hit bug 13869978. this seems only to happen if you are on external redundancy for the cluster diskgroup and therefore only one voting disk was created.

two solutions are available:

  • migrate the votings disk to an asm mirrored diskgroup ( normal or high redundancy )
  • or apply PSU4 on top of 11.2.0.3

there seems to be the same issue on linux.

jumping from one host to another because it is not possible to reach the destination host directly can be annoying and time consuming. if you use ssh to connect to your hosts ( and you probably should ) there is an easy way to simplify this.

let’s say you want to jump to host1, from host1 to host2 and from host2 to host3.

here’s the trick ( thanks to a colleague of mine ):

ssh -t -l [USER_ID] -A [HOST1] ssh -t -l [USER_ID] -A [HOST2] ssh -t -l [USER_ID] -A [HOST3]

define an alias for this:

alias my_jump='ssh -t -l [USER_ID] -A [HOST1] ssh -t -l [USER_ID] -A [HOST2] ssh -t -l [USER_ID] -A [HOST3]'

… and it’s easy going:

my_jump

we are currently in the process of moving an oracle data warehouse to postgresql and had some fun on how oracle handles dates in contrast to postgresql. in oracle, if you subtract one date from another you’ll get an integer:

select to_date('05.01.2012','dd.mm.yyyy') - to_date ( '01.01.2012','dd.mm.yyyy') diff from dual;
      DIFF
----------
         4

if you do the same in postgresql you’ll get an interval:

select to_date('05.01.2012','dd.mm.yyyy') - to_date ( '01.01.2012','dd.mm.yyyy') diff;
  diff
--------
 4 days
(1 row)

the issue was, that a view used this in a where clause and did some calculations based on the return of the expression. obviously this failed when we created the view in postgresql. the trick was to use the to_char function on the interval:

select to_char ( to_date('05.01.2012','dd.mm.yyyy') - to_date ( '01.01.2012','dd.mm.yyyy'), 'DD' ) diff;                                      
diff
------
 04

… we thought :) but the result can still not be used to do some calculations:

select to_char ( to_date('05.01.2012','dd.mm.yyyy') - to_date ( '01.01.2012','dd.mm.yyyy'), 'DD' ) + 8;
ERROR:  operator does not exist: text + integer
LINE 1: ...m.yyyy') - to_date ( '01.01.2012','dd.mm.yyyy'), 'DD' ) + 8;
                                                                   ^
HINT:  No operator matches the given name and argument type(s). You might need to add explicit type casts.

another conversion was necessary to make this work:

select cast ( to_char ( to_date('05.01.2012','dd.mm.yyyy') - to_date ( '01.01.2012','dd.mm.yyyy'), 'DD' ) as int ) + 8;
 ?column?
----------
       12
(1 row)

lessons learned ? oracle is doing implicit type conversions silently in the background. postgresql does not …

another example:

oracle:

SQL> create table t ( a number );
Table created.
SQL> insert into t values ( 20120101 );
1 row created.
SQL>  select to_date(a,'yyyymmdd') from t;
TO_DATE(A,'YYYYMMDD'
--------------------
01-JAN-2012 00:00:00

postgresql:

create table t ( a number );
CREATE TABLE
insert into t values ( 20120101 );
INSERT 0 1
select to_date(a,'yyyymmdd') from t;
ERROR:  function to_date(numeric, unknown) does not exist
LINE 1: select to_date(a,'yyyymmdd') from t;
               ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.

if you want to manage vips in the grid infrastructure which are not on the default network and you get this: “CRS-2534: Resource type ‘ora.cluster_vip_net2.type’ is not registered” don’t panic, it is easy to fix. basically you need create the “ora.cluster_vip_net2.type”-type before adding the vip with appvipcfg:

./srvctl add network -k 2 -S x.x.x.0/255.255.255.0/igb0
./crsctl start resource ora.net2.network
./crsctl add type ora.cluster_vip_net2.type -basetype ora.cluster_vip.type
./appvipcfg create -network=2 -ip=x.x.x.x -vipname=myvip1 -user=root
./crsctl start resource vip1 -n server1
./appvipcfg create -network=2 -ip=x.x.x.x -vipname=myvip2 -user=root
./crsctl start resource vip2 -n server2
./crsctl stat res –t
./crsctl modify res 'myvip1' -attr "HOSTING_MEMBERS=server1 server2"
./crsctl modify res 'myvip2' -attr "HOSTING_MEMBERS=server1 server2"

not sure, but I think this is a bug as appvipcfg should manage this.

the above is valid for 11.2.0.3 on Solaris SPARC 64bit

back from holidays and the first ouch on the first day. we tried to setup a clustered grid infrastructure on a solaris 10 t4-2 machine. business as usual, one might think. but when running the rootcrs.pl setup script asmca ( which in invoked from rootcrs.pl automatically ) reports:

...
main] [ 2012-11-02 12:58:59.282 MET ] [OracleHome.getVersion:1023] Current version from sqlplus: 11.2.0.3.0
[main] [ 2012-11-02 12:58:59.282 MET ] [UsmcaLogger.logInfo:143] Role SYSASM
[main] [ 2012-11-02 12:58:59.282 MET ] [UsmcaLogger.logInfo:143] OS Auth true
[main] [ 2012-11-02 12:59:21.347 MET ] [SQLEngine.done:2189] Done called
[main] [ 2012-11-02 12:59:21.349 MET ] [USMInstance.configureLocalASM:3033] ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","SELECT DECODE(null,'','Total...","SQLA","tmp")
...

hm. there is a support note about this behaviour ( 1416083.1 ) but there is no way to give asm parameters to rootcrs.pl or the crsconfig_params files. what we needed to do:
disable some CPUs ( so there are less than 64 available ), re-configure the cluster, adjust the asm memory parameters and then enable the cpus again. this seems to be the only workaround at the moment ( 11.2.0.4 and 12.1 should contain a fix for this ).

while learning more about postgresql I came across the default case in postgresql ( it is lower case ). so, when querying the dictionary/catalog you’ll have to provide the lower case names to get any results:

postgres=# create table test1 ( a numeric );
CREATE TABLE
postgres=# select relname from pg_class where relname = 'TEST1';
 relname 
---------
(0 rows)
postgres=# select relname from pg_class where relname = 'test1';
 relname 
---------
 test1
(1 row)

in oracle you’ll need to use upper case by default:

SQL> create table test1 ( a number );
Table created.
SQL> select table_name from dba_tables where table_name = 'TEST1';
TABLE_NAME
------------------------------
TEST1
SQL> select table_name from dba_tables where table_name = 'test1';
no rows selected
SQL> 

if you want postgresql to respect the case when creating objects you’ll need to put double quotes around the names:

postgres=# create table "TEST2" ( a numeric );
CREATE TABLE
postgres=# select relname from pg_class where relname = 'TEST2';
 relname 
---------
 TEST2
(1 row)
postgres=# select relname from pg_class where relname = 'test22';
 relname 
---------
(0 rows)

same in oracle:

SQL> create table "test2" ( a number );
Table created.
SQL> select table_name from dba_tables where table_name = 'test2';
TABLE_NAME
------------------------------
test2
SQL> select table_name from dba_tables where table_name = 'TEST2';
no rows selected

knowing this, is it possible to create identical tables which just differ in the case of their name ? :

postgresql:

postgres=# create table test3 ( a numeric );
CREATE TABLE
postgres=# create table "Test3" ( a numeric );
CREATE TABLE
postgres=# create table "TesT3" ( a numeric );
CREATE TABLE
postgres=# select relname from pg_class where upper(relname) like 'TEST3%';
 relname 
---------
 Test3
 TesT3
 test3
(3 rows)

not an issue with postgresql. what about oracle ?

SQL> create table test3 ( a number );
Table created.
SQL> create table "Test3" ( a number );
Table created.
SQL> create table "TesT3" ( a number );
Table created.
SQL> select table_name from dba_tables where upper(table_name) like 'TEST3%';
TABLE_NAME
------------------------------
TesT3
Test3
TEST3

same behaviour. If someone had asked me if this is possible in oracle before, I would have said: no, definitely not. lessons learned ? :)

going further: what about constraint names ?
in postgresql:

postgres=# alter table test3 add constraint c1 check ( a is not null );
ALTER TABLE
postgres=# alter table test3 add constraint "C1" check ( a > 5 );
ALTER TABLE
postgres=# select conname,consrc from pg_constraint where upper(conname) = 'C1';
 conname |       consrc       
---------+--------------------
 c1      | (a IS NOT NULL)
 C1      | (a > (5)::numeric)

ok, this is consistent. what about oracle ? :

SQL> alter table test3 add constraint c1 check ( a is not null );
Table altered.
SQL> alter table test3 add constraint "C1" check ( a > 5 );
alter table test3 add constraint "C1" check ( a > 5 )
                                *
ERROR at line 1:
ORA-02264: name already used by an existing constraint

what about indexes ? postgresql:

postgres=# create index i1 on test3(a);
CREATE INDEX
postgres=# create index "i1" on test3(a);
ERROR:  relation "i1" already exists
postgres=# create index "I1" on test3(a);
CREATE INDEX
postgres=# select indexname,indexdef from pg_indexes where upper(indexname) = 'I1';
 indexname |                  indexdef                  
-----------+--------------------------------------------
 i1        | CREATE INDEX i1 ON test3 USING btree (a)
 I1        | CREATE INDEX "I1" ON test3 USING btree (a)
(2 rows)

oracle:

SQL> create index i1 on test3 ( a );
Index created.
SQL> create index "i1" on test3 ( a );
create index "i1" on test3 ( a )
                            *
ERROR at line 1:
ORA-01408: such column list already indexed

as oracle checks if an index is defined on the same column(s) this is not possible. slightly modified test:

SQL> alter table test3 add ( b number );
Table altered.
SQL> create index "I1" on test3 ( b );
create index "I1" on test3 ( b )
            *
ERROR at line 1:
ORA-00955: name is already used by an existing object

still not possible.

I did not check all the objects but it seems that oracle is not as consistent as postgresql in this case.

if you see errors like this in the alert-log of your standby database:

MRP0: Background Media Recovery terminated with error 1274
Errors in file /oradata/dbs100/admin/diag/rdbms/dbs100dg2/dbs100dg2/trace/dbs100dg2_mrp0_4371.trc:
ORA-01274: cannot add datafile '+DBS100_DATA_DG/dbs100dg1/datafile/tbs1.276.793899763' - file could not be created
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovery stopped due to failure in applying recovery marker (opcode 17.30).
Datafiles are recovered to a consistent state at change 10967593 but controlfile could be ahead of datafiles.
MRP0: Background Media Recovery process shutdown (dbs100dg2)

or this:

Errors in file /oradata/dbs100/admin/diag/rdbms/dbs100dg2/dbs100dg2/trace/dbs100dg2_mrp0_4981.trc:
ORA-01111: name for data file 6 is unknown - rename to correct file
ORA-01110: data file 6: '/opt/oracle/product/base/11.2.0.3.1/dbs/UNNAMED00006'
ORA-01157: cannot identify/lock data file 6 - see DBWR trace file
ORA-01111: name for data file 6 is unknown - rename to correct file
ORA-01110: data file 6: '/opt/oracle/product/base/11.2.0.3.1/dbs/UNNAMED00006'

… don’t panic. it is easy to fix.

first of all this happens if the STANDBY_FILE_MANAGEMENT parameter is set to “MANUAL” on the standby database. but how do you fix it ? the first option would be to re-create the standby database, but this would be a lot of work and depending on the size of the database could require a huge amount of time.

the easy way is to do it like this ( on the standby database, of course ):

SQL> ALTER DATABASE CREATE DATAFILE '/opt/oracle/product/base/11.2.0.3.1/dbs/UNNAMED00006' AS '+DBS100_DATA_DG';
Database altered.
SQL> alter database recover managed standby database disconnect from session;
Database altered.
SQL> alter system set standby_file_management='AUTO' scope=both;
System altered.

don’t forget to set the STANDBY_FILE_MANAGEMENT to “AUTO” after you created the file so that you don’t have to worry about such things in the future…