Daniel Westermann's Blog

OS ready, Grid Infrastructure ready, patched to the latest release. Time to start with the “real” things…

Thanks to the virtualization technology ( Virtual Box ) in this case, it is easy to add storage to you machine. If running, shut down your Virtual Machine and add two disks to it.

From your workstation, execute:

vboxmanage storagectl oracleplayground --name "SCSI Controller" --add scsi --controller LSILogic
vboxmanage createhd --filename ~/VirtualBox\ VMs/oracleplayground/oracleplayground_asm1.vdi --size=10240 --variant Fixed
vboxmanage createhd --filename ~/VirtualBox\ VMs/oracleplayground/oracleplayground_asm2.vdi --size=10240 --variant Fixed
vboxmanage storageattach oracleplayground --storagectl "SCSI Controller" --port 1 --device 0 --type hdd --medium ~/VirtualBox\ VMs/oracleplayground/oracleplayground_asm1.vdi
vboxmanage storageattach oracleplayground --storagectl "SCSI Controller" --port 2 --device 0 --type hdd --medium ~/VirtualBox\ VMs/oracleplayground/oracleplayground_asm2.vdi

What happened: We created a new SCSI Controller for our Virtual Machine, created two 10gb hard disks and attached them to the Controller.

Startup your Virtual Machine and check the devices:

su -
ls -la /dev/sd*
brw-r----- 1 root disk 8, 0 Mar 23 13:02 /dev/sda
brw-r----- 1 root disk 8, 16 Mar 23 13:02 /dev/sdb

These are our newly created disks. What we will need for ASM is one partition for each disk, so lets create the partitions:

su -
fdisk /dev/sda
fdisk /dev/sdb

The commands for creating the partitions are ( in this sequence ): p n p 1 enter enter w quit

If we check the devices again, we now can see the newly created partitions:

ls -la /dev/sd*
brw-r----- 1 root disk 8, 0 Mar 23 13:06 /dev/sda
brw-r----- 1 root disk 8, 1 Mar 23 13:06 /dev/sda1
brw-r----- 1 root disk 8, 16 Mar 23 13:09 /dev/sdb
brw-r----- 1 root disk 8, 17 Mar 23 13:09 /dev/sdb1

To be usable to asm, the disks must belong to the grid user. We could change the permissions now with chmod, but because we want this to be persistent across reboots we create udev rules for it:

su -
echo "KERNEL==\"sd[a-b][1-9]*\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" > /etc/udev/rules.d/61-asm.rules
reboot

Once the Virtual Machine rebooted, check the devices again and you will see that the owner,group and permissions changed:

su -
ls -la /dev/sd*
brw-r----- 1 root disk 8, 0 Mar 23 13:17 /dev/sda
brw-rw---- 1 grid asmadmin 8, 1 Mar 23 13:17 /dev/sda1
brw-r----- 1 root disk 8, 16 Mar 23 13:17 /dev/sdb
brw-rw---- 1 grid asmadmin 8, 17 Mar 23 13:17 /dev/sdb1

Now we are prepared for creating our ASM instance. Because we are lazy, lets define two variables for the init.ora and the spfile:

su - grid
INITORA=$ORACLE_HOME/dbs/init+ASM.ora
SPFILE=$ORACLE_HOME/dbs/spfile+ASM.ora

Create a minimal set of parameters:

echo "instance_type='asm'
memory_target=128m
memory_max_target=128m
" > ${INITORA}

… and start the cssd resource ( which is the cluster synchronization service, required by ASM ):

crsctl start resource ora.cssd

You should see the following output:

CRS-2672: Attempting to start 'ora.cssd' on 'oracleplayground'
CRS-2672: Attempting to start 'ora.diskmon' on 'oracleplayground'
CRS-2676: Start of 'ora.diskmon' on 'oracleplayground' succeeded
CRS-2676: Start of 'ora.cssd' on 'oracleplayground' succeeded

Let’s try if we can startup and shutdown our ASM instance:

sqlplus /nolog <<EOF
connect / as sysasm
startup force nomount pfile=${INITORA};
shutdown abort;
exit;
EOF

The output should look like:

SQL*Plus: Release 11.2.0.3.0 Production on Fri Mar 23 13:28:29 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
SQL> Connected to an idle instance.
SQL> ASM instance started
Total System Global Area 267227136 bytes
Fixed Size 2227504 bytes
Variable Size 239833808 bytes
ASM Cache 25165824 bytes
SQL> ASM instance shutdown
SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Automatic Storage Management option

Ok, basically we are able to startup our ASM instance. As we want our ASM instance managed by the cluster stack, we will need to create and register some resources.

Start by adding the listener for the ASM instance:

srvctl add listener -l listener_ASM -o $ORACLE_HOME -p 1526

… continue with adding ASM itself:

srvctl add asm -l listener_ASM -p ${INITORA}

… startup listener and ASM:

srvctl start listener -l listener_ASM
srvctl start asm

… and check the resources ( we will use the alias again. you can also type “crsctl status resource -t” ):

crsstat

If you followed the steps, this is what you will see:

--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER_ASM.lsnr
ONLINE ONLINE oracleplayground
ora.asm
ONLINE ONLINE oracleplayground Started
ora.ons
OFFLINE OFFLINE oracleplayground
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE oracleplayground
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE oracleplayground

Two new resources are showing up, the listener and the ASM instance.

Lets create a spfile for the ASM instance:

srvctl stop asm
sqlplus /nolog <<EOF
connect / as sysasm
create spfile='${SPFILE}' from pfile='${INITORA}';
startup force;
show parameter pfile
shutdown abort;
exit;
EOF

You can ignore the “ORA-15110: no disk-groups mounted” message in the output. We will create disk-groups shortly.

As we registered ASM with a pfile in the cluster stack, we need to modify this:

srvctl modify asm -p ${SPFILE}
srvctl start asm
srvctl config asm
crsctl status resource -t
rm -f ${INITORA}

Fine, ASM is now managed by cluster stack. We do not need to worry about shutting down or starting up the instance when the server reboots anymore. And we even do not need to create startup or shutdown scripts for the ASM instance.

ASM without any disks and disk-group does not make much sense. Lets create some:

sqlplus /nolog <<EOF
connect / as sysasm
alter system set asm_diskstring='/dev/sd*' scope=both;
create diskgroup DB112_DATA_DG external redundancy
disk '/dev/sda1' name DB112_DATA_DG_VOL01
attribute
'au_size' = '4M'
, 'compatible.asm' = '11.2'
, 'compatible.rdbms' = '11.2'
;
create diskgroup DB112_ARCH_DG external redundancy
disk '/dev/sdb1' name DB112_ARCH_DG_VOL01
attribute
'au_size' = '4M'
, 'compatible.asm' = '11.2'
, 'compatible.rdbms' = '11.2'
;
exit;
EOF

Work done. You can check the cluster stack for the status of the resources:

crsstat

Notice the disk-groups showing up:

--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DB112_ARCH_DG.dg
ONLINE ONLINE oracleplayground
ora.DB112_DATA_DG.dg
ONLINE ONLINE oracleplayground
ora.LISTENER_ASM.lsnr
ONLINE ONLINE oracleplayground
ora.asm
ONLINE ONLINE oracleplayground Started
ora.ons
OFFLINE OFFLINE oracleplayground
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE oracleplayground
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE oracleplayground

Although it is optional, let’s create some directories for holding the audit logs and diagnostic files. I personally prefer to have it outside the ORACLE_HOME:

su -
mkdir -p /oradata/+ASM
chown grid:oinstall /oradata/+ASM
su - grid
mkdir /oradata/+ASM/admin
mkdir /oradata/+ASM/admin/adump
sqlpa
alter system set audit_file_dest='/oradata/+ASM/admin/adump' scope=spfile;
alter system set diagnostic_dest='/oradata/+ASM/admin' scope=spfile;
exit;
srvctl stop asm -f
srvctl start asm
sqlpa
show parameter audit_file_dest
show parameter diagnostic_dest
exit
echo "ADR_BASE_LISTENER_ASM=/oradata/+ASM/admin" >> /opt/oracle/product/crs/11.2.0.3/network/admin/listener.ora
srvctl stop listener -l listener_asm
srvctl start listener -l listener_asm

What happened: We told the listener and the asm instance to place the traces and logs at the location we want to.

Finally, let’s create an alias for the sqlplus connections:

echo "alias sqlpa='sqlplus '\''/ as sysasm'\'''" >> ~/.bash_profile

If you re-login as grid user, you can now connect the easy way:

sqlpa

As usual ( on your workstation ), if you like, create a snapshot to save your work:

vboxmanage snapshot oracleplayground take "asm running"

For your convenience, here’s the complete script:

#!/bin/bash
if [ $(id -un) != "grid" ]; then
echo "you must run this as root"
exit 0
fi
INITORA=$ORACLE_HOME/dbs/init+ASM.ora
SPFILE=$ORACLE_HOME/dbs/spfile+ASM.ora
echo "instance_type='asm'
memory_target=128m
memory_max_target=128m
" > ${INITORA}
crsctl start resource ora.cssd
sqlplus /nolog <<EOF
connect / as sysasm
startup force nomount pfile=${INITORA};
shutdown abort;
exit;
EOF
srvctl add listener -l listener_ASM -o $ORACLE_HOME -p 1526
srvctl add asm -l listener_ASM -p ${INITORA}
srvctl start listener -l listener_ASM
srvctl start asm
srvctl stop asm -f
crsctl status resource -t
sqlplus /nolog <<EOF
connect / as sysasm
create spfile='${SPFILE}' from pfile='${INITORA}';
startup force;
show parameter pfile
shutdown abort;
exit;
EOF
srvctl modify asm -p ${SPFILE}
srvctl start asm
srvctl config asm
crsctl status resource -t
rm -f ${INITORA}
sqlplus /nolog <<EOF
connect / as sysasm
alter system set asm_diskstring='/dev/sd*' scope=both;
create diskgroup DB112_DATA_DG external redundancy
disk '/dev/sda1' name DB112_DATA_DG_VOL01
attribute
'au_size' = '4M'
, 'compatible.asm' = '11.2'
, 'compatible.rdbms' = '11.2'
;
create diskgroup DB112_ARCH_DG external redundancy
disk '/dev/sdb1' name DB112_ARCH_DG_VOL01
attribute
'au_size' = '4M'
, 'compatible.asm' = '11.2'
, 'compatible.rdbms' = '11.2'
;
exit;
EOF

The OS is prepared and everything is configured. Time to come to the first part of the installation: Oracle Grid Infrastructure

If you did not already transfer the source files to your Virtual Machine do it now. If your workstation runs on linux, here is an example:

scp -P 3000 p10404530_112030_Linux-x86-64_1of7.zip grid@localhost:/opt/oracle/stage
scp -P 3000 p10404530_112030_Linux-x86-64_2of7.zip grid@localhost:/opt/oracle/stage
scp -P 3000 p10404530_112030_Linux-x86-64_3of7.zip grid@localhost:/opt/oracle/stage
scp -P 3000 p13348650_112030_LINUX.zip grid@localhost:/opt/oracle/stage

Windows users may use tools like winscp, filezilla or cygwin to name a few of it.

As the Grid Infrastructure will be installed under the “grid” user, establish a ssh connection to your Virtual Machine and do the installation ( we will do a software only installation here, nothing will be configured for the moment ):

su - grid
cd /opt/oracle/stage
unzip p10404530_112030_Linux-x86-64_3of7.zip
cd grid
./runInstaller \
INVENTORY_LOCATION=/opt/oracle/oraInventory \
SELECTED_LANGUAGES=en \
oracle.install.option=CRS_SWONLY \
ORACLE_BASE=/opt/oracle/product/base \
ORACLE_HOME=/opt/oracle/product/crs/11.2.0.3 \
ORACLE_HOME_NAME=HCRS112030 \
UNIX_GROUP_NAME=oinstall \
oracle.install.asm.OSDBA=asmdba \
oracle.install.asm.OSASM=asmadmin \
FROM_LOCATION=../stage/products.xml \
DECLINE_SECURITY_UPDATES=true \
-ignoreSysPrereqs \
-ignorePrereq \
-waitforcompletion \
-silent
su -
/opt/oracle/oraInventory/orainstRoot.sh
/opt/oracle/product/crs/11.2.0.3/root.sh

That’s it for the software installation. Surprised ? The most important thing to remember if installing oracle software is: Read the installation guides and fulfill all the requirements. If you do this, the installation will be fast and easy.

Some explanations for the above commands:

• runInstaller: this is oracle installer on linux/unix ( it is setup.exe on windows )
• INVENTORY_LOCATION: the location where you want to store the central oracle inventory
• SELECTED_LANGUAGES: the languages you want to install for the product
• oracle.install.option: how do you want to install ( CRS_SWONLY is for installing the software, but do not configure anything )
• ORACLE_BASE: the location you want to use for the $ORACLE_BASE
• ORACLE_HOME: the location where you want to install the product, this is the $ORACLE_HOME
• UNIX_GROUP_NAME: the OS group you want to own the installed software
• oracle.install.asm.OSDBA: the OS group you want to use as asm dba
• oracle.install.asm.OSASM: the OS group you want to use for sysasm
• FROM_LOCATION: references the xml where all the products are defined
• DECLINE_SECURITY_UPDATES: true means: do not try to connect to oracle support for downloading the latest patches
• -ignoreSysPrereqs: ignore the system prerequisites
• -ignorePrereq: ignore all the other prerequisites
• -waitforcompletion: wait until the installation completes before returning to the shell
• -silent: do a silent installation ( do not use the graphical mode )

Take a look at the sample response file if you want to learn about all the other options you can use ( /opt/oracle/stage/grid/response/grid_install.rsp ).

The root scripts:

• /opt/oracle/oraInventory/orainstRoot.sh : Creates the pointer to the oraInventory and sets the correct permissions
• /opt/oracle/product/crs/11.2.0.3/root.sh: Calls various other scripts to set up the installation

Lets see if if everything works as expected and configure the cluster stack for single node ( The Grid Infrastructure sofware is exactly the same for Cluster and Single Node installations. For single node installations, oracle calls this “Oracle Restart” ):

su -
/opt/oracle/product/crs/11.2.0.3/crs/install/roothas.pl

The output should like this:

[root@oracleplayground stage]# /opt/oracle/product/crs/11.2.0.3/crs/install/roothas.pl
Using configuration parameter file: /opt/oracle/product/crs/11.2.0.3/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node oracleplayground successfully pinned.
Adding Clusterware entries to inittab
oracleplayground 2012/03/23 09:18:15 /opt/oracle/product/crs/11.2.0.3/cdata/oracleplayground/backup_20120323_091815.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server

If you had troubles you can deconfigure the whole stack and try again:

su -
/opt/oracle/product/crs/11.2.0.3/crs/install/roothas.pl -deconfig -force
### check the logfiles ( $ORACLE_HOME/log/oracleplayground/* ) for the issue and try to configure again
/opt/oracle/product/crs/11.2.0.3/crs/install/roothas.pl

There is one important thing to note: Take a look at this line:

Using configuration parameter file: /opt/oracle/product/crs/11.2.0.3/crs/install/crsconfig_params

This is the one and only configuration file you will need to adjust if setting up the stack for a cluster ( If you want to configure for a cluster rootcrs.pl will be used instead of roothas.pl, but the configuration file keeps the same ).
In our case we do not need to modify it, as we will work with a single node.

What happened here:
The script created same trace directories, the olr ( oracle local registry ), pinned the node and added a call to inittab. Can verify this with:

cat /etc/inittab | grep ohas

If you take a look at your processes now, you’ll notice that several oracle processes started up:

ps -ef | grep ora

This are the basic cluster stack processes which will manage you oracle infrastructure.

• ohasd.bin: the process resonsible for starting up all the cluster processes
• oraagent.bin: will spawn agents to start resources
• evmd.bin: the event volume manager daemon
• evmlogger.bin: spawned by evmd to log events

To make a life a bit easier for us, lets define some aliases and environment variables for the grid user:

su - grid
echo "ORACLE_HOME=/opt/oracle/product/crs/11.2.0.3
ORACLE_BASE=/opt/oracle/base
ORACLE_SID=+ASM
PATH=$ORACLE_HOME/bin:$PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
PS1='\[33[1;34m\]${ORACLE_SID}\[33[1;33m\]@\[33[1;32m\]`uname -n`\[33[0m\] $LOGNAME:$PWD $ '
export ORACLE_HOME ORACLE_BASE ORACLE_SID PATH LD_LIBRARY_PATH PS1
alias crsstat='crsctl status resource -t'" >> ~/.bash_profile

It is a good idea to reboot your Virtual Machine now and to check if the cluster stack comes up again.

su -
reboot

Once the Virtual Machine is up and running again ( notice that it will take some time for the oracle processes to come up ), verify if you can interact with the stack ( we will use the alias defined above for this ):

su - grid
crsstat

If everything is fine, the output should like this:

--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ons
OFFLINE OFFLINE oracleplayground
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 OFFLINE OFFLINE
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE oracleplayground

To summarize: We installed the Grid Infrastructure software without doing any configuration initially and executed the root scripts for setting up permissions, the oracle inventory and the grid infrastructure software. roothas.pl did the configuration of the cluster stack, added an entry to the inittab and generated the oracle local registry. We did a reboot to check if everything comes up again and defined some environment variables and aliases to make life a bit easier for us.

If you’ll ask yourself what the crsctl is for: We will come back to all the commands in next posts and introduce them when appropriate.

In the next post we will apply the first patch to the grid infrastructure and will learn about the oracle patch utility opatch.

Again, remember to take a snapshot of your Virtual Machine if you want to save your work or if you want to revert to this point. Delete the source files if you do not want to become your Virtual Machine too big before:

su - grid
cd /opt/oracle/stage/
rm -rf grid/ p10404530_112030_Linux-x86-64_3of7.zip

On your workstation create the snapshot:

vboxmanage snapshot oracleplayground take "after gi installation and configuration"