• /
    • Archives /
  • November 30, 1999

Archives For November 30, 1999

shared memory – interprocess communication (2)

April 12, 2012 — 2 Comments

as mentioned in the previous post about semaphores there are more things to consider when it comes to interprocess communication. as semaphores are used to protect critical regions, there must be some critical regions to protect and this is the shared memory oracle uses for its communication.

to give an example on how the shared memory addressing works we will take a look at what happens when the database starts up.
for this you’ll need two sessions to a test infrastructure ( one as the database owner, the other as root ).

session one ( oracle ):
connect to sqlplus as sysdba make sure you shutdown the database ( do not exit sqlplus once the database is down ):

sqlplus / as sysdba
shutdown immediate

session two ( root ): discover the PID for then sqlplus session above …

ps -ef | grep sqlp
oracle    3062  3036  0 09:49 pts/1    00:00:00 sqlplus

… check the shared memory segments and trace the sqlplus PID from above:

ipcs -m
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x7401003e 1310720    root      600        4          0                       
0x74010014 1998849    root      600        4          0                       
0x00000000 2359298    root      644        80         2                       
0x74010013 1966083    root      600        4          0                       
0x00000000 2392068    root      644        16384      2                       
0x00000000 2424837    root      644        280        2                       
0x00000000 2490374    grid      640        4096       0                       
0x00000000 2523143    grid      640        4096       0                       
0x8e11371c 2555912    grid      640        4096       0
# start the trace
strace -o db_startup.log -fp 3062

it is important to specify the “-f” flag for the strace call. this will tell strace to follow the child processes spawned.

in session one startup the database…

startup

… and stop the tracing in the root session once the database is up and re-check the shared memory segments.

ipcs -m
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x7401003e 1310720    root      600        4          0                       
0x74010014 1998849    root      600        4          0                       
0x00000000 2359298    root      644        80         2                       
0x74010013 1966083    root      600        4          0                       
0x00000000 2392068    root      644        16384      2                       
0x00000000 2424837    root      644        280        2                       
0x00000000 2490374    grid      640        4096       0                       
0x00000000 2523143    grid      640        4096       0                       
0x8e11371c 2555912    grid      640        4096       0                       
0x00000000 3538953    oracle    640        4096       0                       
0x00000000 3571722    oracle    640        4096       0                       
0x3393b3a4 3604491    oracle    640        4096       0

as you can see, three more segments appeared after the database started up.

you’ll probably noticed some trace output on the screen similar to this:

Process 3468 detached
Process 3470 attached (waiting for parent)
Process 3470 resumed (parent 3409 ready)
Process 3471 attached (waiting for parent)
Process 3471 resumed (parent 3470 ready)
Process 3469 detached
Process 3470 detached

this is because of the “-f” flag given to strace.
the complete trace output is now available in the db_startup.log trace file and we are ready to take a look at it.

the first thing that catches the eye are the various references to the “/proc” filesystem. in may trace file there are 1213 calls to it. you can check this with:

grep "/proc/" db_startup.log | wc -l

take a look at the previous post which introduces the “/proc” filesystem for more information. for the scope of this post just notice how much depends on it.

the actual startup of the database is triggered by the following line:

execve("/opt/oracle/product/base/11.2.0.3/bin/oracle", ["oracleDB112", "(DESCRIPTION=(LOCAL=YES)(ADDRESS"], [/* 22 vars */]) = 0

this is the call to the oracle binary ( execve executes the binary ) with 22 arguments omitted. from now on the oracle instance starts up.

the calls important to the shared memory stuff are the following:

  • brk: changes a data segment’s size
  • mmap, munmap: maps/unmaps files or devices into memory
  • mprotect: sets protection on a region of memory
  • shmget: allocates a shared memory segment
  • shmat, shmdt: performs attach/detach operations on shared memory
  • get_mempolicy: return NUMA memory policies for a process
  • semget: get a semaphore identifier
  • semctl: perform control operations on a semaphore
  • semop, semtimedop: perform sempahore operations

for each of the above commands you can check the man-pages for more information.
as the trace file is rather large and a lot of things are happening i will focus on the minimum ( this is not about re-engineering oracle :) ):

let’s check the keys returned by the ipcs command above:

egrep "3538953|3571722|3604491" db_startup.log
...
5365  shmget(IPC_PRIVATE, 4096, IPC_CREAT|IPC_EXCL|0640) = 3538953
5365  shmget(IPC_PRIVATE, 4096, IPC_CREAT|IPC_EXCL|0640) = 3571722
5365  shmget(0x3393b3a4, 4096, IPC_CREAT|IPC_EXCL|0640) = 3604491
...

as you can see the identifiers returned by the shmget call ( 3604491,3571722,3538953 ) correspond to the ones reported by ipcs. you wonder about the size of 4096 bytes ? this is because memory_target/memory_max_target is in use by the instance. if the database is configured using sga_target/sga_max_target you would see the actual size. let’s check this:

su - oracle
sqlplus / as sysdba
alter system reset memory_max_target scope=spfile;
alter system reset memory_target scope=spfile;
alter system set sga_max_size=256m scope=spfile;
alter system set sga_target=256m scope=spfile;
alter system set pga_aggregate_target=24m scope=spfile;
startup force;
exit;
# re-check the shared memory segments
ipcs -m
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 2359298    root      644        80         2                       
0x00000000 2392068    root      644        16384      2                       
0x00000000 2424837    root      644        280        2                       
0x00000000 2490374    grid      640        4096       0                       
0x00000000 2523143    grid      640        4096       0                       
0x8e11371c 2555912    grid      640        4096       0                       
0x00000000 3801097    oracle    640        8388608    25                      
0x00000000 3833866    oracle    640        260046848  25                      
0x3393b3a4 3866635    oracle    640        2097152    25

the “260046848” corresponds to the sga size of 256m and the nattch column shows that 25 processes are attached to it. you can double check the 25
attached processes if you want:

ps -ef | grep DB112 | grep -v LISTENER | grep -v grep | wc -l

let’s return to the memory_target/memory_max_target configuration. as oracle puts together all the memory junks ( pga and sga ) the management of memory changes to the virtual shared memory filesystem ( tmpfs ). unfortunately this is not visible with the ipcs command.
but you can map your memory_* sizes to the shm filesystem:

ls -la /dev/shm/ | grep -v "+ASM"
total 466100
drwxrwxrwt  2 root   root        2640 Apr 10 13:09 .
drwxr-xr-x 10 root   root        3400 Apr 10 09:44 ..
-rw-r-----  1 oracle asmadmin 4194304 Apr 10 13:27 ora_DB112_3932169_0
-rw-r-----  1 oracle asmadmin 4194304 Apr 10 13:09 ora_DB112_3932169_1
-rw-r-----  1 oracle asmadmin 4194304 Apr 10 13:09 ora_DB112_3964938_0
-rw-r-----  1 oracle asmadmin 4194304 Apr 10 13:20 ora_DB112_3964938_1
-rw-r-----  1 oracle asmadmin 4194304 Apr 10 13:09 ora_DB112_3964938_10
-rw-r-----  1 oracle asmadmin 4194304 Apr 10 13:20 ora_DB112_3964938_11
-rw-r-----  1 oracle asmadmin 4194304 Apr 10 13:10 ora_DB112_3964938_12

note that i have excluded the ASM stuff here. in my case each segment ( or granule ) is 4mb of size ( this depends on the avaible memory of the system ) and the sum of all the segments should get you near to your memory_* configuration.

as ipcs can not tell you much here there are other commands to use. if you want to know which process has a memory granule open:

fuser -v /dev/shm/ora_DB112_4358154_49
                     USER        PID ACCESS COMMAND
/dev/shm/ora_DB112_4358154_49:
                     oracle     6626 ....m oracle
                     oracle     6628 ....m oracle
                     oracle     6630 ....m oracle
                     oracle     6634 ....m oracle
                     oracle     6636 ....m oracle
                     oracle     6638 ....m oracle
                     oracle     6640 ....m oracle
                     oracle     6642 ....m oracle
                     oracle     6644 ....m oracle
                     oracle     6646 ....m oracle
                     oracle     6648 ....m oracle
                     oracle     6650 ....m oracle
                     oracle     6652 ....m oracle
                     oracle     6654 ....m oracle
                     oracle     6656 ....m oracle
                     oracle     6658 ....m oracle
                     oracle     6662 ....m oracle
                     oracle     6669 ....m oracle
                     oracle     6744 ....m oracle
                     oracle     6767 ....m oracle
                     oracle     6769 ....m oracle
                     oracle     6791 ....m oracle
                     oracle     7034 ....m oracle

or the other way around, if you want to know which files are opened by a specific process:

ps -ef | grep pmon | grep -v "ASM"
oracle    6626     1  0 13:40 ?        00:00:05 ora_pmon_DB112
root      7075  5338  0 14:33 pts/0    00:00:00 grep pmon
# use the pmap command on the PID
pmap 6626
6626:   ora_pmon_DB112
0000000000400000 183436K r-x--  /opt/oracle/product/base/11.2.0.3/bin/oracle
000000000b922000   1884K rwx--  /opt/oracle/product/base/11.2.0.3/bin/oracle
000000000baf9000    304K rwx--    [ anon ]
0000000010c81000    660K rwx--    [ anon ]
0000000060000000      4K r-xs-  /dev/shm/ora_DB112_4325385_0
0000000060001000   4092K rwxs-  /dev/shm/ora_DB112_4325385_0
0000000060400000   4096K rwxs-  /dev/shm/ora_DB112_4325385_1
0000000060800000   4096K rwxs-  /dev/shm/ora_DB112_4358154_0
0000000060c00000   4096K rwxs-  /dev/shm/ora_DB112_4358154_1
0000000061000000   4096K rwxs-  /dev/shm/ora_DB112_4358154_2
0000000061400000   4096K rwxs-  /dev/shm/ora_DB112_4358154_3
0000000061800000   4096K rwxs-  /dev/shm/ora_DB112_4358154_4
0000000061c00000   4096K rwxs-  /dev/shm/ora_DB112_4358154_5
0000000062000000   4096K rwxs-  /dev/shm/ora_DB112_4358154_6
...

if you have troubles starting up your instance with this configuration ( ORA-00845 ) check the size of the virtual filesystem:

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hdc1              28G   14G   12G  54% /
tmpfs                 741M  456M  286M  62% /dev/shm

depending on your configuration ( memory_* or sga_* parameters ) the way that memory is managed changes ( from System V to POSIX, to be exact ).

lots and lots of information. not all of it is important to keep in mind. but what you should remember:
there are several processes and memory segments that make up the oracle instance. as several processes are attached to the same memory regions there must be a way to protect them from concurrent access ( think of semaphores ) … and oracle heavily depends on shared memory. if you scroll through the trace file you’ll notice that there are thousands of operations going on when an oracle instance starts up. imagine what is going on if the instance is under heavy workload and lots and lots of things need protection.

ps: for those interested:

there is plenty of more interesting stuff which you can find in the db_startup.log trace, for example:

writing the audit files:

grep -i adump db_startup.log  | grep -v ASM
3404  open("/oradata/DB112/admin/adump/DB112_ora_3404_2.aud", O_RDWR|O_CREAT|O_EXCL, 0660) = 10
3404  write(10, "/oradata/DB112/admin/adump/DB112"..., 47) = 47
3444  open("/oradata/DB112/admin/adump/DB112_ora_3444_1.aud", O_RDWR|O_CREAT|O_EXCL, 0660) = -1 EEXIST (File exists)
3444  open("/oradata/DB112/admin/adump/DB112_ora_3444_2.aud", O_RDWR|O_CREAT|O_EXCL, 0660) = 8
3444  write(8, "/oradata/DB112/admin/adump/DB112"..., 47) = 47
3481  open("/oradata/DB112/admin/adump/DB112_ora_3481_1.aud", O_RDWR|O_CREAT|O_EXCL, 0660 
3481  write(8, "/oradata/DB112/admin/adump/DB112"..., 47) = 47

writing the alert.log:

grep -i "alert_DB112.log" db_startup.log
3404  lstat("/oradata/DB112/admin/diag/rdbms/db112/DB112/trace/alert_DB112.log", {st_mode=S_IFREG|0640, st_size=110201, ...}) = 0
3404  open("/oradata/DB112/admin/diag/rdbms/db112/DB112/trace/alert_DB112.log", O_WRONLY|O_CREAT|O_APPEND, 0660) = 5
3404  lstat("/oradata/DB112/admin/diag/rdbms/db112/DB112/trace/alert_DB112.log", {st_mode=S_IFREG|0640, st_size=110260, ...}) = 0
3404  open("/oradata/DB112/admin/diag/rdbms/db112/DB112/trace/alert_DB112.log", O_WRONLY|O_CREAT|O_APPEND, 0660) = 11

reading the oracle message files:

grep msb db_startup.log
db_startup.log:5438  open("/opt/oracle/product/base/11.2.0.3/oracore/mesg/lrmus.msb", O_RDONLY) = 18
db_startup.log:5438  open("/opt/oracle/product/base/11.2.0.3/oracore/mesg/lrmus.msb", O_RDONLY) = 18
db_startup.log:5430  open("/opt/oracle/product/base/11.2.0.3/rdbms/mesg/oraus.msb", O_RDONLY 
db_startup.log:5494  open("/opt/oracle/product/base/11.2.0.3/rdbms/mesg/oraus.msb", O_RDONLY

getting sempahores:

grep semget db_startup.log 
5365  semget(IPC_PRIVATE, 1, IPC_CREAT|IPC_EXCL|0600) = 1081346
5365  semget(IPC_PRIVATE, 124, IPC_CREAT|IPC_EXCL|0666) = 1114114
5365  semget(IPC_PRIVATE, 124, IPC_CREAT|0660) = 1146882
5365  semget(0x710dfe10, 0, 0)          = -1 ENOENT (No such file or directory)
5365  semget(0x46db3f80, 0, 0)          = -1 ENOENT (No such file or directory)
5365  semget(0x9ae46084, 0, 0)          = -1 ENOENT (No such file or directory)
5365  semget(0xf6dcc368, 0, 0)          = -1 ENOENT (No such file or directory)
5365  semget(0x710dfe10, 124, IPC_CREAT|IPC_EXCL|0640) = 1179650

some exadata stuff:

3404  open("/etc/oracle/cell/network-config/cellinit.ora", O_RDONLY) = -1 ENOENT (No such file or directory)

and … and …

In main , ,
permalink

sema… what ? interprocess communication (1)

April 9, 2012 — Leave a comment

if you followed the oracle installation guide there are some kernel parameters to be configured for the oracle database. one of them specified by four values is about semaphores:

  • semmsl: the maximum number of semaphores per semaphore set
  • semmns: the maximum number of semaphore of the entire system
  • semopm: number of maximum operations per semop call
  • semmni: the maximum number of semaphore sets of the entire system

the question is: what are these semaphores about and what are they for?

 

a semaphore is a counter associated with a data structure which provides locking and synchronization of critical regions. there is one semaphore ( initialized to 1 ) for each data structure to be protected. the atomic methods “down” and “up” are used to decrease and increase the counter. if the kernel wants access to a protected structure it executes the “down” method and if the result is not negative ( the counter is equal or greater than zero ) access to the resource is granted. if the counter is negative the process which wishes to access the resource is blocked and added to the sempahore list ( a kind of queue ). as time goes by some process finishes its work and executes the “up” method which allows one process in the semaphore list to proceed.

in linux there are two kinds of semaphores:

  • kernel semaphores ( for kernel control paths )
  • system V IPC semaphores ( for user mode processes ), IPC stands for “interprocess communication”

the IPC semaphores are the ones relevant to the oracle database. semaphores are created by the function semget() which returns the semaphore identifier. there are two other functions for creating ipc resources, which are:

  • msgget(): which is for message queues
  • shmget(): which is for shared memory

there must be at least one semaphore for each oracle process ( the processes parameter of the database ). as each session to the database needs to be synchronized with other sessions ( and sessions are memory structures ) oracle must request resources from the operation system to be able to handle concurrency in the sga to which all sessions are connected to.

the semaphores queue of pending requests is implemented as a double linked list. you remember? the same concepts over and over again. actually semaphores are sometimes called mutexes ( and there are internal functions like init_MUTEX )… surprised ?

to display the current limits the following command can be used:

ipcs -ls
------ Semaphore Limits --------
max number of arrays = 1024
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

or you can directly query the /proc filesystem:

cat /proc/sys/kernel/sem
250 32000 32 1024

to check the currently allocated semaphores:

ipcs -s
------ Semaphore Arrays --------
key semid owner perms nsems
0x127f81f8 163842 oracle 640 124
0x3d2c0d44 1933315 oracle 640 129
0x3d2c0d45 1966084 oracle 640 129
0x3d2c0d46 1998853 oracle 640 129
0x3d2c0d47 2031622 oracle 640 129
0x3d2c0d48 2064391 oracle 640 129

if you want to see some semaphore operations in action do, for example, a strace on the smon process and wait one or two seconds:

ps -ef | grep dbw
oracle 2723 1 0 08:31 ? 00:00:03 ora_smon_dbs300
root 3153 3111 2 09:04 pts/1 00:00:00 grep smon
strace -p 2723
...
semtimedop(819203, {{17, -1, 0}}, 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
gettimeofday({1333697562, 861932}, NULL) = 0
semtimedop(819203, {{17, -1, 0}}, 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
gettimeofday({1333697565, 871767}, NULL) = 0
semtimedop(819203, {{17, -1, 0}}, 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
gettimeofday({1333697568, 893455}, NULL) = 0
semtimedop(819203, {{17, -1, 0}}, 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
gettimeofday({1333697571, 905050}, NULL) = 0
semtimedop(819203, {{17, -1, 0}}, 1, {3, 0}) = -1 EAGAIN (Resource temporarily unavailable)
gettimeofday({1333697574, 920094}, NULL) = 0
...

here you can clearly see calls to semtimedop which return with -1 ( EAGAIN, the call expired ).

if you followed the series on how to setup a test infrastructure or you have a test infrastructure available to play with here is a little demonstration:

be sure to save your current kernel semaphore settings:

cat /proc/sys/kernel/sem
250 32000 32 1024

minimize the settings to very low values and try to restart ( or start ) the oracle instance:

su-
echo "2 10 1 2" > /proc/sys/kernel/sem

if you write the values to the /proc/sys/kernel/…-parameter files the values will be in effect immediately. be sure what you are doing if you’re touching these parameters.

startup or restart the oracle instance:

su - oracle
sqlplus / as sysdba
startup force
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpbitsper

what happened ? the database was not able to allocate the resources ( sempahores ) it needs, thus can not start up. the message “No space left on device” indicates some issue with disk space, but it’s actually the memory in question here. fyi: the allocation of semaphores ( and shared memory ) does only occur during the “startup nomount” phase. this is the only time oracle requests this resources.

fix it, and try again:

su -
echo "250 32000 32 1024" > /proc/sys/kernel/sem
su - oracle
startup force

conclusion: semaphores are one critical part for interprocess communication and provide a way for locking and synchronization. if the database can not allocate the resources it needs from the operating system it will fail to start up.

the values recommended in the installation guides are minimum values. if your session count to the database grows you may need to adjust the semaphore parameters.

In main , , ,
permalink

linux and its processes

April 4, 2012 — Leave a comment

as mentioned in earlier posts as a dba you need to know how the operating system works. this post is an introduction to processes on linux.

the definition of process is: a process is an instance of a program in execution.
to manage a process the linux kernel must know a lot of things about the process, e.g. which files the process is allowed to handle, if it is running on CPU or blocked, the address space of the process etc. all this information is present in the so called process descriptor. you can think of the process descriptor as a strcuture containing all the information the kernel needs to know about the process ( internally the structure is called: task_structure ). on of the information stored in the process descriptor is the process id which is used to identify the process.

let’s take a look at the processes that make up the oracle database:

ps -ef | grep $ORACLE_SID | egrep -v "DESCRIPTION|grep|tnslsnr" 
oracle    2944     1  0 08:30 ?        00:00:03 ora_pmon_DB112
oracle    2946     1  0 08:30 ?        00:00:06 ora_psp0_DB112
oracle    2948     1  2 08:30 ?        00:01:11 ora_vktm_DB112
oracle    2952     1  0 08:30 ?        00:00:02 ora_gen0_DB112
oracle    2954     1  0 08:30 ?        00:00:03 ora_diag_DB112
oracle    2956     1  0 08:30 ?        00:00:02 ora_dbrm_DB112
oracle    2958     1  0 08:30 ?        00:00:06 ora_dia0_DB112
oracle    2961     1  0 08:30 ?        00:00:02 ora_mman_DB112
oracle    2963     1  0 08:30 ?        00:00:03 ora_dbw0_DB112
oracle    2965     1  0 08:30 ?        00:00:03 ora_lgwr_DB112
oracle    2967     1  0 08:30 ?        00:00:06 ora_ckpt_DB112
oracle    2969     1  0 08:30 ?        00:00:01 ora_smon_DB112
oracle    2971     1  0 08:30 ?        00:00:00 ora_reco_DB112
oracle    2973     1  0 08:30 ?        00:00:02 ora_rbal_DB112
oracle    2975     1  0 08:30 ?        00:00:01 ora_asmb_DB112
oracle    2977     1  0 08:30 ?        00:00:05 ora_mmon_DB112
oracle    2979     1  0 08:30 ?        00:00:09 ora_mmnl_DB112
oracle    2987     1  0 08:30 ?        00:00:04 ora_mark_DB112
oracle    3013     1  0 08:30 ?        00:00:00 ora_qmnc_DB112
oracle    3054     1  0 08:30 ?        00:00:01 ora_q000_DB112
oracle    3056     1  0 08:30 ?        00:00:00 ora_q001_DB112
oracle    3182     1  0 08:35 ?        00:00:02 ora_smco_DB112
oracle    3359     1  0 09:05 ?        00:00:00 ora_w000_DB112

notice that the grep command excluded the listener process and all the current connections to the database.
if you want to check the local connections connections to the database from the os, you can do something like this:

ps -ef | grep $ORACLE_SID | grep "LOCAL=YES"
oracle    2933     1  0 10:48 ?        00:00:01 oracleDB112 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    2969     1  0 10:48 ?        00:00:00 oracleDB112 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    2979     1  0 10:48 ?        00:00:00 oracleDB112 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    3088  3087  0 10:50 ?        00:00:00 oracleDB112 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle    3096     1  0 10:50 ?        00:00:00 oracleDB112 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

checking the processes from inside the database would be as simple as this ( for the background processes ):

SQL> select pname from v$process where pname is not null;
PNAME
-----
PMON
PSP0
VKTM
GEN0
DIAG
DBRM
DIA0
MMAN
DBW0
LGWR
CKPT
SMON
RECO
RBAL
ASMB
MMON
MMNL
MARK
SMCO
W000
QMNC
Q000
Q001

with the above arguments ( -ef ) supplied to the ps command, the columns displayed are:

  • the os-user the process runs under
  • the process id
  • the parent process id
  • processor utilization
  • start time of the process
  • the terminal the process was started on ( if any )
  • the cumulative CPU time
  • the command

but where does the ps command get the information to display from ? in fact you can get all of the information displayed above without using the ps command. all you need to do is to check the pseudo filesystem /proc ( it is called a pseudo filesystem because it is a virtual filesystem that maps to the kernel structures ).

if you do a “ls” on the proc filesystem you’ll see a lot of directories and files. for this post we will concentrate on the numbered directories which map to process ids.

let’s take smon as an example, which is the oracle system monitor ( you will need to adjust the process-id for your environment ):

ls -la /proc/2969/
dr-xr-xr-x   6 oracle asmadmin 0 Apr  3 13:40 .
dr-xr-xr-x 154 root   root     0 Apr  3  2012 ..
dr-xr-xr-x   2 oracle asmadmin 0 Apr  3 14:07 attr
-r--------   1 root   root     0 Apr  3 14:07 auxv
-r--r--r--   1 root   root     0 Apr  3 13:45 cmdline
-rw-r--r--   1 root   root     0 Apr  3 14:07 coredump_filter
-r--r--r--   1 root   root     0 Apr  3 14:07 cpuset
lrwxrwxrwx   1 root   root     0 Apr  3 14:07 cwd -> /opt/oracle/product/base/11.2.0.3/dbs
-r--------   1 root   root     0 Apr  3 14:07 environ
lrwxrwxrwx   1 root   root     0 Apr  3 14:07 exe -> /opt/oracle/product/base/11.2.0.3/bin/oracle
dr-x------   2 root   root     0 Apr  3 13:40 fd
dr-x------   2 root   root     0 Apr  3 14:07 fdinfo
-r--------   1 root   root     0 Apr  3 14:07 io
-r--r--r--   1 root   root     0 Apr  3 14:07 limits
-rw-r--r--   1 root   root     0 Apr  3 14:07 loginuid
-r--r--r--   1 root   root     0 Apr  3 13:40 maps
-rw-------   1 root   root     0 Apr  3 14:07 mem
-r--r--r--   1 root   root     0 Apr  3 14:07 mounts
-r--------   1 root   root     0 Apr  3 14:07 mountstats
-r--r--r--   1 root   root     0 Apr  3 14:07 numa_maps
-rw-r--r--   1 root   root     0 Apr  3 14:07 oom_adj
-r--r--r--   1 root   root     0 Apr  3 14:07 oom_score
lrwxrwxrwx   1 root   root     0 Apr  3 14:07 root -> /
-r--r--r--   1 root   root     0 Apr  3 14:07 schedstat
-r--r--r--   1 root   root     0 Apr  3 14:07 smaps
-r--r--r--   1 root   root     0 Apr  3 13:40 stat
-r--r--r--   1 root   root     0 Apr  3 14:07 statm
-r--r--r--   1 root   root     0 Apr  3 13:45 status
dr-xr-xr-x   3 oracle asmadmin 0 Apr  3 14:07 task
-r--r--r--   1 root   root     0 Apr  3 14:07 wchan

what do we see here? lots and lots of information of the smon process. for a detailed description of what all the files and directories are about, you can go to the man-pages:

man proc

for example, if we take a look at the statm file of the process:

cat /proc/2969/statm
126385 16013 14717 45859 0 994 0

… and check the man pages for the meaning of the numbers, things are getting clearer:

/proc/[number]/statm
      Provides information about memory status in pages.  The columns are:
               size       total program size
               resident   resident set size
               share      shared pages
               text       text (code)
               lib        library
               data       data/stack
               dt         dirty pages (unused in Linux 2.6)

wanting to know the environment of the process? just take a look at the environ file:

cat /proc/2969/environ 
__CLSAGFW_TYPE_NAME=ora.listener.typeORA_CRS_HOME=/opt/oracle/product/crs/11.2.0.3SELINUX_INIT=YESCONSOLE=/dev/consoleTERM=linuxSHELL=/bin/bash__CRSD_CONNECT_STR=(ADDRESS=(PROTOCOL=IPC)(KEY=OHASD_IPC_SOCKET_11))NLS_LANG=AMERICAN_AMERICA.AL32UTF8CRF_HOME=/opt/oracle/product/crs/11.2.0.3GIPCD_PASSTHROUGH=false__CRSD_AGENT_NAME=/opt/oracle/product/crs/11.2.0.3/bin/oraagent_grid__CRSD_MSG_FRAME_VERSION=2USER=gridINIT_VERSION=sysvinit-2.86__CLSAGENT_INCARNATION=2ORASYM=/opt/oracle/product/crs/11.2.0.3/bin/oraagent.binPATH=RUNLEVEL=3runlevel=3PWD=/ENV_FILE=/opt/oracle/product/crs/11.2.0.3/crs/install/s_crsconfig_oracleplayground_env.txtLANG=en_US.UTF-8TZ=Europe/Zurich__IS_HASD_AGENT=TRUEPREVLEVEL=Nprevious=N__CLSAGENT_LOG_NAME=ora.listener.type_gridHOME=/home/gridSHLVL=3__CLSAGENT_LOGDIR_NAME=ohasdLD_ASSUME_KERNEL=__CLSAGENT_USER_NAME=gridLOGNAME=gridORACLE_HOME=/opt/oracle/product/base/11.2.0.3ORACLE_SID=DB112ORA_NET2_DESC=34,37ORACLE_SPAWNED_PROCESS=1SKGP_SPAWN_DIAG_PRE_FORK_TS=1333453218SKGP_SPAWN_DIAG_POST_FORK_TS=1333453218SKGP_HIDDEN_ARGS=0SKGP_SPAWN_DIAG_PRE_EXEC_TS=1333453218[root@oracleplayground 2642]#

… which files were opened by the process ?:

ls -la fd/
total 0
dr-x------ 2 root   root      0 Apr  3 13:40 .
dr-xr-xr-x 6 oracle asmadmin  0 Apr  3 13:40 ..
lr-x------ 1 root   root     64 Apr  3 16:23 0 -> /dev/null
l-wx------ 1 root   root     64 Apr  3 16:23 1 -> /dev/null
lr-x------ 1 root   root     64 Apr  3 16:23 10 -> /dev/null
lr-x------ 1 root   root     64 Apr  3 16:23 11 -> /dev/null
lr-x------ 1 root   root     64 Apr  3 16:23 12 -> /dev/null
lrwx------ 1 root   root     64 Apr  3 16:23 13 -> /opt/oracle/product/base/11.2.0.3/dbs/hc_DB112.dat
lr-x------ 1 root   root     64 Apr  3 16:23 14 -> /dev/null
lr-x------ 1 root   root     64 Apr  3 16:23 15 -> /dev/null
lr-x------ 1 root   root     64 Apr  3 16:23 16 -> /dev/zero
lr-x------ 1 root   root     64 Apr  3 16:23 17 -> /dev/zero
lrwx------ 1 root   root     64 Apr  3 16:23 18 -> /opt/oracle/product/base/11.2.0.3/dbs/hc_DB112.dat
lr-x------ 1 root   root     64 Apr  3 16:23 19 -> /opt/oracle/product/base/11.2.0.3/rdbms/mesg/oraus.msb
l-wx------ 1 root   root     64 Apr  3 16:23 2 -> /dev/null
lr-x------ 1 root   root     64 Apr  3 16:23 20 -> /proc/2875/fd
lr-x------ 1 root   root     64 Apr  3 16:23 21 -> /opt/oracle/product/crs/11.2.0.3/dbs/hc_+ASM.dat
lr-x------ 1 root   root     64 Apr  3 16:23 22 -> /dev/zero
lrwx------ 1 root   root     64 Apr  3 16:23 23 -> /opt/oracle/product/base/11.2.0.3/dbs/hc_DB112.dat
lrwx------ 1 root   root     64 Apr  3 16:23 24 -> /opt/oracle/product/base/11.2.0.3/dbs/lkDB112
lr-x------ 1 root   root     64 Apr  3 16:23 25 -> /opt/oracle/product/base/11.2.0.3/rdbms/mesg/oraus.msb
lrwx------ 1 root   root     64 Apr  3 16:23 256 -> /dev/sda1
lrwx------ 1 root   root     64 Apr  3 16:23 3 -> /opt/oracle/product/crs/11.2.0.3/log/oracleplayground/agent/ohasd/oraagent_grid/oraagent_gridOUT.log
l-wx------ 1 root   root     64 Apr  3 16:23 4 -> /opt/oracle/product/crs/11.2.0.3/log/oracleplayground/agent/ohasd/oraagent_grid/oraagent_grid.l01
lr-x------ 1 root   root     64 Apr  3 16:23 5 -> /dev/null
lrwx------ 1 root   root     64 Apr  3 16:23 6 -> socket:[7791]
lrwx------ 1 root   root     64 Apr  3 16:23 7 -> socket:[7792]
lrwx------ 1 root   root     64 Apr  3 16:23 8 -> socket:[7793]
lrwx------ 1 root   root     64 Apr  3 16:23 9 -> socket:[7794]

conclusion: it’s really worth to read the man pages and understand the /proc/[PID] structures. this can be a very good starting point if you have troubles with one of the processes running on your system.

and last but not least: maybe you don’t believe that the ps command is reading the /proc/[PID] structures to diplay it’s information. you can always trace the commands and check what’s happening behind:

strace -o strace.log ps -ef

this will write the strace output to a file named strace.log. grep for you smon process and check which files were read:

grep 2969 strace.log
stat("/proc/2969", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/2969/stat", O_RDONLY)       = 6
read(6, "2969 (oracle) S 1 2921 2921 0 -1"..., 1023) = 191
open("/proc/2969/status", O_RDONLY)     = 6
open("/proc/2969/cmdline", O_RDONLY)    = 6
write(1, "oracle    2969     1  0 10:48 ? "..., 63) = 63

here we go: a subset of the same files listed above:

/proc/2969/stat
/proc/2969/status
/proc/2969/cmdline

happy processing …

In main ,
permalink

Master Note of Linux OS Requirements for Database Server

April 4, 2012 — Leave a comment

if you have access to oracle support there is a good note about deploying the database on linux.

check it out:

Master Note of Linux OS Requirements for Database Server

In smalltalk ,
permalink

little linux helpers – watch

April 3, 2012 — Leave a comment

did you ever had the situation where you’d want to quickly monitor how a filesystem grows or what gets written to it without typing the commands again and again ?

some examples:

watch "ls -latr"
Every 2.0s: ls -latr Tue Apr 3 08:29:48 2012
total 60
-rw-r--r-- 1 oracle oinstall 515 Mar 23 08:41 .emacs
-rw-r--r-- 1 oracle oinstall 124 Mar 23 08:41 .bashrc
-rw-r--r-- 1 oracle oinstall 33 Mar 23 08:41 .bash_logout
drwxr-xr-x 4 root root 4096 Mar 23 08:41 ..
drwxr-xr-x 3 oracle oinstall 4096 Mar 26 14:34 oradiag_oracle
-rw-r--r-- 1 oracle oinstall 685 Mar 26 14:36 .bash_profile
-rw------- 1 oracle oinstall 1802 Mar 28 13:06 .viminfo
drwx------ 3 oracle oinstall 4096 Mar 28 13:06 .
-rw------- 1 oracle oinstall 6626 Mar 28 16:36 .bash_history

Without providing any additional arguments watch displays the command one provides and refreshes the output every two seconds.

Another example:
watch -d -n 5 -t "ls -latr"
total 60
-rw-r--r-- 1 oracle oinstall 515 Mar 23 08:41 .emacs
-rw-r--r-- 1 oracle oinstall 124 Mar 23 08:41 .bashrc
-rw-r--r-- 1 oracle oinstall 33 Mar 23 08:41 .bash_logout
drwxr-xr-x 4 root root 4096 Mar 23 08:41 ..
drwxr-xr-x 3 oracle oinstall 4096 Mar 26 14:34 oradiag_oracle
-rw-r--r-- 1 oracle oinstall 685 Mar 26 14:36 .bash_profile
-rw------- 1 oracle oinstall 1802 Mar 28 13:06 .viminfo
-rw------- 1 oracle oinstall 6626 Mar 28 16:36 .bash_history
-rw-r--r-- 1 oracle oinstall 0 Apr 3 08:34 a
drwx------ 3 oracle oinstall 4096 Apr 3 08:34 .

“-d” highlightes the differences since the last refresh
“-n” specifies the refresh interval
“-t” supresses the heading

grepping?
watch -t "grep aaa" bbb

happy watching…

In quick&dirty ,
permalink

summary on the “create your test infrastrucure” series

March 28, 2012 — Leave a comment

To summarize all the posts which will get you to your test infrastructure, here is the overview:

  1. Operating System setup
  2. Operating System preperation for oracle
  3. Grid Infrastrucuture Installation
  4. Applying the Latest Patchset Update for the Grid Infrastructure
  5. Creating the ASM instance
  6. Database Software Installation
  7. Creating the database
  8. Applying the Latest Patchset Update for the database
In smalltalk ,
permalink