Archives For November 30, 1999

edbs ppas comes with an oracle compatibility layer. in this and some future posts I’ll take a look at what this layer is about and what you can do with it.

there are four parameters which control the behaviour of the oracle compatibility layer:

  1. edb_redwood_date
  2. edb_redwood_raw_names
  3. edb_redwood_strings
  4. edb_stmt_level_tx

lets check the catalog for these parameters and see if we can change them on the fly or if we need to restart the database server:

select name,setting,context 
      from pg_settings 
     where name in ('edb_redwood_date'
                   ,'edb_redwood_raw_names'
                   ,'edb_redwood_strings'
                   ,'edb_stmt_level_tx');

         name          | setting | context 
-----------------------+---------+---------
 edb_redwood_date      | on      | user
 edb_redwood_raw_names | off     | user
 edb_redwood_strings   | on      | user
 edb_stmt_level_tx     | off     | user

fine, all have the context “user” which means we can change them without restarting the server (btw: the above settings are the default if ppas is installed in oracle compatibility mode).

what is edb_redwood_date about?

in postgres, if you specify a column as date there is no time information. setting this parameter to “on” tells the server to use a timestamp instead of pure date data type whenever date is specified for a column. in oracle a column of type date includes the time component, too.

lets switch it to off for now:

edb=# alter system set edb_redwood_date=off;
ALTER SYSTEM
edb=# select pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

edb=# select name,setting,context from pg_settings where name = 'edb_redwood_date';
       name       | setting | context 
------------------+---------+---------
 edb_redwood_date | off     | user
(1 row)

and now lets create a simple table with a date column and insert one row:

edb=# create table t1 ( a date );
CREATE TABLE
edb=# \d t1
  Table "enterprisedb.t1"
 Column | Type | Modifiers 
--------+------+-----------
 a      | date | 

edb=# insert into t1 values (date '2014-01-01');
INSERT 0 1
edb=# select * from t1;
     a     
-----------
 01-JAN-14
(1 row)

no time component available. now switch edb_redwood_date to “on”:

edb=# alter system set edb_redwood_date=on;
ALTER SYSTEM
edb=# select pg_reload_conf();
 pg_reload_conf 
----------------
 t
(1 row)

edb=# select name,setting,context from pg_settings where name = 'edb_redwood_date';
       name       | setting | context 
------------------+---------+---------
 edb_redwood_date | on      | user
(1 row)

… and create another table with type date for the column and do the same insert:

edb=# create table t2 ( a date );
CREATE TABLE
edb=# insert into t2 values ( date '2014-01-01');
INSERT 0 1
edb=# select * from t2;
         a          
--------------------
 01-JAN-14 00:00:00
(1 row)

here we go. the time component is now included. but how is this possible? the server created the column with type “timestamp (0)” on the fly:

edb=# \d t2
             Table "enterprisedb.t2"
 Column |            Type             | Modifiers 
--------+-----------------------------+-----------
 a      | timestamp without time zone | 

what is edb_redwood_raw_names about?

when oracle compatibilty mode is used various oracle catalog views are accessible in ppas, e.g;

edb=# select viewname from pg_views where viewname like 'dba%' limit 5;
    viewname     
-----------------
 dba_tables
 dba_users
 dba_constraints
 dba_all_tables
 dba_triggers
(5 rows)

now lets create two tables while edb_redwood_raw_names is set to its default (false/off):

edb=# create table TEST1 (A int);
CREATE TABLE
edb=# create table test2 (a int);
CREATE TABLE

both of these tables are displayed in upper case when looking at the oracle catalog views:

edb=# select table_name from dba_tables where table_name in  ('TEST1','TEST2');
 table_name 
------------
 TEST1
 TEST2
(2 rows)

setting edb_redwood_raw_names to true/on changes this behaviour:

edb=# set edb_redwood_raw_names to on;
SET
edb=# show edb_redwood_raw_names;
 edb_redwood_raw_names 
-----------------------
 on
(1 row)
edb=# create table TEST3 (A int);
CREATE TABLE
edb=# create table test4 (a int);
CREATE TABLE
edb=# select table_name from dba_tables where table_name in  ('TEST3','TEST4');
 table_name 
------------
(0 rows)

edb=# select table_name from dba_tables where table_name in  ('test3','test4');
 table_name 
------------
 test3
 test4
(2 rows)

what is edb_redwood_strings about?

edb_redwood_strings controls concatenation of strings. in plain postgres, if a string is concatenated with null the result is null:

edb=# show edb_redwood_strings;
 edb_redwood_strings 
---------------------
 on
(1 row)

edb=# set edb_redwood_strings to off;
SET
edb=# show edb_redwood_strings;      
 edb_redwood_strings 
---------------------
 off
(1 row)

edb=# select 'aaaa'||null;
 ?column? 
----------
 
(1 row)

in oracle the behaviour is the other way around. if a string is concatenated with null the result is the original string:

edb=# set edb_redwood_strings to on;
SET
edb=# show edb_redwood_strings;
 edb_redwood_strings 
---------------------
 on
(1 row)

edb=# select 'aaaa'||null;
 ?column? 
----------
 aaaa
(1 row)

what is edb_stmt_level_tx about?

this is all about “statement level transaction isolation”, which is the default behaviour in oracle. lets set up a little test case to demonstrate this:

edb=# create table t1 ( a int, b varchar(3) );
CREATE TABLE
edb=# alter table t1 add constraint chk_b check ( b in ('aaa','bbb'));
ALTER TABLE
edb=# create table t2 ( c int, d date );
CREATE TABLE
edb=# alter table t2 add constraint chk_c check ( c in (1,2,3));
ALTER TABLE

the default setting for edb_stmt_level_tx is off:

edb=# show edb_stmt_level_tx;
 edb_stmt_level_tx 
-------------------
 off
(1 row)

lets insert some rows in the tables and let the last insert fail (autocommit needs to be off as each statement commits automatically otherwise):

edb=# set autocommit off;
edb=# show autocommit;      
autocommit OFF
edb=# insert into t1 (a,b) values (1,'aaa');
INSERT 0 1
edb=# insert into t1 (a,b) values (2,'aaa');
INSERT 0 1
edb=# insert into t2 (c,d) values (1,sysdate);
INSERT 0 1
edb=# insert into t2 (c,d) values (5,sysdate);
ERROR:  new row for relation "t2" violates check constraint "chk_c"
DETAIL:  Failing row contains (5, 24-FEB-15 10:52:29).

if we now do a commit, which rows are there?

edb=# commit;
ROLLBACK
edb=# select * from t1;
 a | b 
---+---
(0 rows)

edb=# select * from t2;
 c | d 
---+---
(0 rows)

all gone (notice the ROLLBACK after the commit statement). lets witch edb_stmt_level_tx to on and repeat the test:

edb=# set edb_stmt_level_tx to on;                                                                                                                      
SET
edb=# show edb_stmt_level_tx; 
 edb_stmt_level_tx 
-------------------
 on
(1 row)

edb=# show autocommit;
autocommit OFF
edb=# insert into t1 (a,b) values (1,'aaa');
INSERT 0 1
edb=# insert into t1 (a,b) values (2,'aaa');
INSERT 0 1
edb=# insert into t2 (c,d) values (1,sysdate);
INSERT 0 1
edb=#  insert into t2 (c,d) values (5,sysdate);
ERROR:  new row for relation "t2" violates check constraint "chk_c"
DETAIL:  Failing row contains (5, 24-FEB-15 10:55:52).
edb=# commit;
COMMIT
edb=# select * from t1;
 a |  b  
---+-----
 1 | aaa
 2 | aaa
(2 rows)

edb=# select * from t2;
 c |         d          
---+--------------------
 1 | 24-FEB-15 10:55:49
(1 row)

now all the rows are there except for the failing one. this is what edb_stmt_level_tx is about. by default postgresql rolls back everything since the start of the transaction. when switching edb_stmt_level_tx to on only the failing statement is rolled back. in addition, if edb_stmt_level_tx is set to off you can not continue the transaction until either commit or rollback is issued:

edb=# set edb_stmt_level_tx to off;
SET
edb=# insert into t1 (a,b) values (1,'aaa');
INSERT 0 1
edb=# insert into t1 (a,b) values (2,'aaa');
INSERT 0 1
edb=# insert into t2 (c,d) values (1,sysdate);
INSERT 0 1
edb=# insert into t2 (c,d) values (5,sysdate);
ERROR:  new row for relation "t2" violates check constraint "chk_c"
DETAIL:  Failing row contains (5, 24-FEB-15 11:06:52).
edb=# insert into t2 (c,d) values (2,sysdate);
ERROR:  current transaction is aborted, commands ignored until end of transaction block

as json as a datatype for databases is becoming more and more popular here’s a quick example on how to convert a traditional table design to tables containing jsonb in postgres:

-- traditional design
drop schema if exists traditional cascade;
create schema traditional;
create table traditional.customers ( id serial
                                   , name varchar(20)
                                   , active boolean
                                   , country varchar(2)
                                   , phone varchar(20)
                                   , email varchar(50)
                                   );
alter table traditional.customers add constraint customers_pk primary key(id);

create table traditional.orders ( id serial
                                , customer_id int
                                , order_date timestamp with time zone
                                , delivery_date timestamp with time zone
                                );
alter table traditional.orders add constraint orders_pk 
    primary key(id);
alter table traditional.orders add constraint orders_ref_customers 
    foreign key (customer_id) 
    references traditional.customers(id);

DO
$$
BEGIN
   for i in 1..100 loop
     insert into traditional.customers ( name,active,country
                                       , phone,email )
            values ( 'name'||i, case when mod(i,5) = 0 then true else false end
                   , case when mod(i,3) = 0 then 'CH' else 'DE' end
                   , i, i||'@'||i||'.com' );
            for e in 1..10 loop
              insert into traditional.orders ( customer_id, order_date
                                             , delivery_date )
                     values (i, current_timestamp + interval '5 days'
                            , current_timestamp + interval '7 days' );
            end loop;
   end loop;
END
$$;

-- json design 
drop schema if exists jsonschema cascade;
create schema jsonschema;
create table jsonschema.customers ( id serial
                                  , customer_data jsonb
                                  );
alter table jsonschema.customers add constraint customers_pk 
      primary key(id);

create table jsonschema.orders ( id serial
                               , customer_id int
                               , order_data jsonb
                               );
alter table jsonschema.orders add constraint orders_pk primary key(id);
alter table jsonschema.orders add constraint orders_ref_customers 
     foreign key (customer_id) 
     references traditional.customers(id);

insert into jsonschema.customers ( customer_data )
        ( select row_to_json(cust)::jsonb 
            from ( select name, active, country, phone, email
                     from traditional.customers 
                 ) cust 
        );

with tt (id,order_date,delivery_date) as (
  select id,order_date, delivery_date
    from traditional.orders
  order by id )
,    dd (id,customer_id) as (
  select id, customer_id 
    from traditional.orders
  order by id )
insert into jsonschema.orders ( customer_id, order_data )
   select dd.customer_id 
        , row_to_json(tt)::jsonb 
     from dd, tt
    where dd.id = tt.id;

comparing the different approaches:

postgres=# select * from traditional.customers limit 2;
 id | name  | active | country | phone |  email  
----+-------+--------+---------+-------+---------
  1 | name1 | f      | DE      | 1     | 1@1.com
  2 | name2 | f      | DE      | 2     | 2@2.com
(2 rows)

postgres=# select * from jsonschema.customers limit 2;
 id |                                     customer_data                                     
----+---------------------------------------------------------------------------------------
  1 | {"name": "name1", "email": "1@1.com", "phone": "1", "active": false, "country": "DE"}
  2 | {"name": "name2", "email": "2@2.com", "phone": "2", "active": false, "country": "DE"}

this is merely a documenation post for myself as I always forgot the steps to get this working. as postgres plus advanced server 9.4 was released some days ago we wanted to do another poc for an oracle migration. using edbmtk was clearly the preferred way to do this as it automates most of the tasks. but how did we need to set this up the last time?

as a first step one needs to download the oracle jdbc driver for the java version available on the postgres server.

example:
ojdbc6.jar – for use with java 6
ojdbc7.jar – for use with java 7

put one of these under:

ls -la /etc/alternatives/jre/lib/ext/
total 5264
drwxr-xr-x.  2 root root    4096 Feb 12 12:54 .
drwxr-xr-x. 11 root root    4096 Jan 26 18:26 ..
-rw-r--r--.  1 root root   10075 Jan  9 02:39 dnsns.jar
-rw-r--r--.  1 root root  452904 Jan  9 02:48 gnome-java-bridge.jar
-rw-r--r--.  1 root root  558461 Jan  9 02:40 localedata.jar
-rw-r--r--.  1 root root     427 Jan  9 02:45 meta-index
-rw-r--r--.  1 root root 3698857 Feb 12 12:54 ojdbc7.jar
-rw-r--r--.  1 root root   69699 Jan  9 02:46 pulse-java.jar
-rw-r--r--.  1 root root  225679 Jan  9 02:41 sunjce_provider.jar
-rw-r--r--.  1 root root  259918 Jan  9 02:39 sunpkcs11.jar
-rw-r--r--.  1 root root   78194 Jan  9 02:42 zipfs.jar

the next step is to configure the toolkit.properties file:

cat /opt/PostgresPlus/edbmtk/etc/toolkit.properties
SRC_DB_URL=jdbc:oracle:thin:@[ORACLE_SERVER]:[LISTENER_PORT]:[DATABASE]
SRC_DB_USER=system
SRC_DB_PASSWORD=manager

TARGET_DB_URL=jdbc:edb://localhost:5432/[POSTGRES_DATABASE]
TARGET_DB_USER=postgres
TARGET_DB_PASSWORD=postgres

… and then kickoff the migration:

cd /opt/PostgresPlus/edbmtk/bin
./runMTK.sh -fastCopy -logBadSQL -fetchSize 10000 -loaderCount 6 -dropSchema true -useOraCase ORACLE_SCHEMA1,ORACLE_SCHEMA2,...

pretty easy. wait for edbmtk to finish and start fixing the objects that are invalid :)

btw: for migrations to pure community postgres take a look at ora2pg

see the release notes for details

While browsing http://planet.postgresql.org/ today I came across a link to a nice paper which is definitely worth reading:

Architecture of a Database System

today I browsed the red news and came over this: Keep an eye on these 5 new features in RHEL 7. I did know about systemd, docker, xfs. I don’t care about AD integration, at least currently. but what is Performance Co-Pilot?

quickly checked the documentation and it seemed pretty interesting. especially that there is a plugin for postgres. so, lets take a look (short intro, only :) ):

for my tests a quick setup of postgres using the sample makefile posted some while ago is sufficient (do not use a development snapshot as pcp does not support this. I used 9.3.5 for the tests):

yum install -y wget readline-devel bzip2 zlib-devel
groupadd postgres
useradd -g postgres postgres
su - postgres
-- get makefile
make fromscratch
install/bin/psql
psql (9.3.5)
Type "help" for help.

so far, so good. lets get the pcp packages:

yum install pcp pcp-gui

enable and start pcp:

chkconfig pmcd on
chkconfig --list | grep pmcd
/etc/init.d/pmcd start

easy. let’s see if pmatop works:

pmatop

pmatop

great. pminfo tells you what metrics are available currently:

pminfo -f
...
kernel.percpu.interrupts.SPU
    inst [0 or "cpu0"] value 0

kernel.percpu.interrupts.LOC
    inst [0 or "cpu0"] value 747300
...

a lot of stuff but nothing directly related to postgres except some information about the processes:

pminfo -f | grep -i postgres
...
    inst [12665 or "012665 /home/postgres/install/bin/postgres -D /home/postgres/data"] value 12665
    inst [12667 or "012667 postgres: checkpointer process   "] value 12667
    inst [12668 or "012668 postgres: writer process   "] value 12668
    inst [12669 or "012669 postgres: wal writer process   "] value 12669
    inst [12670 or "012670 postgres: autovacuum launcher process   "] value 12670
    inst [12671 or "012671 postgres: stats collector process   "] value 12671
...

according to the documentation pmdapostgres “Extracts performance metrics from the PostgreSQL relational database”. reading over and over again it became clear what to do. all these tools need to be installed first, so:

cd /var/lib/pcp/pmdas/postgresql
./Install 
Perl database interface (DBI) is not installed

ok:

yum install -y perl-DBI

next try:

./Install 
Postgres database driver (DBD::Pg) is not installed

gr:

yum install perl-DBD-Pg

and again:

./Install 
You will need to choose an appropriate configuration for installation of
the "postgresql" Performance Metrics Domain Agent (PMDA).

  collector	collect performance statistics on this system
  monitor	allow this system to monitor local and/or remote systems
  both		collector and monitor configuration for this system

Please enter c(ollector) or m(onitor) or b(oth) [b] b
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Waiting for pmcd to terminate ...
Starting pmcd ... 
Check postgresql metrics have appeared ... 15 warnings, 208 metrics and 0 values

much better. lets see if something is available now:

pminfo -f | grep -i postgres
...
postgresql.stat.all_tables.last_vacuum
postgresql.stat.all_tables.n_dead_tup
postgresql.stat.all_tables.seq_scan
postgresql.stat.all_tables.last_autoanalyze
postgresql.stat.all_tables.schemaname
postgresql.stat.all_tables.n_live_tup
postgresql.stat.all_tables.idx_tup_fetch
...

cool. can I get some values?

pminfo -f postgresql.stat.all_tables.n_tup_upd

postgresql.stat.all_tables.n_tup_upd
No value(s) available!

hm. not really what I expected. looking at the logfile:

tail -100 /var/log/pcp/pmcd/postgresql.log
...
DBI connect('dbname=postgres','postgres',...) failed: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? at /var/lib/pcp/pmdas/postgresql/pmdapostgresql.pl line 252.

ok, seems connections to postgresql are not possible. the issue is that DBI looks for the sockets at “/var/run/postgresql/”. checking my postgresql.conf:

#unix_socket_directories = '/tmp'	# comma-separated list of directories
#unix_socket_group = ''			# (change requires restart)
#unix_socket_permissions = 0777		# begin with 0 to use octal notation

easy to fix:

su -
mkdir /var/run/postgresql/
chown postgres:postgres /var/run/postgresql/
su - postgres
echo "unix_socket_directories = '/var/run/postgresql/'" >> postgresql.conf

restart postgresql:

install/bin/pg_ctl stop -D data/
install/bin/pg_ctl start -D data/

checking again:

pminfo -f postgresql.stat.all_tables.n_tup_upd

postgresql.stat.all_tables.n_tup_upd
    inst [1261 or "pg_auth_members"] value 0
    inst [2617 or "pg_operator"] value 0
    inst [2600 or "pg_aggregate"] value 0
    inst [1136 or "pg_pltemplate"] value 0
    inst [12529 or "sql_implementation_info"] value 0
    inst [2609 or "pg_description"] value 0
    inst [2612 or "pg_language"] value 0
    inst [12539 or "sql_packages"] value 0
    inst [3601 or "pg_ts_parser"] value 0
    inst [3466 or "pg_event_trigger"] value 0
    inst [3592 or "pg_shseclabel"] value 0
    inst [3118 or "pg_foreign_table"] value 0

much better. but not really nice to read. this is where pmchart comes into the game:

pmchart &

metric selection
real time graphs

now performance data can be viewed in realtime…really cool.

if you want to play with the latest postgresql development snapshot here is a very simple makefile which does the work for you ( given that you have installed all the dependencies required for building postgresql ):

PGBASEVER=9.3
PG=http://ftp.postgresql.org/pub/snapshot/dev/postgresql-snapshot.tar.bz2
PGFILE=postgresql-snapshot.tar.bz2
CURRDIR=$(shell pwd)
DATADIR=$(CURRDIR)/data

fromscratch: reset download buildpg initdb startdb

reset: stopdb
        rm -rf build
        rm -rf install
        rm -rf data
        mkdir build
        mkdir install
        mkdir data

download:
        wget ${PG}
        mv ${PGFILE} build/

buildpg:
        ( cd build && tar -axf ${PGFILE} )
        ( cd build/postgresql-${PGBASEVER}* && ./configure --prefix=${CURRDIR}/install )
        ( cd build/postgresql-${PGBASEVER}* && make )
        ( cd build/postgresql-${PGBASEVER}* && make check )
        ( cd build/postgresql-${PGBASEVER}* && make install )

initdb:
        ( cd install/bin && ./initdb -D ${DATADIR} )

startdb:
        ( install/bin/pg_ctl -D ${DATADIR} start ) 

stopdb:
        if [ -f ${DATADIR}/postmaster.pid ]; then \
                ( install/bin/pg_ctl -D ${DATADIR} stop -m fast ) \
        fi

copy this to a directory where you are the owner of, name the file “Makefile” and execute:

make fromscratch

this will create the directories, download the latest snapshot, compile the source and start the postgresql database ( this takes 4 minutes on my mint workstation ). once the script finished you may connect to the database and start playing:

install/bin/psql postgres
psql (9.3devel)
Type "help" for help.

postgres=# \h create materialized view
Command:     CREATE MATERIALIZED VIEW
Description: define a new materialized view
Syntax:
CREATE [ UNLOGGED ] MATERIALIZED VIEW table_name
    [ (column_name [, ...] ) ]
    [ WITH ( storage_parameter [= value] [, ... ] ) ]
    [ TABLESPACE tablespace_name ]
    AS query
    [ WITH [ NO ] DATA ]

postgres=# 

of course you may put this in a shell script, too. but this way it was more fun :)

if you read the title you might think there is not much to say about. but if you come from the oracle world there is actually a huge difference in how postgresql handles updates in contrast to oracle.

let’s start with a simple table:

create table t1 ( a int );
insert into t1 (a) values (1);
insert into t1 (a) values (2);
insert into t1 (a) values (3);

for pointing out the differences I’ll use an extension called pageinspect. for making use of this extension you’ll have to create it:

create extension pageinspect;

this will create a couple of functions which assist in inspecting database pages:

\dx+ *inspect*
    Objects in extension "pageinspect"
            Object Description            
------------------------------------------
 function bt_metap(text)
 function bt_page_items(text,integer)
 function bt_page_stats(text,integer)
 function fsm_page_contents(bytea)
 function get_raw_page(text,integer)
 function get_raw_page(text,text,integer)
 function heap_page_items(bytea)
 function page_header(bytea)

coming back to the test table which contains three rows ( or tuples in postgresql wording ) right now. making use of the heap_page_items and get_raw_page functions of the pageinspect extension we may display some internals of the table’s page:

select lp, lp_len, t_xmin
     , t_xmax, lp_off, t_ctid 
  from heap_page_items(get_raw_page('t1', 0));
 lp | lp_len | t_xmin | t_xmax | lp_off | t_ctid 
----+--------+--------+--------+--------+--------
  1 |     28 |   9271 |      0 |   8160 | (0,1)
  2 |     28 |   9272 |      0 |   8128 | (0,2)
  3 |     28 |   9273 |      0 |   8096 | (0,3)

not going too much into the details right now this tells us that page zero of table t1 contains three rows. this is consistent with the inserts from above. what happens if we update one of the rows ?

update t1 set a=4 where a=1;

from an oracle perspective the answer will be clear: the row in question will be read, updated and written back once committed ( although it is the whole block or page that is read and written rather than the individual row ). oracle uses the undo data to create the old row version for others if they request the row before the commit.

let’s see what postgresql did:

SELECT lp, lp_len, t_xmin
     , t_xmax, lp_off, t_ctid 
  from heap_page_items(get_raw_page('t1', 0));
 lp | lp_len | t_xmin | t_xmax | lp_off | t_ctid 
----+--------+--------+--------+--------+--------
  1 |     28 |   9271 |   9274 |   8160 | (0,4)
  2 |     28 |   9272 |      0 |   8128 | (0,2)
  3 |     28 |   9273 |      0 |   8096 | (0,3)
  4 |     28 |   9274 |      0 |   8064 | (0,4)

ups, four rows/tuples now. what happened ? the first point you might have noticed is the t_xmax that got set for the first row. t_xmin is the transaction id that did the insert of the rows/tuples. t_xmax is the transaction that will get set if a tuple/row either gets updated or deleted. this actually means that transactions after t_xmax will not see this row/tuple. this is how postgresql implements multiversion concurrency control ( mvcc ). in other words: transactions below 9274 will see the first row, transaction greater 9274 will not see the first row anymore.

what actually happened is a delete of the first row and an insert of the fourth row and that’s the difference between oracle and postgresql. in postgresql an update is a delete followed by an insert. you might wonder what happens to the deleted row as it is still present in the page. that’s were vacuum kicks in:

vacuum verbose t1;
INFO:  vacuuming "public.t1"
INFO:  "t1": found 1 removable, 3 nonremovable row versions in 1 out of 1 pages
DETAIL:  0 dead row versions cannot be removed yet.
There were 0 unused item pointers.
0 pages are entirely empty.
CPU 0.00s/0.00u sec elapsed 0.00 sec.
VACUUM
(sysdba@[local]:1540)[ppas92] > SELECT lp, lp_len, t_xmin
                                        , t_xmax, lp_off, t_ctid 
                                     from heap_page_items(get_raw_page('t1', 0));
 lp | lp_len | t_xmin | t_xmax | lp_off | t_ctid 
----+--------+--------+--------+--------+--------
  1 |      0 |        |        |      4 | 
  2 |     28 |   9272 |      0 |   8160 | (0,2)
  3 |     28 |   9273 |      0 |   8128 | (0,3)
  4 |     28 |   9274 |      0 |   8096 | (0,4)

vacuum takes care of the expired rows and deletes the pointers. if you did not disable the autovacuum process the database takes care of this on its own and you do not need to worry about it.
one point to remember is that the space for the deleted row only will get released if you do a vacuum full:

vacuum full verbose t1;
INFO:  vacuuming "public.t1"
INFO:  "t1": found 0 removable, 3 nonremovable row versions in 1 pages
DETAIL:  0 dead row versions cannot be removed yet.
CPU 0.00s/0.00u sec elapsed 0.02 sec.
VACUUM
(sysdba@[local]:1540)[ppas92] > SELECT lp, lp_len, t_xmin
                                     , t_xmax, lp_off, t_ctid 
                                  from heap_page_items(get_raw_page('t1', 0));
 lp | lp_len | t_xmin | t_xmax | lp_off | t_ctid 
----+--------+--------+--------+--------+--------
  1 |     28 |   9290 |      0 |   8160 | (0,1)
  2 |     28 |   9291 |      0 |   8128 | (0,2)
  3 |     28 |   9292 |      0 |   8096 | (0,3)

otherwise the space will just get reused when appropriate.

in postgresql there is the “create role” command to create both: a role and a user. in oracle there is “create user” and “create role”. but in the background it is almost the same:

SQL> create role my_role;

Role created.

SQL> create user u identified by u;

User created.

SQL> select name, type# from user$ where name in ('MY_ROLE','U');

NAME				    TYPE#
------------------------------ ----------
MY_ROLE 				0
U					1

it is the type, that matters.

as it is with oracle postgresql relies on various statistics to produce the explain plan for a query. the catalog table which stores the statistical information is pg_statistic.

for the purpose of this post let’s play with a simple table and see what gets stored in the pg_statistic table.

drop table if exists t1;
create table t1 ( a int );

I will always use the same query for producing the output from the pg_statistics table:

select pa.attname
     , ps.stawidth
     , ps.stadistinct
     , ps.stanullfrac
     , ps.stavalues1
     , ps.stavalues2
     , ps.stavalues3
     , ps.stavalues4
     , ps.stavalues5
  from pg_class pc
     , pg_statistic ps
     , pg_attribute pa
 where pc.relname = 't1'
   and pc.relowner = ( select nspowner 
                         from pg_namespace
                        where nspname = 'public' )
   and pc.oid = ps.starelid
   and pa.attnum = ps.staattnum
   and pa.attrelid = pc.oid;

If you run the query right now it will not return any rows as pg_statistics gets populated once you run analyze and at least one row is inserted into the table ( but not before ).

--exec statistics-query from above:
 attname | stawidth | stadistinct | stanullfrac | stavalues1 | stavalues2 | stavalues3 | stavalues4 | stavalues5 
---------+----------+-------------+-------------+------------+------------+------------+------------+------------
(0 rows)

So let’s create a row and see what’s happening:

insert into t1 (a) values (1);
analyze t1;
--exec statistics-query from above:
 attname | stawidth | stadistinct | stanullfrac | stavalues1 | stavalues2 | stavalues3 | stavalues4 | stavalues5 
---------+----------+-------------+-------------+------------+------------+------------+------------+------------
 a       |        4 |          -1 |           0 |      |      |      |      | 
(1 row)

what does this tell ?

  • the column “a” is reported to have a width of 4 which corresponds to the definition of the integer data-type.
  • distinct values is reported as “-1” which seems a little but surprising for the moment as it should be 1, shouldn’t it ? in fact this is equal to one as it tells that there is statistically one occurrence for the value
  • the number of null values is zero, of course

let’s create another row and have a look at the results:

insert into t1 (a) values (2);
analyze t1;
--exec statistics-query from above:
 attname | stawidth | stadistinct | stanullfrac | stavalues1 | stavalues2 | stavalues3 | stavalues4 | stavalues5 
---------+----------+-------------+-------------+------------+------------+------------+------------+------------
 a       |        4 |          -1 |           0 | {1,2}      |      |      |      | 

the only column that changed is the stavalues1 column. as soon as there is more than one row in the table this column gets populated. distinct values is still reported as “-1” as each row in the table has a different value. this will not change as long as you insert unique values and not too much of them ( more on his later ):

insert into t1 (a) values (generate_series(3,10));
analyze t1;
--exec statistics-query from above:
 attname | stawidth | stadistinct | stanullfrac |       stavalues1       | stavalues2 | stavalues3 | stavalues4 | stavalues5 
---------+----------+-------------+-------------+------------------------+------------+------------+------------+------------
 a       |        4 |          -1 |           0 | {1,2,3,4,5,6,7,8,9,10} |      |      |      | 
(1 row)

once you insert the same values again the stadistinct column gets updated to reflect the change in the data distribution:

insert into t1 (a) values (generate_series(1,10));
analyze t1;
--exec statistics-query from above:
 attname | stawidth | stadistinct | stanullfrac |       stavalues1       | stavalues2 | stavalues3 | stavalues4 | stavalues5 
---------+----------+-------------+-------------+------------------------+------------+------------+------------+------------
 a       |        4 |        -0.5 |           0 | {1,2,3,4,5,6,7,8,9,10} |      |      |      | 

now there is a value of “-0.5” which essentially tells that there are about 2 entries for each distinct value. we can do the same again and the stadistinct value decreases again to match data ( around three occurrences for each distinct value ):

insert into t1 (a) values (generate_series(1,10));
analyze t1;
--exec statistics-query from above:
 attname | stawidth | stadistinct | stanullfrac |       stavalues1       | stavalues2 | stavalues3 | stavalues4 | stavalues5 
---------+----------+-------------+-------------+------------------------+------------+------------+------------+------------
 a       |        4 |   -0.333333 |           0 | {1,2,3,4,5,6,7,8,9,10} |      |      |      | 
(1 row)

as it looks right now postgresql stores all the values of a column another time in the pg_statistic catalog table. as this would be bad practice let’s if this changes once we insert more data:

insert into t1 (a) values (generate_series(11,10000));
analyze t1;
--exec statistics-query from above:
 attname | stawidth | stadistinct | stanullfrac |       stavalues1       |        stavalues2    | stavalues3 | stavalues4 | stavalues5 
---------+----------+-------------+-------------+------------------------+----------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------+---
---------+------------+------------
 a       |        4 |   -0.998004 |           0 | {1,2,3,4,5,6,7,8,9,10} | {11,110,210,310,410,510,610,710,810,910,1009,1109,1209,1309,1409,1509
,1609,1709,1809,1908,2008,2108,2208,2308,2408,2508,2608,2708,2807,2907,3007,3107,3207,3307,3407,3507,3607,3706,3806,3906,4006,4106,4206,4306,440
6,4506,4605,4705,4805,4905,5005,5105,5205,5305,5405,5504,5604,5704,5804,5904,6004,6104,6204,6304,6403,6503,6603,6703,6803,6903,7003,7103,7203,73
02,7402,7502,7602,7702,7802,7902,8002,8102,8201,8301,8401,8501,8601,8701,8801,8901,9001,9100,9200,9300,9400,9500,9600,9700,9800,9900,10000} |      |      | 
(1 row)

now we get the next stavalues column populated in steps of a hundred. the first stavalues columns remains unchanged in this example.

in oracle words the stavaluesN columns are similar to a histogram. it’s a statistical representation of the data distribution which is then used to generate execution plans. more to come …