the performance co-pilot?

today I browsed the red news and came over this: Keep an eye on these 5 new features in RHEL 7. I did know about systemd, docker, xfs. I don’t care about AD integration, at least currently. but what is Performance Co-Pilot?

quickly checked the documentation and it seemed pretty interesting. especially that there is a plugin for postgres. so, lets take a look (short intro, only :) ):

for my tests a quick setup of postgres using the sample makefile posted some while ago is sufficient (do not use a development snapshot as pcp does not support this. I used 9.3.5 for the tests):

yum install -y wget readline-devel bzip2 zlib-devel
groupadd postgres
useradd -g postgres postgres
su - postgres
-- get makefile
make fromscratch
install/bin/psql
psql (9.3.5)
Type "help" for help.

so far, so good. lets get the pcp packages:

yum install pcp pcp-gui

enable and start pcp:

chkconfig pmcd on
chkconfig --list | grep pmcd
/etc/init.d/pmcd start

easy. let’s see if pmatop works:

pmatop

pmatop

great. pminfo tells you what metrics are available currently:

pminfo -f
...
kernel.percpu.interrupts.SPU
    inst [0 or "cpu0"] value 0

kernel.percpu.interrupts.LOC
    inst [0 or "cpu0"] value 747300
...

a lot of stuff but nothing directly related to postgres except some information about the processes:

pminfo -f | grep -i postgres
...
    inst [12665 or "012665 /home/postgres/install/bin/postgres -D /home/postgres/data"] value 12665
    inst [12667 or "012667 postgres: checkpointer process   "] value 12667
    inst [12668 or "012668 postgres: writer process   "] value 12668
    inst [12669 or "012669 postgres: wal writer process   "] value 12669
    inst [12670 or "012670 postgres: autovacuum launcher process   "] value 12670
    inst [12671 or "012671 postgres: stats collector process   "] value 12671
...

according to the documentation pmdapostgres “Extracts performance metrics from the PostgreSQL relational database”. reading over and over again it became clear what to do. all these tools need to be installed first, so:

cd /var/lib/pcp/pmdas/postgresql
./Install 
Perl database interface (DBI) is not installed

ok:

yum install -y perl-DBI

next try:

./Install 
Postgres database driver (DBD::Pg) is not installed

gr:

yum install perl-DBD-Pg

and again:

./Install 
You will need to choose an appropriate configuration for installation of
the "postgresql" Performance Metrics Domain Agent (PMDA).

  collector	collect performance statistics on this system
  monitor	allow this system to monitor local and/or remote systems
  both		collector and monitor configuration for this system

Please enter c(ollector) or m(onitor) or b(oth) [b] b
Updating the Performance Metrics Name Space (PMNS) ...
Terminate PMDA if already installed ...
Updating the PMCD control file, and notifying PMCD ...
Waiting for pmcd to terminate ...
Starting pmcd ... 
Check postgresql metrics have appeared ... 15 warnings, 208 metrics and 0 values

much better. lets see if something is available now:

pminfo -f | grep -i postgres
...
postgresql.stat.all_tables.last_vacuum
postgresql.stat.all_tables.n_dead_tup
postgresql.stat.all_tables.seq_scan
postgresql.stat.all_tables.last_autoanalyze
postgresql.stat.all_tables.schemaname
postgresql.stat.all_tables.n_live_tup
postgresql.stat.all_tables.idx_tup_fetch
...

cool. can I get some values?

pminfo -f postgresql.stat.all_tables.n_tup_upd

postgresql.stat.all_tables.n_tup_upd
No value(s) available!

hm. not really what I expected. looking at the logfile:

tail -100 /var/log/pcp/pmcd/postgresql.log
...
DBI connect('dbname=postgres','postgres',...) failed: could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? at /var/lib/pcp/pmdas/postgresql/pmdapostgresql.pl line 252.

ok, seems connections to postgresql are not possible. the issue is that DBI looks for the sockets at “/var/run/postgresql/”. checking my postgresql.conf:

#unix_socket_directories = '/tmp'	# comma-separated list of directories
#unix_socket_group = ''			# (change requires restart)
#unix_socket_permissions = 0777		# begin with 0 to use octal notation

easy to fix:

su -
mkdir /var/run/postgresql/
chown postgres:postgres /var/run/postgresql/
su - postgres
echo "unix_socket_directories = '/var/run/postgresql/'" >> postgresql.conf

restart postgresql:

install/bin/pg_ctl stop -D data/
install/bin/pg_ctl start -D data/

checking again:

pminfo -f postgresql.stat.all_tables.n_tup_upd

postgresql.stat.all_tables.n_tup_upd
    inst [1261 or "pg_auth_members"] value 0
    inst [2617 or "pg_operator"] value 0
    inst [2600 or "pg_aggregate"] value 0
    inst [1136 or "pg_pltemplate"] value 0
    inst [12529 or "sql_implementation_info"] value 0
    inst [2609 or "pg_description"] value 0
    inst [2612 or "pg_language"] value 0
    inst [12539 or "sql_packages"] value 0
    inst [3601 or "pg_ts_parser"] value 0
    inst [3466 or "pg_event_trigger"] value 0
    inst [3592 or "pg_shseclabel"] value 0
    inst [3118 or "pg_foreign_table"] value 0

much better. but not really nice to read. this is where pmchart comes into the game:

pmchart &

metric selection
real time graphs

now performance data can be viewed in realtime…really cool.