<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<br>
<br>
<div class="moz-cite-prefix">On 07/11/2023 17:57, lejeczek via Users
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:ab32476c-ebe2-41ca-98a7-57ac1593f86f@yahoo.co.uk">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<font size="2" face="Courier 10 Pitch">hi guys<br>
<br>
Having 3-node pgSQL cluster with PAF - when all three systems
are shutdown at virtually the same time then PAF fails to start
when HA cluster is operational again.<br>
<br>
from status:<br>
...<br>
Migration Summary:<br>
* Node: ubusrv2 (2):<br>
* PGSQL-PAF-5433: migration-threshold=1000000
fail-count=1000000 last-failure='Tue Nov 7 17:52:38 2023'<br>
* Node: ubusrv3 (3):<br>
* PGSQL-PAF-5433: migration-threshold=1000000
fail-count=1000000 last-failure='Tue Nov 7 17:52:38 2023'<br>
* Node: ubusrv1 (1):<br>
* PGSQL-PAF-5433: migration-threshold=1000000
fail-count=1000000 last-failure='Tue Nov 7 17:52:38 2023'<br>
<br>
Failed Resource Actions:<br>
* PGSQL-PAF-5433_stop_0 on ubusrv2 'error' (1): call=90,
status='complete', exitreason='Unexpected state for instance
"PGSQL-PAF-5433" (returned 1)', last-rc-change='Tue Nov 7
17:52:38 2023', queued=0ms, exec=84ms<br>
* PGSQL-PAF-5433_stop_0 on ubusrv3 'error' (1): call=82,
status='complete', exitreason='Unexpected state for instance
"PGSQL-PAF-5433" (returned 1)', last-rc-change='Tue Nov 7
17:52:38 2023', queued=0ms, exec=82ms<br>
* PGSQL-PAF-5433_stop_0 on ubusrv1 'error' (1): call=86,
status='complete', exitreason='Unexpected state for instance
"PGSQL-PAF-5433" (returned 1)', last-rc-change='Tue Nov 7
17:52:38 2023', queued=0ms, exec=108ms<br>
<br>
and all three pgSQLs show virtually identical logs:<br>
...<br>
2023-11-07 16:54:45.532 UTC [24936] LOG: starting PostgreSQL
14.9 (Ubuntu 14.9-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu,
compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit<br>
2023-11-07 16:54:45.532 UTC [24936] LOG: listening on IPv4
address "0.0.0.0", port 5433<br>
2023-11-07 16:54:45.532 UTC [24936] LOG: listening on IPv6
address "::", port 5433<br>
2023-11-07 16:54:45.535 UTC [24936] LOG: listening on Unix
socket "/var/run/postgresql/.s.PGSQL.5433"<br>
2023-11-07 16:54:45.547 UTC [24938] LOG: database system was
interrupted while in recovery at log time 2023-11-07 15:30:56
UTC<br>
2023-11-07 16:54:45.547 UTC [24938] HINT: If this has occurred
more than once some data might be corrupted and you might need
to choose an earlier recovery target.<br>
2023-11-07 16:54:45.819 UTC [24938] LOG: entering standby mode<br>
2023-11-07 16:54:45.824 UTC [24938] FATAL: could not open
directory "/var/run/postgresql/14-paf.pg_stat_tmp": No such file
or directory<br>
2023-11-07 16:54:45.825 UTC [24936] LOG: startup process (PID
24938) exited with exit code 1<br>
2023-11-07 16:54:45.825 UTC [24936] LOG: aborting startup due
to startup process failure<br>
2023-11-07 16:54:45.826 UTC [24936] LOG: database system is
shut down<br>
<br>
Is this "test" case's result, as I showed above, expected? It
reproduces every time.<br>
If not - what might it be I'm missing?<br>
<br>
many thanks, L.<br>
</font> <br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">
</pre>
</blockquote>
Actually, the resource fails to start on a node a single node - as
opposed to entire cluster shutdown as I noted originally - which was
powered down in an orderly fashion and powered back on.<br>
That the the time of power-cycle the node was PAF resource master,
it fails:<br>
...<br>
2023-11-09 20:35:04.439 UTC [17727] LOG: starting PostgreSQL 14.9
(Ubuntu 14.9-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit<br>
2023-11-09 20:35:04.439 UTC [17727] LOG: listening on IPv4 address
"0.0.0.0", port 5433<br>
2023-11-09 20:35:04.439 UTC [17727] LOG: listening on IPv6 address
"::", port 5433<br>
2023-11-09 20:35:04.442 UTC [17727] LOG: listening on Unix socket
"/var/run/postgresql/.s.PGSQL.5433"<br>
2023-11-09 20:35:04.452 UTC [17731] LOG: database system was
interrupted while in recovery at log time 2023-11-09 20:25:21 UTC<br>
2023-11-09 20:35:04.452 UTC [17731] HINT: If this has occurred more
than once some data might be corrupted and you might need to choose
an earlier recovery target.<br>
2023-11-09 20:35:04.809 UTC [17731] LOG: entering standby mode<br>
2023-11-09 20:35:04.813 UTC [17731] FATAL: could not open directory
"/var/run/postgresql/14-paf.pg_stat_tmp": No such file or directory<br>
2023-11-09 20:35:04.814 UTC [17727] LOG: startup process (PID
17731) exited with exit code 1<br>
2023-11-09 20:35:04.814 UTC [17727] LOG: aborting startup due to
startup process failure<br>
2023-11-09 20:35:04.815 UTC [17727] LOG: database system is shut
down<br>
<br>
<br>
The master at the time node was shut down did get moved over to
standby/slave node, properly,<br>
<br>
I'm on Ubuntu with:<br>
<br>
ii corosync 3.1.6-1ubuntu1 amd64 cluster
engine daemon and utilities<br>
ii pacemaker 2.1.2-1ubuntu3.1 amd64 cluster
resource manager<br>
ii pacemaker-cli-utils 2.1.2-1ubuntu3.1 amd64 cluster
resource manager command line utilities<br>
ii pacemaker-common 2.1.2-1ubuntu3.1 all cluster
resource manager common files<br>
ii pacemaker-resource-agents 2.1.2-1ubuntu3.1 all cluster
resource manager general resource agents<br>
ii pcs 0.10.11-2ubuntu3 all
Pacemaker Configuration System<br>
<br>
And here is the resource:<br>
-> $ pcs resource config PGSQL-PAF-5433-clone<br>
Clone: PGSQL-PAF-5433-clone<br>
Meta Attrs: failure-timeout=20s master-max=1 notify=true
promotable=true<br>
Resource: PGSQL-PAF-5433 (class=ocf provider=heartbeat
type=pgsqlms)<br>
Attributes: bindir=/usr/lib/postgresql/14/bin
datadir=/var/lib/postgresql/14/paf pgdata=/etc/postgresql/14/paf
pgport=5433<br>
Operations: demote interval=0s timeout=120s
(PGSQL-PAF-5433-demote-interval-0s)<br>
methods interval=0s timeout=5
(PGSQL-PAF-5433-methods-interval-0s)<br>
monitor interval=15s role=Master timeout=10s
(PGSQL-PAF-5433-monitor-interval-15s)<br>
monitor interval=16s role=Slave timeout=10s
(PGSQL-PAF-5433-monitor-interval-16s)<br>
notify interval=0s timeout=60s
(PGSQL-PAF-5433-notify-interval-0s)<br>
promote interval=0s timeout=30s
(PGSQL-PAF-5433-promote-interval-0s)<br>
reload interval=0s timeout=20
(PGSQL-PAF-5433-reload-interval-0s)<br>
start interval=0s timeout=60s
(PGSQL-PAF-5433-start-interval-0s)<br>
stop interval=0s timeout=60s
(PGSQL-PAF-5433-stop-interval-0s)<br>
<br>
Is this my setup/config or there might actually be an issue with the
PAF |& HA not handling node-OS shutdown?<br>
all & any thoughts are much apreciated.<br>
Thanks, L.<br>
<br>
<br>
</body>
</html>