<div dir="ltr"><div><span style="font-size:12.8px">Thanks Ken very much for your helpful infomation.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I am now blocking on I can&#39;t see the pacemaker DC do any further start/promote etc action on my resource agents, no helpful logs founded.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">So my first question is that in what kind of situation DC will decide do call start action?  does the monitor operation need to be return OCF_SUCCESS? in my case, it will return OCF_NOT_RUNNING, and the monitor operation is not being called any more, which should be wrong as I felt that it should be called intervally. </span></div><div><span style="font-size:12.8px"><br></span></div><div><div style="font-size:12.8px"><span style="font-size:12.8px">The resource agent monitor logistic:</span></div><div style="font-size:12.8px"><span style="font-size:12.8px">In the xx_monitor function it will call xx_update, and there always hit </span><span style="font-size:12.8px"> &quot;</span><span style="font-size:12.8px">$CRM_MASTER -D;;&quot; , what does it usually mean? will it stopped that start operation being called? </span></div><div style="font-size:12.8px"><span style="font-size:12.8px"><br></span></div><div><div><span style="font-size:12.8px">ovsdb_server_master_update() {</span></div><div><span style="font-size:12.8px">    ocf_log info &quot;ovsdb_server_master_update: $1}&quot;</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">    case $1 in</span></div><div><span style="font-size:12.8px">        $OCF_SUCCESS)</span></div><div><span style="font-size:12.8px">        $CRM_MASTER -v ${slave_score};;</span></div><div><span style="font-size:12.8px">        $OCF_RUNNING_MASTER)</span></div><div><span style="font-size:12.8px">            $CRM_MASTER -v ${master_score};;</span></div><div><span style="font-size:12.8px">        #*) $CRM_MASTER -D;;</span></div><div><span style="font-size:12.8px">    esac</span></div><div><span style="font-size:12.8px">    ocf_log info &quot;ovsdb_server_master_update end}&quot;</span></div><div><span style="font-size:12.8px">}</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">ovsdb_server_monitor() {</span></div><div><span style="font-size:12.8px">    ocf_log info &quot;ovsdb_server_monitor&quot;</span></div><div><span style="font-size:12.8px">    ovsdb_server_check_status</span></div><div><span style="font-size:12.8px">    rc=$?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">    ovsdb_server_master_update $rc</span></div><div><span style="font-size:12.8px">    ocf_log info &quot;monitor is going to return $rc&quot;</span></div><div><span style="font-size:12.8px">    return $rc</span></div><div><span style="font-size:12.8px">}</span></div></div></div><div><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Below is my cluster configuration:</span></div><div><br></div><div><div style="font-size:12.8px">1. First I have an vip set.</div><div style="font-size:12.8px"><div>[root@node-1 ~]# pcs resource show</div><div> vip__management_old<span style="white-space:pre-wrap">        </span>(ocf::es:ns_IPaddr2):<span style="white-space:pre-wrap">        </span>Started node-1.domain.tld</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">2. Use pcs to create ovndb-servers and constraint </div><div style="font-size:12.8px"><div>[root@node-1 ~]# pcs resource create tst-ovndb ocf:ovn:ovndb-servers  manage_northd=yes master_ip=192.168.0.2 nb_master_port=6641 sb_master_port=6642 master</div><div>     ([root@node-1 ~]# pcs resource meta tst-ovndb-master notify=true</div><div>      Error: unable to find a resource/clone/master/group: tst-ovndb-master) ## returned error, so I changed into below command.</div><div>[root@node-1 ~]# pcs resource master tst-ovndb-master tst-ovndb notify=true</div><div>[root@node-1 ~]# pcs constraint colocation add master tst-ovndb-master with vip__management_old</div><div><br></div><div>3. pcs status</div><div>[root@node-1 ~]# pcs status</div><div> vip__management_old<span style="white-space:pre-wrap">        </span>(ocf::es:ns_IPaddr2):<span style="white-space:pre-wrap">        </span>Started node-1.domain.tld</div><div> Master/Slave Set: tst-ovndb-master [tst-ovndb]</div><div>     Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]</div><div><br></div><div>4. pcs resource show XXX</div><div>[root@node-1 ~]# pcs resource show  vip__management_old</div><div> Resource: vip__management_old (class=ocf provider=es type=ns_IPaddr2)</div><div>  Attributes: nic=br-mgmt base_veth=br-mgmt-hapr ns_veth=hapr-m ip=192.168.0.2 iflabel=ka cidr_netmask=24 ns=haproxy gateway=none gateway_metric=0 iptables_start_rules=false iptables_stop_rules=false iptables_comment=default-<wbr>comment </div><div>  Meta Attrs: migration-threshold=3 failure-timeout=60 resource-stickiness=1 </div><div>  Operations: monitor interval=3 timeout=30 (vip__management_old-monitor-<wbr>3)</div><div>              start interval=0 timeout=30 (vip__management_old-start-0)</div><div>              stop interval=0 timeout=30 (vip__management_old-stop-0)</div><div>[root@node-1 ~]# pcs resource show tst-ovndb-master</div><div> Master: tst-ovndb-master</div><div>  Meta Attrs: notify=true </div><div>  Resource: tst-ovndb (class=ocf provider=ovn type=ovndb-servers)</div><div>   Attributes: manage_northd=yes master_ip=192.168.0.2 nb_master_port=6641 sb_master_port=6642 </div><div>   Operations: start interval=0s timeout=30s (tst-ovndb-start-timeout-30s)</div><div>               stop interval=0s timeout=20s (tst-ovndb-stop-timeout-20s)</div><div>               promote interval=0s timeout=50s (tst-ovndb-promote-timeout-<wbr>50s)</div><div>               demote interval=0s timeout=50s (tst-ovndb-demote-timeout-50s)</div><div>               monitor interval=30s timeout=20s (tst-ovndb-monitor-interval-<wbr>30s)</div><div>               monitor interval=10s role=Master timeout=20s (tst-ovndb-monitor-interval-<wbr>10s-role-Master)</div><div>               monitor interval=30s role=Slave timeout=20s (tst-ovndb-monitor-interval-<wbr>30s-role-Slave)</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div><span style="font-size:12.8px">colocation colocation-tst-ovndb-master-vip__management_old-INFINITY inf: tst-ovndb-master:Master vip__management_old:Started</span><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">5. I have put log in every ovndb-servers op, seems only the monitor op is being called, no promoted by the pacemaker DC:</div><div style="font-size:12.8px"><div>&lt;30&gt;Nov 30 15:22:19 node-1 ovndb-servers(tst-ovndb)[<wbr>2980860]: INFO: ovsdb_server_monitor</div><div>&lt;30&gt;Nov 30 15:22:19 node-1 ovndb-servers(tst-ovndb)[<wbr>2980860]: INFO: ovsdb_server_check_status</div><div>&lt;30&gt;Nov 30 15:22:19 node-1 ovndb-servers(tst-ovndb)[<wbr>2980860]: INFO: return OCFOCF_NOT_RUNNINGG</div><div>&lt;30&gt;Nov 30 15:22:20 node-1 ovndb-servers(tst-ovndb)[<wbr>2980860]: INFO: ovsdb_server_master_update: 7}</div><div>&lt;30&gt;Nov 30 15:22:20 node-1 ovndb-servers(tst-ovndb)[<wbr>2980860]: INFO: ovsdb_server_master_update end}</div><div>&lt;30&gt;Nov 30 15:22:20 node-1 ovndb-servers(tst-ovndb)[<wbr>2980860]: INFO: monitor is going to return 7</div><div>&lt;30&gt;Nov 30 15:22:20 node-1 ovndb-servers(undef)[2980970]: INFO: metadata exit OCF_SUCCESS}</div></div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><span style="font-size:12.8px">6. The cluster property:</span></div><div><div><span style="font-size:12.8px">property cib-bootstrap-options: \</span></div><div><span style="font-size:12.8px">        have-watchdog=false \</span></div><div><span style="font-size:12.8px">        dc-version=1.1.12-a14efad \</span></div><div><span style="font-size:12.8px">        cluster-infrastructure=corosync \</span></div><div><span style="font-size:12.8px">        no-quorum-policy=ignore \</span></div><div><span style="font-size:12.8px">        stonith-enabled=false \</span></div><div><span style="font-size:12.8px">        symmetric-cluster=false \</span></div><div><span style="font-size:12.8px">        last-lrm-refresh=1511802933</span></div></div><div><span style="font-size:12.8px"><br></span></div><div style="font-size:12.8px"><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thank you very much for any help.</span></div><div><span style="font-size:12.8px">Hui.</span></div><div><span style="font-size:12.8px"><br></span></div><div style="font-size:12.8px"><span style="font-size:12.8px"><br></span></div><span style="font-size:12.8px">Date: Mon, 27 Nov 2017 12:07:57 -0600</span><div><span style="font-size:12.8px">From: Ken Gaillot &lt;</span><a href="mailto:kgaillot@redhat.com" style="font-size:12.8px">kgaillot@redhat.com</a><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">To: Cluster Labs - All topics related to open-source clustering</span><br style="font-size:12.8px"><span style="font-size:12.8px">        welcomed        &lt;</span><a href="mailto:users@clusterlabs.org" style="font-size:12.8px">users@clusterlabs.org</a><span style="font-size:12.8px">&gt;, </span><a href="mailto:jpokorny@redhat.com" style="font-size:12.8px">jpokorny@redhat.com</a><br style="font-size:12.8px"><span style="font-size:12.8px">Subject: Re: [ClusterLabs] pcs create master/slave resource doesn&#39;t</span><br style="font-size:12.8px"><span style="font-size:12.8px">        work</span><br style="font-size:12.8px"><span style="font-size:12.8px">Message-ID: &lt;</span><a href="mailto:1511806077.5194.6.camel@redhat.com" style="font-size:12.8px">1511806077.5194.6.camel@<wbr>redhat.com</a><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">Content-Type: text/plain; charset=&quot;UTF-8&quot;</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">On Fri, 2017-11-24 at 18:00 +0800, Hui Xiang wrote:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Jan,</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ? Very appreciated on your help, I am getting further more, but still</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; it looks very strange.</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; 1. To use &quot;debug-promote&quot;, I upgrade pacemaker from 1.12 to 1.16, pcs</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; to 0.9.160.</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; 2. Recreate resource with below commands</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; pcs resource create ovndb_servers ocf:ovn:ovndb-servers \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ? master_ip=192.168.0.99 \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ? op monitor interval=&quot;10s&quot; \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ? op monitor interval=&quot;11s&quot; role=Master</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; pcs resource master ovndb_servers-master ovndb_servers \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ? meta notify=&quot;true&quot; master-max=&quot;1&quot; master-node-max=&quot;1&quot; clone-max=&quot;3&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; clone-node-max=&quot;1&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ? ? op monitor interval=10s</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; pcs constraint colocation add VirtualIP with master ovndb_servers-</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; master \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ? score=INFINITY</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; 3. pcs status</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?Master/Slave Set: ovndb_servers-master [ovndb_servers]</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ? ? ?Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?VirtualIP    (ocf::heartbeat:IPaddr2):       Stopped</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; 4. Manually run &#39;debug-start&#39; on 3 nodes and &#39;debug-promote&#39; on one</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; of nodes</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; run below on [ node-1.domain.tld node-2.domain.tld node-3.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; # pcs resource debug-start ovndb_servers --full</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; run below on [ node-1.domain.tld ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; # pcs resource debug-promote ovndb_servers --full</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">Before running debug-* commands, I&#39;d unmanage the resource or put the</span><br style="font-size:12.8px"><span style="font-size:12.8px">cluster in maintenance mode, so Pacemaker doesn&#39;t try to &quot;correct&quot; your</span><br style="font-size:12.8px"><span style="font-size:12.8px">actions.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; 5. pcs status</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?Master/Slave Set: ovndb_servers-master [ovndb_servers]</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ? ? ?Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?VirtualIP    (ocf::heartbeat:IPaddr2):       Stopped</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; 6. However I have seen that one of ovndb_servers has been indeed</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; promoted as master, but pcs status still showed all &#39;stopped&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; what am I missing?</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">It&#39;s hard to tell from these logs. It&#39;s possible the resource agent&#39;s</span><br style="font-size:12.8px"><span style="font-size:12.8px">monitor command is not exiting with the expected status values:</span><br style="font-size:12.8px"><br style="font-size:12.8px"><a href="http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_requirements_for_multi_state_resource_agents" rel="noreferrer" target="_blank" style="font-size:12.8px">http://clusterlabs.org/doc/en-<wbr>US/Pacemaker/1.1-pcs/html-<wbr>single/Pacemake<br>r_Explained/index.html#_<wbr>requirements_for_multi_state_<wbr>resource_agents</a><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">One of the nodes will be elected the DC, meaning it coordinates the</span><br style="font-size:12.8px"><span style="font-size:12.8px">cluster&#39;s actions. The DC&#39;s logs will have more &quot;pengine:&quot; messages,</span><br style="font-size:12.8px"><span style="font-size:12.8px">with each action that needs to be taken (e.g. &quot;* Start &lt;rsc&gt; &lt;node&gt;&quot;).</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">You can look through those actions to see what the cluster decided to</span><br style="font-size:12.8px"><span style="font-size:12.8px">do -- whether the resources were e</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">ver star</span></span><span style="font-size:12.8px">ted, whether any was</span><br style="font-size:12.8px"><span style="font-size:12.8px">promoted, and whether any were e</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">xplicitl</span></span><span style="font-size:12.8px">y stopped.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ocf_</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">log:327:</span></span><span style="font-size:12.8px"> __OCF_MSG=&#39;ovndb_servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Promoting node-1.doma</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">in.tld a</span></span><span style="font-size:12.8px">s the master&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ocf_log:329: case &quot;${__OCF_PRIO}&quot; in</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ocf_log:333: __OCF_PRIO=INFO</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ocf_log:338: &#39;[&#39; INFO = DEBUG &#39;]&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ocf_log:341: ha_log &#39;INFO: ovndb_servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Promoting nod</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">e-1.doma</span></span><span style="font-size:12.8px">in.tld as the master&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ha_log:253: __ha_log &#39;INFO: ovndb_servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Promoting node-1.domain.tld as the master&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: __ha_log:185: </span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">local ig</span></span><span style="font-size:12.8px">nore_stderr=false</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: __ha</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">_log:186</span></span><span style="font-size:12.8px">: local loglevel</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 1</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">7:45:59:</span></span><span style="font-size:12.8px"> __ha_log:188: &#39;[&#39; &#39;xINFO: ovndb_servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Promoting node-1.domain.tld as the master&#39; = x--ignore-stderr &#39;]&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? st</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">derr: + </span></span><span style="font-size:12.8px">17:45:59: __ha_log:190: &#39;[&#39; none = &#39;&#39; &#39;]&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: __ha_log:192: tty</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: __ha_log:193: &#39;[&#39; x = x0 -a x = xdebug &#39;]&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: __ha_log:195: &#39;[&#39; false = true &#39;]&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: __ha_log:199: &#39;[&#39; &#39;&#39; &#39;]&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: __ha_log:202: echo &#39;INFO: ovndb_servers:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Promoting node-1.domain.tld as the master&#39;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: INFO: ovndb_servers: Promoting node-1.doma</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">in.tld a</span></span><span style="font-size:12.8px">s the</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; master</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: __ha_log:204: return 0</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ovsdb_server_promote:378:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; /usr/sbin/crm_attr</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">ibute --</span></span><span style="font-size:12.8px">type crm_config --name OVN_REPL_INFO -s</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ovn_ovsdb_master_server -v node-1.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ovsdb_server_promote:379:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ovs</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">db_serve</span></span><span style="font-size:12.8px">r_master_update 8</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? st</span><span class="gmail-aBn" tabindex="0" style="font-size:12.8px"><span class="gmail-aQJ">derr: + </span></span><span style="font-size:12.8px">17:45:59: ovsdb_server_master_update:</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">214: case $1 in</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ovsdb_server_master_update:</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">218:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; /usr/sbin/crm_master -l reboot -v 10</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: ovsdb_server_promote:380: return 0</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: 458: rc=0</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ?&gt;? stderr: + 17:45:59: 459: exit 0</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; On 23/11/17 23:52 +0800, Hui Xiang wrote:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; I am working on HA with 3-nodes, which has below configurations:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;?</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; &quot;&quot;&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; pcs resource create ovndb_servers ocf:ovn:ovndb-servers \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;???master_ip=168.254.101.2 \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;???op monitor interval=&quot;10s&quot; \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;???op monitor interval=&quot;11s&quot; role=Master</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; pcs resource master ovndb_servers-master ovndb_servers \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;???meta notify=&quot;true&quot; master-max=&quot;1&quot; master-node-max=&quot;1&quot; clone-</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; max=&quot;3&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; clone-node-max=&quot;1&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; pcs resource create VirtualIP ocf:heartbeat:IPaddr2</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ip=168.254.101.2 \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;?????op monitor interval=10s</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; pcs constraint order promote ovndb_servers-master then VirtualIP</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; pcs constraint colocation add VirtualIP with master ovndb_servers-</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; master \</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;???score=INFINITY</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; &quot;&quot;&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; (Out of curiosity, this looks like a mix of output from?</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; pcs config export pcs-commands [or clufter cib2pcscmd -s]</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; and manual editing.??Is this a good guess?)</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; It&#39;s the output of &quot;pcs status&quot;.</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;???However, after setting it as above, the master is not being</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; selected, all</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; are stopped, from pacemaker log, node-1 has been chosen as the</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; master, I am</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; confuse where is wrong, can anybody give a help, it would be very</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; appreciated.</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;?</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;?</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;??Master/Slave Set: ovndb_servers-master [ovndb_servers]</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;??????Stopped: [ node-1.domain.tld node-2.domain.tld node-</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; 3.domain.tld ]</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;??VirtualIP (ocf::heartbeat:IPaddr2): Stopped</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;?</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;?</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; # pacemaker log</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++ /cib/configuration/resources:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">?&lt;primitive</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; class=&quot;ocf&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; id=&quot;ovndb_servers&quot; provider=&quot;ovn&quot; type=&quot;ovndb-servers&quot;/&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">??????&lt;instance_attributes</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; id=&quot;ovndb_servers-instance_</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">attributes&quot;&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????????&lt;nvpair</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; id=&quot;ovndb_servers-instance_</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">attributes-master_ip&quot; name=&quot;master_ip&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; value=&quot;168.254.101.2&quot;/&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">??????&lt;/instance_attributes&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">??????&lt;operations&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????????&lt;op</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; id=&quot;ovndb_servers-start-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">timeout-30s&quot; interval=&quot;0s&quot; name=&quot;start&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; timeout=&quot;30s&quot;/&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????????&lt;op</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; id=&quot;ovndb_servers-stop-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">timeout-20s&quot; interval=&quot;0s&quot; name=&quot;stop&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; timeout=&quot;20s&quot;/&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????????&lt;op</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; id=&quot;ovndb_servers-promote-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">timeout-50s&quot; interval=&quot;0s&quot; name=&quot;promote&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; timeout=&quot;50s&quot;/&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????????&lt;op</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; id=&quot;ovndb_servers-demote-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">timeout-50s&quot; interval=&quot;0s&quot; name=&quot;demote&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; timeout=&quot;50s&quot;/&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????????&lt;op</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; id=&quot;ovndb_servers-monitor-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">interval-10s&quot; interval=&quot;10s&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; name=&quot;monitor&quot;/&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????????&lt;op</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; id=&quot;ovndb_servers-monitor-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">interval-11s-role-Master&quot; interval=&quot;11s&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; name=&quot;monitor&quot; role=&quot;Master&quot;/&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">??????&lt;/operations&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++????????????????????????????</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????&lt;/primitive&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;?</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665249] node-1.domain.tld??????attrd:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; attrd_peer_update: Setting master-ovndb_servers[node-1.</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">domain.tld]:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; (null)</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; -&gt; 5 from node-1.domain.tld</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; If it&#39;s probable your ocf:ovn:ovndb-servers agent in master mode can</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; run something like &quot;attrd_updater -n master-ovndb_servers -U 5&quot;, then</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; it was indeed launched OK, and if it does not continue to run as</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; expected, there may be a problem with the agent itself.</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; no change.</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; You can try running &quot;pcs resource debug-promote ovndb_servers --full&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; to examine the executation details (assuming the agent responds to</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; OCF_TRACE_RA=1 environment variable, which is what shell-based</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; agents built on top ocf-shellfuncs sourcable shell library from</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; resource-agents project, hence incl. also agents it ships,</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; customarily do).</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Yes, thank, it&#39;s helpful.</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665251] node-1.domain.tld???????crmd:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">??notice:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; process_lrm_event: Operation ovndb_servers_monitor_0: ok</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; (node=node-1.domain.tld, call=185, rc=0, cib-update=88,</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; confirmed=true)</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; &lt;29&gt;Nov 23 23:06:03 node-1 crmd[665251]:???notice:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; process_lrm_event:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Operation ovndb_servers_monitor_0: ok (node=node-1.domain.tld,</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; call=185,</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; rc=0, cib-update=88, confirmed=true)</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: Diff: --- 0.630.2 2</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: Diff: +++ 0.630.3 (null)</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: +??/cib:??@num_updates=3</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_perform_op: ++</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; /cib/status/node_state[@id=&#39;1&#39;</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">]/transient_attributes[@id=&#39;1&#39;</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">]/instanc</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; e_attributes[@id=&#39;status-1&#39;]:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; &lt;nvpair id=&quot;status-1-master-ovndb_</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">servers&quot; name=&quot;master-</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ovndb_servers&quot;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; value=&quot;5&quot;/&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; Nov 23 23:06:03 [665246] node-1.domain.tld????????cib:?</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">????info:</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; cib_process_request: Completed cib_modify operation for section</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; status: OK</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; &gt; (rc=0, origin=node-3.domain.tld/</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">attrd/80, version=0.630.3)</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Also depends if there&#39;s anything interesting after this point...</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; ______________________________</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">_________________</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Users mailing list: </span><a href="mailto:Users@clusterlabs.org" style="font-size:12.8px">Users@clusterlabs.org</a><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; </span><a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank" style="font-size:12.8px">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br style="font-size:12.8px"><span style="font-size:12.8px">&gt;</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Project Home: </span><a href="http://www.clusterlabs.org/" rel="noreferrer" target="_blank" style="font-size:12.8px">http://www.clusterlabs.org</a><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Getting started: </span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch" rel="noreferrer" target="_blank" style="font-size:12.8px">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch</a><span style="font-size:12.8px">.</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; pdf</span><br style="font-size:12.8px"><span style="font-size:12.8px">&gt; Bugs: </span><a href="http://bugs.clusterlabs.org/" rel="noreferrer" target="_blank" style="font-size:12.8px">http://bugs.clusterlabs.org</a><br style="font-size:12.8px"><span style="font-size:12.8px">--</span><br style="font-size:12.8px"><span style="font-size:12.8px">Ken Gaillot &lt;</span><a href="mailto:kgaillot@redhat.com" style="font-size:12.8px">kgaillot@redhat.com</a><span style="font-size:12.8px">&gt;</span><br></div></div>