<div dir="ltr">Hi Ken,<div><br></div><div>Please see inline comments of last mail</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 8, 2015 at 8:25 PM, Pritam Kharat <span dir="ltr"><<a href="mailto:pritam.kharat@oneconvergence.com" target="_blank">pritam.kharat@oneconvergence.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Ken,<div><br></div><div>Thanks for reply.</div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, Oct 8, 2015 at 8:13 PM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 10/02/2015 01:47 PM, Pritam Kharat wrote:<br>
> Hi,<br>
><br>
> I have set up a ACTIVE/PASSIVE HA<br>
><br>
</span>> *Issue 1) *<br>
><br>
> *corosync.conf* file is<br>
<div><div>><br>
> # Please read the openais.conf.5 manual page<br>
><br>
> totem {<br>
><br>
> version: 2<br>
><br>
> # How long before declaring a token lost (ms)<br>
> token: 10000<br>
><br>
> # How many token retransmits before forming a new configuration<br>
> token_retransmits_before_loss_const: 20<br>
><br>
> # How long to wait for join messages in the membership protocol (ms)<br>
> join: 10000<br>
><br>
> # How long to wait for consensus to be achieved before starting a<br>
> new round of membership configuration (ms)<br>
> consensus: 12000<br>
><br>
> # Turn off the virtual synchrony filter<br>
> vsftype: none<br>
><br>
> # Number of messages that may be sent by one processor on receipt<br>
> of the token<br>
> max_messages: 20<br>
><br>
> # Limit generated nodeids to 31-bits (positive signed integers)<br>
> clear_node_high_bit: yes<br>
><br>
> # Disable encryption<br>
> secauth: off<br>
><br>
> # How many threads to use for encryption/decryption<br>
> threads: 0<br>
><br>
> # Optionally assign a fixed node id (integer)<br>
> # nodeid: 1234<br>
><br>
> # This specifies the mode of redundant ring, which may be none,<br>
> active, or passive.<br>
> rrp_mode: none<br>
> interface {<br>
> # The following values need to be set based on your<br>
> environment<br>
> ringnumber: 0<br>
> bindnetaddr: 192.168.101.0<br>
> mcastport: 5405<br>
> }<br>
><br>
> transport: udpu<br>
> }<br>
><br>
> amf {<br>
> mode: disabled<br>
> }<br>
><br>
> quorum {<br>
> # Quorum for the Pacemaker Cluster Resource Manager<br>
> provider: corosync_votequorum<br>
> expected_votes: 1<br>
<br>
</div></div>If you're using a recent version of corosync, use "two_node: 1" instead<br>
of "expected_votes: 1", and get rid of "no-quorum-policy: ignore" in the<br>
pacemaker cluster options.<br>
<div><div><br></div></div></blockquote></div></div><div> -> We are using corosync version 2.3.3. Do we above mentioned change for this version ?</div><div><div class="h5"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
> }<br>
><br>
><br>
> nodelist {<br>
><br>
> node {<br>
> ring0_addr: 192.168.101.73<br>
> }<br>
><br>
> node {<br>
> ring0_addr: 192.168.101.74<br>
> }<br>
> }<br>
><br>
> aisexec {<br>
> user: root<br>
> group: root<br>
> }<br>
><br>
><br>
> logging {<br>
> fileline: off<br>
> to_stderr: yes<br>
> to_logfile: yes<br>
> to_syslog: yes<br>
> syslog_facility: daemon<br>
> logfile: /var/log/corosync/corosync.log<br>
> debug: off<br>
> timestamp: on<br>
> logger_subsys {<br>
> subsys: AMF<br>
> debug: off<br>
> tags: enter|leave|trace1|trace2|trace3|trace4|trace6<br>
> }<br>
> }<br>
><br>
> And I have added 5 resources - 1 is VIP and 4 are upstart jobs<br>
> Node names are configured as -> sc-node-1(ACTIVE) and sc-node-2(PASSIVE)<br>
> Resources are running on ACTIVE node<br>
><br>
> Default cluster properties -<br>
><br>
> <cluster_property_set id="cib-bootstrap-options"><br>
> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"<br>
> value="1.1.10-42f2063"/><br>
> <nvpair id="cib-bootstrap-options-cluster-infrastructure"<br>
> name="cluster-infrastructure" value="corosync"/><br>
> <nvpair name="no-quorum-policy" value="ignore"<br>
> id="cib-bootstrap-options-no-quorum-policy"/><br>
> <nvpair name="stonith-enabled" value="false"<br>
> id="cib-bootstrap-options-stonith-enabled"/><br>
> <nvpair name="cluster-recheck-interval" value="3min"<br>
> id="cib-bootstrap-options-cluster-recheck-interval"/><br>
> <nvpair name="default-action-timeout" value="120s"<br>
> id="cib-bootstrap-options-default-action-timeout"/><br>
> </cluster_property_set><br>
><br>
><br>
> But sometimes after 2-3 migrations from ACTIVE to STANDBY and then from<br>
> STANDBY to ACTIVE,<br>
> both nodes become OFFLINE and Current DC becomes None, I have disabled the<br>
> stonith property and even quorum is ignored<br>
<br>
</div></div>Disabling stonith isn't helping you. The cluster needs stonith to<br>
recover from difficult situations, so it's easier to get into weird<br>
states like this without it.<br>
<span><br>
> root@sc-node-2:/usr/lib/python2.7/dist-packages/sc# crm status<br>
> Last updated: Sat Oct 3 00:01:40 2015<br>
> Last change: Fri Oct 2 23:38:28 2015 via crm_resource on sc-node-1<br>
> Stack: corosync<br>
> Current DC: NONE<br>
> 2 Nodes configured<br>
> 5 Resources configured<br>
><br>
> OFFLINE: [ sc-node-1 sc-node-2 ]<br>
><br>
> What is going wrong here ? What is the reason for node Current DC becoming<br>
> None suddenly ? Is corosync.conf okay ? Are default cluster properties fine<br>
> ? Help will be appreciated.<br>
<br>
</span>I'd recommend seeing how the problem behaves with stonith enabled, but<br>
in any case you'll need to dive into the logs to figure what starts the<br>
chain of events.<br>
<br></blockquote><div><br></div></div></div><div> -> We are seeing this issue when we try rebooting the vms</div><div><div class="h5"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
><br>
> *Issue 2)*<br>
<span>> Command used to add upstart job is<br>
><br>
> crm configure primitive service upstart:service meta allow-migrate=true<br>
> migration-threshold=5 failure-timeout=30s op monitor interval=15s<br>
> timeout=60s<br>
><br>
> But still sometimes I see fail count going to INFINITY. Why ? How can we<br>
> avoid it ? Resource should have migrated as soon as it reaches migration<br>
> threshold.<br>
><br>
> * Node sc-node-2:<br>
> service: migration-threshold=5 fail-count=1000000 last-failure='Fri Oct<br>
> 2 23:38:53 2015'<br>
> service1: migration-threshold=5 fail-count=1000000 last-failure='Fri Oct<br>
> 2 23:38:53 2015'<br>
><br>
> Failed actions:<br>
> service_start_0 (node=sc-node-2, call=-1, rc=1, status=Timed Out,<br>
> last-rc-change=Fri Oct 2 23:38:53 2015<br>
> , queued=0ms, exec=0ms<br>
> ): unknown error<br>
> service1_start_0 (node=sc-node-2, call=-1, rc=1, status=Timed Out,<br>
> last-rc-change=Fri Oct 2 23:38:53 2015<br>
> , queued=0ms, exec=0ms<br>
<br>
</span>migration-threshold is used for monitor failures, not (by default) start<br>
or stop failures.<br>
<br>
This is a start failure, which (by default) makes the fail-count go to<br>
infinity. The rationale is that a monitor failure indicates some sort of<br>
temporary error, but failing to start could well mean that something is<br>
wrong with the installation or configuration.<br>
<br>
You can tell the cluster to apply migration-threshold to start failures<br>
too, by setting the start-failure-is-fatal=false cluster option.<br>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div></div></div><br><br clear="all"><span class=""><div><br></div>-- <br><div>Thanks and Regards,<br>Pritam Kharat.<br></div>
</span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Thanks and Regards,<br>Pritam Kharat.<br></div>
</div>