<div dir="ltr">Thanks Ken,<div><br></div><div>I have figure out root cause. There was race condition while adding resource and creating constrain(location).</div><div><br></div><div>When resource was added to cluster, all nodes trying to 'probe' resource via ocf - and because ocf didn't exist on 2/3 nodes at that time - pacemaker mark them as -INFINITY and even updating constrains(adding locations) - doesn't automatically reprobe/clean resource. It was hard to debug - because there were no errors/failed action for this resource, when you run cli commands.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 16, 2016 at 1:29 PM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 08/05/2016 05:12 PM, Nikita Koshikov wrote:<br>
> Thanks, Ken,<br>
><br>
> On Fri, Aug 5, 2016 at 7:21 AM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a><br>
</span><span class="">> <mailto:<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>>> wrote:<br>
><br>
> On 08/05/2016 03:48 AM, Andreas Kurz wrote:<br>
> > Hi,<br>
> ><br>
> > On Fri, Aug 5, 2016 at 2:08 AM, Nikita Koshikov <<a href="mailto:koshikov@gmail.com">koshikov@gmail.com</a> <mailto:<a href="mailto:koshikov@gmail.com">koshikov@gmail.com</a>><br>
</span><div><div class="h5">> > <mailto:<a href="mailto:koshikov@gmail.com">koshikov@gmail.com</a> <mailto:<a href="mailto:koshikov@gmail.com">koshikov@gmail.com</a>>>> wrote:<br>
> ><br>
> > Hello list,<br>
> ><br>
> > Can you, please, help me in debugging 1 resource not being started<br>
> > after node failover ?<br>
> ><br>
> > Here is configuration that I'm testing:<br>
> > 3 nodes(kvm VM) cluster, that have:<br>
> ><br>
> > node 10: aic-controller-58055.test.<wbr>domain.local<br>
> > node 6: aic-controller-50186.test.<wbr>domain.local<br>
> > node 9: aic-controller-12993.test.<wbr>domain.local<br>
> > primitive cmha cmha \<br>
> > params conffile="/etc/cmha/cmha.conf"<br>
> > daemon="/usr/bin/cmhad" pidfile="/var/run/cmha/cmha.<wbr>pid"<br>
> user=cmha \<br>
> > meta failure-timeout=30 resource-stickiness=1<br>
> > target-role=Started migration-threshold=3 \<br>
> > op monitor interval=10 on-fail=restart timeout=20 \<br>
> > op start interval=0 on-fail=restart timeout=60 \<br>
> > op stop interval=0 on-fail=block timeout=90<br>
> ><br>
> ><br>
> > What is the output of crm_mon -1frA once a node is down ... any failed<br>
> > actions?<br>
> ><br>
> ><br>
> > primitive sysinfo_aic-controller-12993.<wbr>test.domain.local<br>
> > ocf:pacemaker:SysInfo \<br>
> > params disk_unit=M disks="/ /var/log" min_disk_free=512M \<br>
> > op monitor interval=15s<br>
> > primitive sysinfo_aic-controller-50186.<wbr>test.domain.local<br>
> > ocf:pacemaker:SysInfo \<br>
> > params disk_unit=M disks="/ /var/log" min_disk_free=512M \<br>
> > op monitor interval=15s<br>
> > primitive sysinfo_aic-controller-58055.<wbr>test.domain.local<br>
> > ocf:pacemaker:SysInfo \<br>
> > params disk_unit=M disks="/ /var/log" min_disk_free=512M \<br>
> > op monitor interval=15s<br>
> ><br>
> ><br>
> > You can use a clone for this sysinfo resource and a symmetric cluster<br>
> > for a more compact configuration .... then you can skip all these<br>
> > location constraints.<br>
> ><br>
> ><br>
> > location cmha-on-aic-controller-12993.<wbr>test.domain.local cmha 100:<br>
> > aic-controller-12993.test.<wbr>domain.local<br>
> > location cmha-on-aic-controller-50186.<wbr>test.domain.local cmha 100:<br>
> > aic-controller-50186.test.<wbr>domain.local<br>
> > location cmha-on-aic-controller-58055.<wbr>test.domain.local cmha 100:<br>
> > aic-controller-58055.test.<wbr>domain.local<br>
> > location sysinfo-on-aic-controller-<wbr>12993.test.domain.local<br>
> > sysinfo_aic-controller-12993.<wbr>test.domain.local inf:<br>
> > aic-controller-12993.test.<wbr>domain.local<br>
> > location sysinfo-on-aic-controller-<wbr>50186.test.domain.local<br>
> > sysinfo_aic-controller-50186.<wbr>test.domain.local inf:<br>
> > aic-controller-50186.test.<wbr>domain.local<br>
> > location sysinfo-on-aic-controller-<wbr>58055.test.domain.local<br>
> > sysinfo_aic-controller-58055.<wbr>test.domain.local inf:<br>
> > aic-controller-58055.test.<wbr>domain.local<br>
> > property cib-bootstrap-options: \<br>
> > have-watchdog=false \<br>
> > dc-version=1.1.14-70404b0 \<br>
> > cluster-infrastructure=<wbr>corosync \<br>
> > cluster-recheck-interval=15s \<br>
> ><br>
> ><br>
> > Never tried such a low cluster-recheck-interval ... wouldn't do<br>
> that. I<br>
> > saw setups with low intervals burning a lot of cpu cycles in bigger<br>
> > cluster setups and side-effects from aborted transitions. If you<br>
> do this<br>
> > for "cleanup" the cluster state because you see resource-agent errors<br>
> > you should better fix the resource agent.<br>
><br>
> Strongly agree -- your recheck interval is lower than the various action<br>
> timeouts. The only reason recheck interval should ever be set less than<br>
> about 5 minutes is if you have time-based rules that you want to trigger<br>
> with a finer granularity.<br>
><br>
> Your issue does not appear to be coming from recheck interval, otherwise<br>
> it would go away after the recheck interval passed.<br>
><br>
><br>
> As of small cluster-recheck-interval - this was only for testing.<br>
><br>
><br>
> > Regards,<br>
> > Andreas<br>
> ><br>
> ><br>
> > no-quorum-policy=stop \<br>
> > stonith-enabled=false \<br>
> > start-failure-is-fatal=false \<br>
> > symmetric-cluster=false \<br>
> > node-health-strategy=migrate-<wbr>on-red \<br>
> > last-lrm-refresh=1470334410<br>
> ><br>
> > When 3 nodes online, everything seemed OK, this is output of<br>
> > scoreshow.sh:<br>
> > Resource Score<br>
> > Node Stickiness #Fail<br>
> > Migration-Threshold<br>
> > cmha -INFINITY<br>
> > aic-controller-12993.test.<wbr>domain.local 1 0<br>
> > cmha<br>
> > 101 aic-controller-50186.test.<wbr>domain.local 1 0<br>
> > cmha -INFINITY<br>
><br>
> Everything is not OK; cmha has -INFINITY scores on two nodes, meaning it<br>
> won't be allowed to run on them. This is why it won't start after the<br>
> one allowed node goes down, and why cleanup gets it working again<br>
> (cleanup removes bans caused by resource failures).<br>
><br>
> It's likely the resource previously failed the maximum allowed times<br>
> (migration-threshold=3) on those two nodes.<br>
><br>
> The next step would be to figure out why the resource is failing. The<br>
> pacemaker logs will show any output from the resource agent.<br>
><br>
><br>
> Resource was never started on these nodes. Maybe problem is in flow ? We<br>
> deploy:<br>
><br>
> 1) 1 node with all 3 IPs in corosync.conf<br>
> 2) set no-quorum policy = ignore<br>
> 3) add 2 nodes to corosync cluster<br>
> 4) create resource + 1 location constrain<br>
> 5) add 2 additional constrains<br>
> 6) set no-quorum policy = stop<br>
><br>
> The time between 4-5 is about 1 minute. And it's clear why 2 nodes were<br>
> -INFINITY in this period. But why when we add 2 more constrains - they<br>
> are not updating scores and cam this be changed ?<br>
<br>
</div></div>The resource may have never started on those nodes, but are you sure a<br>
start wasn't attempted and failed? If the start failed, the -INFINITY<br>
score would come from the failure, rather than only the cluster being<br>
asymmetric.<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
><br>
><br>
> > aic-controller-58055.test.<wbr>domain.local 1 0<br>
> > sysinfo_aic-controller-12993.<wbr>test.domain.local INFINITY<br>
> > aic-controller-12993.test.<wbr>domain.local 0 0<br>
> > sysinfo_aic-controller-50186.<wbr>test.domain.local -INFINITY<br>
> > aic-controller-50186.test.<wbr>domain.local 0 0<br>
> > sysinfo_aic-controller-58055.<wbr>test.domain.local INFINITY<br>
> > aic-controller-58055.test.<wbr>domain.local 0 0<br>
> ><br>
> > The problem starts when 1 node, goes offline<br>
> (aic-controller-50186).<br>
> > The resource cmha is stocked in stopped state.<br>
> > Here is the showscores:<br>
> > Resource Score<br>
> > Node Stickiness #Fail<br>
> > Migration-Threshold<br>
> > cmha -INFINITY<br>
> > aic-controller-12993.test.<wbr>domain.local 1 0<br>
> > cmha -INFINITY<br>
> > aic-controller-50186.test.<wbr>domain.local 1 0<br>
> > cmha -INFINITY<br>
> > aic-controller-58055.test.<wbr>domain.local 1 0<br>
> ><br>
> > Even it has target-role=Started pacemaker skipping this resource.<br>
> > And in logs I see:<br>
> > pengine: info: native_print: cmha<br>
> (ocf::heartbeat:cmha):<br>
> > Stopped<br>
> > pengine: info: native_color: Resource cmha cannot run<br>
> anywhere<br>
> > pengine: info: LogActions: Leave cmha (Stopped)<br>
> ><br>
> > To recover cmha resource I need to run either:<br>
> > 1) crm resource cleanup cmha<br>
> > 2) crm resource reprobe<br>
> ><br>
> > After any of the above commands, resource began to be picked up be<br>
> > pacemaker and I see valid scores:<br>
> > Resource Score<br>
> > Node Stickiness #Fail<br>
> > Migration-Threshold<br>
> > cmha 100<br>
> > aic-controller-58055.test.<wbr>domain.local 1 0 3<br>
> > cmha 101<br>
> > aic-controller-12993.test.<wbr>domain.local 1 0 3<br>
> > cmha -INFINITY<br>
> > aic-controller-50186.test.<wbr>domain.local 1 0 3<br>
> ><br>
> > So the questions here - why cluster-recheck doesn't work, and<br>
> should<br>
> > it do reprobing ?<br>
> > How to make migration work or what I missed in configuration that<br>
> > prevents migration?<br>
> ><br>
> > corosync 2.3.4<br>
> > pacemaker 1.1.14<br>
</div></div></blockquote></div><br></div>