<div dir="ltr">I'm so lucky :) thanks for your help!<div><br></div><div>Gerard</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 19, 2017 at 12:04 AM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Wed, 2017-10-18 at 16:58 +0200, Gerard Garcia wrote:<br>
> I'm using version 1.1.15-11.el7_3.2-e174ec8. As far as I know the<br>
> latest stable version in Centos 7.3<br>
><br>
> Gerard<br>
<br>
</span>Interesting ... this was an undetected bug that was coincidentally<br>
fixed by the recent fail-count work released in 1.1.17. The bug only<br>
affected cloned resources where one clone's name ended with the<br>
other's.<br>
<br>
FYI, CentOS 7.4 has 1.1.16, but that won't help this issue.<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> On Wed, Oct 18, 2017 at 4:42 PM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
> wrote:<br>
> > On Wed, 2017-10-18 at 14:25 +0200, Gerard Garcia wrote:<br>
> > > So I think I found the problem. The two resources are named<br>
> > forwarder<br>
> > > and bgpforwarder. It doesn't matter if bgpforwarder exists. It is<br>
> > > just that when I set the failcount to INFINITY to a resource<br>
> > named<br>
> > > bgpforwarder (crm_failcount -r bgpforwarder -v INFINITY) it<br>
> > directly<br>
> > > affects the forwarder resource. <br>
> > ><br>
> > > If I change the name to forwarderbgp, the problem disappears. So<br>
> > it<br>
> > > seems that the problem is that Pacemaker mixes the bgpforwarder<br>
> > and<br>
> > > forwarder names. Is it a bug?<br>
> > ><br>
> > > Gerard<br>
> ><br>
> > That's really surprising. What version of pacemaker are you using?<br>
> > There were a lot of changes in fail count handling in the last few<br>
> > releases.<br>
> ><br>
> > ><br>
> > > On Tue, Oct 17, 2017 at 6:27 PM, Gerard Garcia <<a href="mailto:gerard@talaia.io">gerard@talaia.io</a>><br>
> > > wrote:<br>
> > > > That makes sense. I've tried copying the anything resource and<br>
> > > > changed its name and id (which I guess should be enough to make<br>
> > > > pacemaker think they are different) but I still have the same<br>
> > > > problem.<br>
> > > ><br>
> > > > After more debugging I have reduced the problem to this:<br>
> > > > * First cloned resource running fine<br>
> > > > * Second cloned resource running fine<br>
> > > > * Manually set failcount to INFINITY to second cloned resource<br>
> > > > * Pacemaker triggers an stop operation (without monitor<br>
> > operation<br>
> > > > failing) for the two resources in the node where the failcount<br>
> > has<br>
> > > > been set to INFINITY.<br>
> > > > * Reset failcount starts the two resources again<br>
> > > ><br>
> > > > Weirdly enough the second resource doesn't stop if I set the<br>
> > the<br>
> > > > the first resource failcount to INFINITY (not even the first<br>
> > > > resource stops...). <br>
> > > ><br>
> > > > But:<br>
> > > > * If I set the first resource as globally-unique=true it does<br>
> > not<br>
> > > > stop so somehow this breaks the relation.<br>
> > > > * If I manually set the failcount to 0 in the first resource<br>
> > that<br>
> > > > also breaks the relation so it does not stop either. It seems<br>
> > like<br>
> > > > the failcount value is being inherited from the second resource<br>
> > > > when it does not have any value. <br>
> > > ><br>
> > > > I must have something wrongly configuration but I can't really<br>
> > see<br>
> > > > why there is this relationship...<br>
> > > ><br>
> > > > Gerard<br>
> > > ><br>
> > > > On Tue, Oct 17, 2017 at 3:35 PM, Ken Gaillot <kgaillot@redhat.c<br>
> > om><br>
> > > > wrote:<br>
> > > > > On Tue, 2017-10-17 at 11:47 +0200, Gerard Garcia wrote:<br>
> > > > > > Thanks Ken. Yes, inspecting the logs seems that the<br>
> > failcount<br>
> > > > > of the<br>
> > > > > > correctly running resource reaches the maximum number of<br>
> > > > > allowed<br>
> > > > > > failures and gets banned in all nodes.<br>
> > > > > ><br>
> > > > > > What is weird is that I just see how the failcount for the<br>
> > > > > first<br>
> > > > > > resource gets updated, is like the failcount are being<br>
> > mixed.<br>
> > > > > In<br>
> > > > > > fact, when the two resources get banned the only way I have<br>
> > to<br>
> > > > > make<br>
> > > > > > the first one start is to disable the failing one and clean<br>
> > the<br>
> > > > > > failcount of the two resources (it is not enough to only<br>
> > clean<br>
> > > > > the<br>
> > > > > > failcount of the first resource) does it make sense?<br>
> > > > > ><br>
> > > > > > Gerard<br>
> > > > ><br>
> > > > > My suspicion is that you have two instances of the same<br>
> > service,<br>
> > > > > and<br>
> > > > > the resource agent monitor is only checking the general<br>
> > service,<br>
> > > > > rather<br>
> > > > > than a specific instance of it, so the monitors on both of<br>
> > them<br>
> > > > > return<br>
> > > > > failure if either one is failing.<br>
> > > > ><br>
> > > > > That would make sense why you have to disable the failing<br>
> > > > > resource, so<br>
> > > > > its monitor stops running. I can't think of why you'd have to<br>
> > > > > clean its<br>
> > > > > failcount for the other one to start, though.<br>
> > > > ><br>
> > > > > The "anything" agent very often causes more problems than it<br>
> > > > > solves ...<br>
> > > > > I'd recommend writing your own OCF agent tailored to your<br>
> > > > > service.<br>
> > > > > It's not much more complicated than an init script.<br>
> > > > ><br>
> > > > > > On Mon, Oct 16, 2017 at 6:57 PM, Ken Gaillot <kgaillot@redh<br>
> > at.c<br>
> > > > > om><br>
> > > > > > wrote:<br>
> > > > > > > On Mon, 2017-10-16 at 18:30 +0200, Gerard Garcia wrote:<br>
> > > > > > > > Hi,<br>
> > > > > > > ><br>
> > > > > > > > I have a cluster with two ocf:heartbeat:anything<br>
> > resources<br>
> > > > > each<br>
> > > > > > > one<br>
> > > > > > > > running as a clone in all nodes of the cluster. For<br>
> > some<br>
> > > > > reason<br>
> > > > > > > when<br>
> > > > > > > > one of them fails to start the other one stops. There<br>
> > is<br>
> > > > > not any<br>
> > > > > > > > constrain configured or any kind of relation between<br>
> > them. <br>
> > > > > > > ><br>
> > > > > > > > Is it possible that there is some kind of implicit<br>
> > relation<br>
> > > > > that<br>
> > > > > > > I'm<br>
> > > > > > > > not aware of (for example because they are the same<br>
> > type?)<br>
> > > > > > > ><br>
> > > > > > > > Thanks,<br>
> > > > > > > ><br>
> > > > > > > > Gerard<br>
> > > > > > ><br>
> > > > > > > There is no implicit relation on the Pacemaker side.<br>
> > However<br>
> > > > > if the<br>
> > > > > > > agent returns "failed" for both resources when either one<br>
> > > > > fails,<br>
> > > > > > > you<br>
> > > > > > > could see something like that. I'd look at the logs on<br>
> > the DC<br>
> > > > > and<br>
> > > > > > > see<br>
> > > > > > > why it decided to restart the second resource.<br>
> > > > > > > --<br>
> > > > > > > Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
> > > > > > ><br>
> > > > > > > ______________________________<wbr>_________________<br>
> > > > > > > Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> > > > > > > <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> > > > > > ><br>
> > > > > > > Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> > > > > > > Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_f" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_f</a><br>
> > rom_<br>
> > > > > Scratc<br>
> > > > > > > h.pdf<br>
> > > > > > > Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> > > > > > ><br>
> > > > > ><br>
> > > > > > ______________________________<wbr>_________________<br>
> > > > > > Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> > > > > > <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> > > > > ><br>
> > > > > > Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> > > > > > Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_fro" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_fro</a><br>
> > m_Sc<br>
> > > > > ratch.<br>
> > > > > > pdf<br>
> > > > > > Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> > > > > --<br>
> > > > > Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
> > > > ><br>
> > > > > ______________________________<wbr>_________________<br>
> > > > > Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> > > > > <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> > > > ><br>
> > > > > Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> > > > > Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_</a><br>
> > Scra<br>
> > > > > tch.pdf<br>
> > > > > Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> > > > ><br>
> > > ><br>
> > > ><br>
> > ><br>
> > > ______________________________<wbr>_________________<br>
> > > Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> > > <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> > ><br>
> > > Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> > > Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scra" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scra</a><br>
> > tch.<br>
> > > pdf<br>
> > > Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> > --<br>
> > Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> > <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> ><br>
> > Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> > Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratc" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratc</a><br>
> > h.pdf<br>
> > Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
> ><br>
><br>
> ______________________________<wbr>_________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch</a>.<br>
> pdf<br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
--<br>
Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>