<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">2015-06-09 23:26 GMT-06:00 Digimer <span dir="ltr"><<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 10/06/15 01:19 AM, Jonathan Vargas wrote:<br>
> Thanks Andrei, Digimer.<br>
><br>
> I see. Since I need to address this discussion to a definitive solution,<br>
> I am sharing you a diagram of how we are designing this HA architecture,<br>
> to clarify the problem we are trying to solve:<br>
><br>
> <a href="http://i.imgur.com/BFPcZSx.png" target="_blank">http://i.imgur.com/BFPcZSx.png</a><br>
<br>
</span>Last block is DRBD. If DRBD will be managed by the cluster, it must have<br>
fencing.<br>
<br>
This is your definitive answer.<br>
<br>
Without it, you *will* get a split-brain. That leads to, at best, data<br>
divergence or data loss.<br>
<span class=""><br>
> The first layer, Load Balancer; and the third later, Database, are both<br>
> already setup. The Load Balancer cluster uses only an VIP resource,<br>
> while Database cluster uses DRBD+VIP resources. They are on production<br>
> and work fine, test passed :-)<br>
><br>
> Now we are handling the Web Server layer, which I am discussing with<br>
> experts like you. These servers require to be all active and see the<br>
> same data for read & write, as quickly as possible, mainly reads.<br>
><br>
</span>> *So, If we stay with OCFS2: *Since we need to protect the service<br>
<span class="">> availability and keep most of nodes up, what choices do I have to avoid<br>
> reboots on both Web nodes caused by a split-brain situation when one of<br>
> them is disconnected from network?<br>
<br>
</span>None of this matters relative to the importance of working, tested<br>
fencing for replicated storage.<br>
<br>
In any HA setup, the reboot of a node should matter not. If you are<br>
afraid of rebooting a node, you need to reconsider your design.<br>
<span class=""><br></span></blockquote><div><br></div><div><br></div><div>Well, the problem is caused by a pretty common scenario: A simple network disconnection on node 1 causes both nodes to reboot, even when the node 1 is still offline, it will keep rebooting the active node 2. There were no disk issues, but the service availability was lost. <b>That's the main complain now :-/</b></div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
> Correct me if I'm wrong:<br>
><br>
</span>> *1. Redundant Channel:* This is pretty difficult, since we would have to<br>
<span class="">> add two new physical netword cards to the virtual machine hosts, and<br>
> that changes network configuration a lot in the virtualization platform.<br>
<br>
</span>High Availability must put priorities like hassle and cost second to<br>
what makes a system more resilient. If you choose not to spend the extra<br>
money or time, then you must accept the risks.</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> *2. Three Node Cluster:* This is possible, but it will consume more<br>
<span class="">> resources. We can have it only for cluster communication though, not for<br>
> web processing, that will decrease load.<br>
<br>
</span>Quorum is NOT a substitution for fencing. They solve different problems.<br>
<br>
Quorum is a tool for when all nodes are behaving properly. Fencing is a<br>
tool for when a node is not behaving properly.<br></blockquote><div><br></div><div><br></div><div>Yes, but by adding a 3rd node, it will help to determine which node could be failing and which are not, to fence the proper one. Right?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> *3. Disable Fencing:* You said this should not happen at all if we use a<br>
<span class="">> shared disk like OCFS. So I am discarding it.<br>
<br>
</span>Correct.<br>
<br>
> *4. Use NFS: *Yes, this will cause a SPoF, and to solve it we would have<br>
<span class="">> to setup another cluster with DRBD as described here<br>
</span>> <<a href="https://www.suse.com/documentation/sle_ha/singlehtml/book_sleha_techguides/book_sleha_techguides.html" target="_blank">https://www.suse.com/documentation/sle_ha/singlehtml/book_sleha_techguides/book_sleha_techguides.html</a>>,<br>
<span class="">> and add more infrastructure resources, or do we can setup NFS over OCFS2?<br>
<br>
</span>... Which would require fencing anyway, so you gain nothing but another<br>
layer of things to break. First rule of HA; Keep it simple.<br>
<br>
Complexity is the enemy of availability.<br></blockquote><div><br></div><div><br></div><div>Sure, fencing must be added to if this would be the case.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> Thanks in advance.<br>
><br>
> *Jonathan Vargas Rodríguez*<br>
> Founder and Solution Engineer<br>
> Alkaid <<a href="https://alkaid.cr/" target="_blank">https://alkaid.cr/</a>> | Open Source Software<br>
><br>
> * mail ** *<a href="mailto:jonathan.vargas@alkaid.cr">jonathan.vargas@alkaid.cr</a> <mailto:<a href="mailto:jonathan.vargas@alkaid.cr">jonathan.vargas@alkaid.cr</a>><br>
<span class="">> telf <a href="tel:%2B506%204001%206259%20Ext.%2001" value="+50640016259">+506 4001 6259 Ext. 01</a><br>
> mobi <a href="tel:%2B506%204001%206259%20Ext.%2051" value="+50640016259">+506 4001 6259 Ext. 51</a><br>
><br>
</span>> <<a href="http://linkedin.com/in/jonathanvargas/" target="_blank">http://linkedin.com/in/jonathanvargas/</a>><br>
> <<a href="https://plus.google.com/+JonathanVargas/" target="_blank">https://plus.google.com/+JonathanVargas/</a>><br>
> <<a href="https://www.facebook.com/alkaid.cr" target="_blank">https://www.facebook.com/alkaid.cr</a>> <<a href="https://twitter.com/alkaidcr" target="_blank">https://twitter.com/alkaidcr</a>><br>
<span class="">><br>
><br>
> 2015-06-09 22:03 GMT-06:00 Andrei Borzenkov <<a href="mailto:arvidjaar@gmail.com">arvidjaar@gmail.com</a><br>
</span>> <mailto:<a href="mailto:arvidjaar@gmail.com">arvidjaar@gmail.com</a>>>:<br>
<span class="">><br>
> В Tue, 9 Jun 2015 21:53:41 -0600<br>
> Jonathan Vargas <<a href="mailto:jonathan.vargas@alkaid.cr">jonathan.vargas@alkaid.cr</a><br>
</span>> <mailto:<a href="mailto:jonathan.vargas@alkaid.cr">jonathan.vargas@alkaid.cr</a>>> пишет:<br>
<span class="im HOEnZb">><br>
> > Thanks,<br>
> ><br>
> > Those nodes do not need coordination between them. They have been working<br>
> > so far until now without HA and OCFS2. A load balancer distributes the<br>
> > requests between both nodes, they do not know about the existence of each<br>
> > other.<br>
> ><br>
> > However, they do require shared storage to work with the same data. Before<br>
> > setting up the OCFS2 cluster, we have been syncing disks using rsync, but<br>
> > it syncs each minute, not real time.<br>
> ><br>
> > So, our requirement would depend on OCFS2, and it works, but not of an HA<br>
> > and stonith setup I think. I see no way how it could add value to the<br>
> > required solution. Or it does?<br>
> ><br>
><br>
> You need coordination between nodes on write and even if you mount your<br>
> system read-only you still have at least boot time journal replay. So<br>
> no, your nodes cannot free run.<br>
><br>
> You probably want to use NFS for this.<br>
><br>
><br>
><br>
><br>
</span><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> <a href="http://clusterlabs.org/mailman/listinfo/users" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
><br>
<br>
<br>
--<br>
Digimer<br>
Papers and Projects: <a href="https://alteeve.ca/w/" target="_blank">https://alteeve.ca/w/</a><br>
What if the cure for cancer is trapped in the mind of a person without<br>
access to education?<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div></div>