<div dir="ltr"><div><div>No recent network changes. Will check for abnormal traffic using wireshark.<br><br></div>I also notice that the XML lines are partial (no ending '>', closing " and sometimes partial words) in logs. Any lines > 472 characters are truncated to 472 characters. Wondering is it due to anyother limitations. <br><br></div>I can post some line tomorrow when i am back to work.<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 8, 2016 at 8:00 PM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 06/08/2016 06:14 AM, Narayanamoorthy Srinivasan wrote:<br>
> I have a pacemaker cluster with two pacemaker remote nodes. Recently the<br>
> remote nodes started throwing below errors and SDB started self-fencing.<br>
> Appreciate if someone throws light on what could be the issue and the fix.<br>
><br>
> OS - SLES 12 SP1<br>
> Pacemaker Remote version - pacemaker-remote-1.1.13-14.7.x86_64<br>
><br>
> 2016-06-08T14:11:46.009073+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser<br>
> error : AttValue: ' expected<br>
> 2016-06-08T14:11:46.009314+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error:<br>
> key="neutron-ha-tool_monitor_0" operation="monitor"<br>
> crm-debug-origin="do_update_<br>
> 2016-06-08T14:11:46.009443+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error:<br>
> ^<br>
> 2016-06-08T14:11:46.009567+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser<br>
> error : attributes construct error<br>
> 2016-06-08T14:11:46.009697+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error:<br>
> key="neutron-ha-tool_monitor_0" operation="monitor"<br>
> crm-debug-origin="do_update_<br>
> 2016-06-08T14:11:46.009824+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error:<br>
> ^<br>
> 2016-06-08T14:11:46.009948+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser<br>
> error : Couldn't find end of Start Tag lrm_rsc_op line 1<br>
> 2016-06-08T14:11:46.010070+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error:<br>
> key="neutron-ha-tool_monitor_0" operation="monitor"<br>
> crm-debug-origin="do_update_<br>
> 2016-06-08T14:11:46.010191+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error:<br>
> ^<br>
> 2016-06-08T14:11:46.010460+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser<br>
> error : Premature end of data in tag lrm_resource line 1<br>
> 2016-06-08T14:11:46.010718+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error:<br>
> key="neutron-ha-tool_monitor_0" operation="monitor"<br>
> crm-debug-origin="do_update_<br>
> 2016-06-08T14:11:46.010977+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error:<br>
> ^<br>
> 2016-06-08T14:11:46.011234+05:30 d18-fb-7b-18-f1-8e<br>
> pacemaker_remoted[6190]: error: XML Error: Entity: line 1: parser<br>
> error : Premature end of data in tag lrm_resources line 1<br>
><br>
><br>
> --<br>
> Thanks & Regards<br>
> Moorthy<br>
<br>
</div></div>This sounds like the network traffic between the cluster nodes and the<br>
remote nodes is being corrupted. Have there been any network changes<br>
lately? Switch/firewall/etc. equipment/settings? MTU?<br>
<br>
You could try using a packet sniffer such as wireshark to see if the<br>
traffic looks abnormal in some way. The payload is XML so it should be<br>
more or less readable.<br>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Thanks & Regards<br>Moorthy</div>
</div>