<div class="zcontentRow"> <p>please note that everything works fine when there is only one clone resource configured, the resource will get restarted and the vip will get moved.&nbsp;</p><p>anyway, I will check my ocfs again.</p><p><br></p><div class="zMailSign"><p style="font-family: 宋体; font-size: medium; line-height: normal; widows: 1;"><br></p><span style="line-height: normal; widows: 1; font-size: 7.0px;;color:#58595b;font-size:10px"></span></div><div class="zMailFrom"></div><div><div class="zhistoryRow" style="display:block"><div class="zhistoryDes" style="width: 100%; height: 28px; line-height: 28px; background-color: #E0E5E9; color: #1388FF; text-align: center;" language-data="HistoryOrgTxt">原始邮件</div><div id="zwriteHistoryContainer"><div class="control-group zhistoryPanel"><div class="zhistoryHeader" style="padding: 8px; background-color: #F5F6F8;"><div><strong language-data="HistorySenderTxt">发件人:</strong><span class="zreadUserName"> <kgaillot@redhat.com>;</span></div><div><strong language-data="HistoryTOTxt">收件人:</strong><span class="zreadUserName" style="display: inline-block;">何海龙10164561;</span></div><div><strong language-data="HistoryCCTxt">抄送人:</strong><span class="zreadUserName" style="display: inline-block;"> <users@clusterlabs.org>;</span></div><div><strong language-data="HistoryDateTxt">日 期 :</strong><span class="">2017年02月15日 23:22</span></div><div><strong language-data="HistorySubjectTxt">主 题 :</strong><span class="zreadTitle"><strong>Re: 答复: Re: 答复: Re: [ClusterLabs] clone resource not get restarted on fail</strong></span></div></div><p class="zhistoryContent"><br></p><div>On&nbsp;02/15/2017&nbsp;03:57&nbsp;AM,&nbsp;he.hailong5@zte.com.cn&nbsp;wrote:<br>>&nbsp;I&nbsp;just&nbsp;tried&nbsp;using&nbsp;colocation,&nbsp;it&nbsp;dosen't&nbsp;work.<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;I&nbsp;failed&nbsp;the&nbsp;node&nbsp;paas-controller-3,&nbsp;but&nbsp;sdclient_vip&nbsp;didn't&nbsp;get&nbsp;moved:<br><br>The&nbsp;colocation&nbsp;would&nbsp;work,&nbsp;but&nbsp;the&nbsp;problem&nbsp;you're&nbsp;having&nbsp;with&nbsp;router&nbsp;and<br>apigateway&nbsp;is&nbsp;preventing&nbsp;it&nbsp;from&nbsp;getting&nbsp;that&nbsp;far.&nbsp;In&nbsp;other&nbsp;words,<br>router&nbsp;and&nbsp;apigateway&nbsp;are&nbsp;still&nbsp;running&nbsp;on&nbsp;the&nbsp;node&nbsp;(they&nbsp;have&nbsp;not&nbsp;been<br>successfully&nbsp;stopped),&nbsp;so&nbsp;the&nbsp;colocation&nbsp;is&nbsp;still&nbsp;valid.<br><br>I&nbsp;suspect&nbsp;that&nbsp;the&nbsp;return&nbsp;codes&nbsp;from&nbsp;your&nbsp;custom&nbsp;resource&nbsp;agents&nbsp;may&nbsp;be<br>the&nbsp;issue.&nbsp;Make&nbsp;sure&nbsp;that&nbsp;your&nbsp;agents&nbsp;conform&nbsp;to&nbsp;these&nbsp;guidelines:<br><br>http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#ap-ocf<br><br>In&nbsp;particular,&nbsp;"start"&nbsp;should&nbsp;not&nbsp;return&nbsp;until&nbsp;a&nbsp;monitor&nbsp;operation&nbsp;would<br>return&nbsp;success,&nbsp;"stop"&nbsp;should&nbsp;not&nbsp;return&nbsp;until&nbsp;a&nbsp;monitor&nbsp;would&nbsp;return<br>"not&nbsp;running",&nbsp;and&nbsp;"monitor"&nbsp;should&nbsp;return&nbsp;"not&nbsp;running"&nbsp;if&nbsp;called&nbsp;on&nbsp;a<br>host&nbsp;where&nbsp;the&nbsp;service&nbsp;hasn't&nbsp;started&nbsp;yet.&nbsp;Be&nbsp;sure&nbsp;you&nbsp;are&nbsp;returning&nbsp;the<br>proper&nbsp;OCF_*&nbsp;codes&nbsp;according&nbsp;to&nbsp;the&nbsp;table&nbsp;in&nbsp;the&nbsp;link&nbsp;above.<br><br>If&nbsp;the&nbsp;documentation&nbsp;is&nbsp;unclear,&nbsp;please&nbsp;ask&nbsp;here&nbsp;about&nbsp;anything&nbsp;you&nbsp;are<br>unsure&nbsp;of.<br><br>>&nbsp;<br>>&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-1&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-2&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;router&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:router):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started<br>>&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;apigateway&nbsp;(ocf::heartbeat:apigateway):&nbsp;&nbsp;&nbsp;&nbsp;Started<br>>&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;here&nbsp;is&nbsp;the&nbsp;configuration:<br>>&nbsp;<br>>&nbsp;>crm&nbsp;configure&nbsp;show<br>>&nbsp;<br>>&nbsp;node&nbsp;$id="336855579"&nbsp;paas-controller-1<br>>&nbsp;<br>>&nbsp;node&nbsp;$id="336855580"&nbsp;paas-controller-2<br>>&nbsp;<br>>&nbsp;node&nbsp;$id="336855581"&nbsp;paas-controller-3<br>>&nbsp;<br>>&nbsp;primitive&nbsp;apigateway&nbsp;ocf:heartbeat:apigateway&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>>&nbsp;<br>>&nbsp;primitive&nbsp;apigateway_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="20.20.2.7"&nbsp;cidr_netmask="24"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>>&nbsp;<br>>&nbsp;primitive&nbsp;router&nbsp;ocf:heartbeat:router&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>>&nbsp;<br>>&nbsp;primitive&nbsp;router_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="10.10.1.7"&nbsp;cidr_netmask="24"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>>&nbsp;<br>>&nbsp;primitive&nbsp;sdclient&nbsp;ocf:heartbeat:sdclient&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>>&nbsp;<br>>&nbsp;primitive&nbsp;sdclient_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="10.10.1.8"&nbsp;cidr_netmask="24"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>>&nbsp;<br>>&nbsp;clone&nbsp;apigateway_rep&nbsp;apigateway<br>>&nbsp;<br>>&nbsp;clone&nbsp;router_rep&nbsp;router<br>>&nbsp;<br>>&nbsp;clone&nbsp;sdclient_rep&nbsp;sdclient<br>>&nbsp;<br>>&nbsp;colocation&nbsp;apigateway_colo&nbsp;+inf:&nbsp;apigateway_vip&nbsp;apigateway_rep:Started<br>>&nbsp;<br>>&nbsp;colocation&nbsp;router_colo&nbsp;+inf:&nbsp;router_vip&nbsp;router_rep:Started<br>>&nbsp;<br>>&nbsp;colocation&nbsp;sdclient_colo&nbsp;+inf:&nbsp;sdclient_vip&nbsp;sdclient_rep:Started<br>>&nbsp;<br>>&nbsp;property&nbsp;$id="cib-bootstrap-options"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dc-version="1.1.10-42f2063"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cluster-infrastructure="corosync"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;stonith-enabled="false"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;no-quorum-policy="stop"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;start-failure-is-fatal="false"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;last-lrm-refresh="1486981647"<br>>&nbsp;<br>>&nbsp;op_defaults&nbsp;$id="op_defaults-options"&nbsp;\<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;on-fail="restart"<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;原始邮件<br>>&nbsp;*发件人:*何海龙10164561<br>>&nbsp;*收件人:*<kgaillot@redhat.com>;<br>>&nbsp;*抄送人:*<users@clusterlabs.org>;<br>>&nbsp;*日&nbsp;期&nbsp;:*2017年02月15日&nbsp;10:54<br>>&nbsp;*主&nbsp;题&nbsp;:**答复:&nbsp;Re:&nbsp;答复:&nbsp;Re:&nbsp;[ClusterLabs]&nbsp;clone&nbsp;resource&nbsp;not&nbsp;get<br>>&nbsp;restarted&nbsp;on&nbsp;fail*<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;Is&nbsp;there&nbsp;a&nbsp;reason&nbsp;not&nbsp;to&nbsp;use&nbsp;a&nbsp;colocation&nbsp;constraint&nbsp;instead?&nbsp;If&nbsp;X_vip<br>>&nbsp;is&nbsp;colocated&nbsp;with&nbsp;X,&nbsp;it&nbsp;will&nbsp;be&nbsp;moved&nbsp;if&nbsp;X&nbsp;fails.<br>>&nbsp;<br>>&nbsp;[hhl]:&nbsp;the&nbsp;movement&nbsp;should&nbsp;take&nbsp;place&nbsp;as&nbsp;well&nbsp;if&nbsp;X&nbsp;stopped&nbsp;(the&nbsp;start&nbsp;is<br>>&nbsp;on-going).&nbsp;I&nbsp;don't&nbsp;know&nbsp;if&nbsp;the&nbsp;colocation&nbsp;would&nbsp;satisfy&nbsp;this&nbsp;requirement.<br>>&nbsp;<br>>&nbsp;I&nbsp;don't&nbsp;see&nbsp;any&nbsp;reason&nbsp;in&nbsp;your&nbsp;configuration&nbsp;why&nbsp;the&nbsp;services&nbsp;wouldn't<br>>&nbsp;be&nbsp;restarted.&nbsp;It's&nbsp;possible&nbsp;the&nbsp;cluster&nbsp;tried&nbsp;to&nbsp;restart&nbsp;the&nbsp;service,<br>>&nbsp;but&nbsp;the&nbsp;stop&nbsp;action&nbsp;failed.&nbsp;Since&nbsp;you&nbsp;have&nbsp;stonith&nbsp;disabled,&nbsp;the&nbsp;cluster<br>>&nbsp;can't&nbsp;recover&nbsp;from&nbsp;a&nbsp;failed&nbsp;stop&nbsp;action.<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;[hhl]:&nbsp;the&nbsp;ocf&nbsp;logs&nbsp;showed&nbsp;the&nbsp;pacemaker&nbsp;never&nbsp;entered&nbsp;the&nbsp;stop&nbsp;function<br>>&nbsp;in&nbsp;this&nbsp;case.<br>>&nbsp;<br>>&nbsp;Is&nbsp;there&nbsp;a&nbsp;reason&nbsp;you&nbsp;disabled&nbsp;quorum?&nbsp;With&nbsp;3&nbsp;nodes,&nbsp;if&nbsp;they&nbsp;get&nbsp;split<br>>&nbsp;into&nbsp;groups&nbsp;of&nbsp;1&nbsp;node&nbsp;and&nbsp;2&nbsp;nodes,&nbsp;quorum&nbsp;is&nbsp;what&nbsp;keeps&nbsp;the&nbsp;groups&nbsp;from<br>>&nbsp;both&nbsp;starting&nbsp;all&nbsp;resources.<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;[hhl]:&nbsp;I&nbsp;enabled&nbsp;the&nbsp;quorum&nbsp;and&nbsp;had&nbsp;a&nbsp;retry,&nbsp;the&nbsp;same&nbsp;happens.<br>>&nbsp;<br>>&nbsp;b.t.w,&nbsp;I&nbsp;repeat&nbsp;sevaral&nbsp;times&nbsp;today,&nbsp;and&nbsp;found&nbsp;when&nbsp;I&nbsp;trigger&nbsp;the<br>>&nbsp;condition&nbsp;on&nbsp;one&nbsp;node&nbsp;that&nbsp;would&nbsp;fail&nbsp;all&nbsp;the&nbsp;clone&nbsp;resources,&nbsp;only&nbsp;one<br>>&nbsp;would&nbsp;get&nbsp;restared,&nbsp;the&nbsp;other&nbsp;two&nbsp;would&nbsp;fail&nbsp;to&nbsp;restart.<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;>&nbsp;trigger&nbsp;the&nbsp;failure&nbsp;conditon&nbsp;on&nbsp;paas-controller-1<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-2&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-1&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;router&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:router):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started<br>>&nbsp;paas-controller-1&nbsp;FAILED&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;apigateway&nbsp;(ocf::heartbeat:apigateway):&nbsp;&nbsp;&nbsp;&nbsp;Started<br>>&nbsp;paas-controller-1&nbsp;FAILED&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;<br>>&nbsp;<br>>&nbsp;>&nbsp;trigger&nbsp;the&nbsp;failure&nbsp;conditon&nbsp;on&nbsp;paas-controller-3<br>>&nbsp;<br>>&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-2&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-controller-3&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sdclient&nbsp;&nbsp;&nbsp;(ocf::heartbeat:sdclient):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started<br>>&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;<br>>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;apigateway&nbsp;(ocf::heartbeat:apigateway):&nbsp;&nbsp;&nbsp;&nbsp;Started<br>>&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>>&nbsp;<br>>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;*发件人:*<kgaillot@redhat.com>;<br>>&nbsp;*收件人:*何海龙10164561;<br>>&nbsp;*抄送人:*<users@clusterlabs.org>;<br>>&nbsp;*日&nbsp;期&nbsp;:*2017年02月15日&nbsp;06:14<br>>&nbsp;*主&nbsp;题&nbsp;:**Re:&nbsp;答复:&nbsp;Re:&nbsp;[ClusterLabs]&nbsp;clone&nbsp;resource&nbsp;not&nbsp;get&nbsp;restarted<br>>&nbsp;on&nbsp;fail*<br>>&nbsp;<br>>&nbsp;<br>>&nbsp;On&nbsp;02/13/2017&nbsp;07:08&nbsp;PM,&nbsp;he.hailong5@zte.com.cn&nbsp;wrote:<br>>&nbsp;>&nbsp;Hi,<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;crm&nbsp;configure&nbsp;show<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;+&nbsp;crm&nbsp;configure&nbsp;show<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;node&nbsp;$id="336855579"&nbsp;paas-controller-1<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;node&nbsp;$id="336855580"&nbsp;paas-controller-2<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;node&nbsp;$id="336855581"&nbsp;paas-controller-3<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;primitive&nbsp;apigateway&nbsp;ocf:heartbeat:apigateway&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;primitive&nbsp;apigateway_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="20.20.2.7"&nbsp;cidr_netmask="24"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;primitive&nbsp;router&nbsp;ocf:heartbeat:router&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;primitive&nbsp;router_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="10.10.1.7"&nbsp;cidr_netmask="24"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;primitive&nbsp;sdclient&nbsp;ocf:heartbeat:sdclient&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;interval="2s"&nbsp;timeout="20s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="200s"&nbsp;on-fail="restart"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="9999h"&nbsp;on-fail="restart"<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;primitive&nbsp;sdclient_vip&nbsp;ocf:heartbeat:IPaddr2&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;params&nbsp;ip="10.10.1.8"&nbsp;cidr_netmask="24"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;start&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;stop&nbsp;interval="0"&nbsp;timeout="20"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;op&nbsp;monitor&nbsp;timeout="20s"&nbsp;interval="2s"&nbsp;depth="0"<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;clone&nbsp;apigateway_rep&nbsp;apigateway<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;clone&nbsp;router_rep&nbsp;router<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;clone&nbsp;sdclient_rep&nbsp;sdclient<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;location&nbsp;apigateway_loc&nbsp;apigateway_vip&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rule&nbsp;$id="apigateway_loc-rule"&nbsp;+inf:&nbsp;apigateway_workable&nbsp;eq&nbsp;1<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;location&nbsp;router_loc&nbsp;router_vip&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rule&nbsp;$id="router_loc-rule"&nbsp;+inf:&nbsp;router_workable&nbsp;eq&nbsp;1<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;location&nbsp;sdclient_loc&nbsp;sdclient_vip&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rule&nbsp;$id="sdclient_loc-rule"&nbsp;+inf:&nbsp;sdclient_workable&nbsp;eq&nbsp;1<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;property&nbsp;$id="cib-bootstrap-options"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dc-version="1.1.10-42f2063"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cluster-infrastructure="corosync"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;stonith-enabled="false"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;no-quorum-policy="ignore"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;start-failure-is-fatal="false"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;last-lrm-refresh="1486981647"<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;op_defaults&nbsp;$id="op_defaults-options"&nbsp;\<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;on-fail="restart"<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;-------------------------------------------------------------------------------------------------<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;and&nbsp;B.T.W,&nbsp;I&nbsp;am&nbsp;using&nbsp;"crm_attribute&nbsp;-N&nbsp;$HOSTNAME&nbsp;-q&nbsp;-l&nbsp;reboot&nbsp;--name<br>>&nbsp;>&nbsp;<prefix>_workable&nbsp;-v&nbsp;<1&nbsp;or&nbsp;0>"&nbsp;in&nbsp;the&nbsp;monitor&nbsp;to&nbsp;update&nbsp;the<br>>&nbsp;>&nbsp;transient&nbsp;attributes,&nbsp;which&nbsp;control&nbsp;the&nbsp;vip&nbsp;location.<br>>&nbsp;<br>>&nbsp;Is&nbsp;there&nbsp;a&nbsp;reason&nbsp;not&nbsp;to&nbsp;use&nbsp;a&nbsp;colocation&nbsp;constraint&nbsp;instead?&nbsp;If&nbsp;X_vip<br>>&nbsp;is&nbsp;colocated&nbsp;with&nbsp;X,&nbsp;it&nbsp;will&nbsp;be&nbsp;moved&nbsp;if&nbsp;X&nbsp;fails.<br>>&nbsp;<br>>&nbsp;I&nbsp;don't&nbsp;see&nbsp;any&nbsp;reason&nbsp;in&nbsp;your&nbsp;configuration&nbsp;why&nbsp;the&nbsp;services&nbsp;wouldn't<br>>&nbsp;be&nbsp;restarted.&nbsp;It's&nbsp;possible&nbsp;the&nbsp;cluster&nbsp;tried&nbsp;to&nbsp;restart&nbsp;the&nbsp;service,<br>>&nbsp;but&nbsp;the&nbsp;stop&nbsp;action&nbsp;failed.&nbsp;Since&nbsp;you&nbsp;have&nbsp;stonith&nbsp;disabled,&nbsp;the&nbsp;cluster<br>>&nbsp;can't&nbsp;recover&nbsp;from&nbsp;a&nbsp;failed&nbsp;stop&nbsp;action.<br>>&nbsp;<br>>&nbsp;Is&nbsp;there&nbsp;a&nbsp;reason&nbsp;you&nbsp;disabled&nbsp;quorum?&nbsp;With&nbsp;3&nbsp;nodes,&nbsp;if&nbsp;they&nbsp;get&nbsp;split<br>>&nbsp;into&nbsp;groups&nbsp;of&nbsp;1&nbsp;node&nbsp;and&nbsp;2&nbsp;nodes,&nbsp;quorum&nbsp;is&nbsp;what&nbsp;keeps&nbsp;the&nbsp;groups&nbsp;from<br>>&nbsp;both&nbsp;starting&nbsp;all&nbsp;resources.<br>>&nbsp;<br>>&nbsp;>&nbsp;and&nbsp;also&nbsp;found,&nbsp;the&nbsp;vip&nbsp;resource&nbsp;won't&nbsp;get&nbsp;moved&nbsp;if&nbsp;the&nbsp;related&nbsp;clone<br>>&nbsp;>&nbsp;resource&nbsp;failed&nbsp;to&nbsp;restart.<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;原始邮件<br>>&nbsp;>&nbsp;*发件人:*<kgaillot@redhat.com>;<br>>&nbsp;>&nbsp;*收件人:*<users@clusterlabs.org>;<br>>&nbsp;>&nbsp;*日&nbsp;期&nbsp;:*2017年02月13日&nbsp;23:04<br>>&nbsp;>&nbsp;*主&nbsp;题&nbsp;:**Re:&nbsp;[ClusterLabs]&nbsp;clone&nbsp;resource&nbsp;not&nbsp;get&nbsp;restarted&nbsp;on&nbsp;fail*<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;On&nbsp;02/13/2017&nbsp;07:57&nbsp;AM,&nbsp;he.hailong5@zte.com.cn&nbsp;wrote:<br>>&nbsp;>&nbsp;>&nbsp;Pacemaker&nbsp;1.1.10<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;Corosync&nbsp;2.3.3<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;this&nbsp;is&nbsp;a&nbsp;3&nbsp;nodes&nbsp;cluster&nbsp;configured&nbsp;with&nbsp;3&nbsp;clone&nbsp;resources,&nbsp;each<br>>&nbsp;>&nbsp;>&nbsp;attached&nbsp;wih&nbsp;a&nbsp;vip&nbsp;resource&nbsp;of&nbsp;IPAddr2:<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;>crm&nbsp;status<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>>&nbsp;controller-1&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>>&nbsp;controller-3&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>>&nbsp;controller-2&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-<br>>&nbsp;controller-3&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-<br>>&nbsp;controller-3&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-<br>>&nbsp;controller-3&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;It&nbsp;is&nbsp;observed&nbsp;that&nbsp;sometimes&nbsp;the&nbsp;clone&nbsp;resource&nbsp;is&nbsp;stuck&nbsp;to&nbsp;monitor<br>>&nbsp;>&nbsp;>&nbsp;when&nbsp;the&nbsp;service&nbsp;fails:<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;router_vip&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>>&nbsp;controller-1&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>>&nbsp;controller-2&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;apigateway_vip&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>>&nbsp;controller-3&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;router_rep&nbsp;[router]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;router&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(ocf::heartbeat:router):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started<br>>&nbsp;>&nbsp;>&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;apigateway_rep&nbsp;[apigateway]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;apigateway&nbsp;(ocf::heartbeat:apigateway):&nbsp;&nbsp;&nbsp;&nbsp;Started<br>>&nbsp;>&nbsp;>&nbsp;paas-controller-3&nbsp;FAILED&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;in&nbsp;the&nbsp;example&nbsp;above.&nbsp;the&nbsp;sdclient_rep&nbsp;get&nbsp;restarted&nbsp;on&nbsp;node&nbsp;3,&nbsp;while<br>>&nbsp;>&nbsp;>&nbsp;the&nbsp;other&nbsp;two&nbsp;hang&nbsp;at&nbsp;monitoring&nbsp;on&nbsp;node&nbsp;3,&nbsp;here&nbsp;are&nbsp;the&nbsp;ocf&nbsp;logs:<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;abnormal&nbsp;(apigateway_rep):<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:53&nbsp;[23586]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;Starting&nbsp;health&nbsp;check.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:53&nbsp;[23586]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;health&nbsp;check&nbsp;succeed.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[24010]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;Starting&nbsp;health&nbsp;check.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[24010]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;Failed:&nbsp;docker&nbsp;daemon&nbsp;is&nbsp;not&nbsp;running.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:57&nbsp;[24095]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;Starting&nbsp;health&nbsp;check.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:57&nbsp;[24095]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;Failed:&nbsp;docker&nbsp;daemon&nbsp;is&nbsp;not&nbsp;running.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:59&nbsp;[24159]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;Starting&nbsp;health&nbsp;check.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:59&nbsp;[24159]===print_log&nbsp;test_monitor&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;Failed:&nbsp;docker&nbsp;daemon&nbsp;is&nbsp;not&nbsp;running.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;normal&nbsp;(sdclient_rep):<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:52&nbsp;[23507]===print_log&nbsp;sdclient_monitor&nbsp;run_func<br>>&nbsp;>&nbsp;>&nbsp;main===&nbsp;health&nbsp;check&nbsp;succeed.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:54&nbsp;[23630]===print_log&nbsp;sdclient_monitor&nbsp;run_func<br>>&nbsp;>&nbsp;>&nbsp;main===&nbsp;Starting&nbsp;health&nbsp;check.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:54&nbsp;[23630]===print_log&nbsp;sdclient_monitor&nbsp;run_func<br>>&nbsp;>&nbsp;>&nbsp;main===&nbsp;Failed:&nbsp;docker&nbsp;daemon&nbsp;is&nbsp;not&nbsp;running.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[23710]===print_log&nbsp;sdclient_stop&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;Starting&nbsp;stop&nbsp;the&nbsp;container.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[23710]===print_log&nbsp;sdclient_stop&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;docker&nbsp;daemon&nbsp;lost,&nbsp;pretend&nbsp;stop&nbsp;succeed.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[23763]===print_log&nbsp;sdclient_start&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;Starting&nbsp;run&nbsp;the&nbsp;container.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:27:55&nbsp;[23763]===print_log&nbsp;sdclient_start&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;docker&nbsp;daemon&nbsp;lost,&nbsp;try&nbsp;again&nbsp;in&nbsp;5&nbsp;secs.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:28:00&nbsp;[23763]===print_log&nbsp;sdclient_start&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;docker&nbsp;daemon&nbsp;lost,&nbsp;try&nbsp;again&nbsp;in&nbsp;5&nbsp;secs.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;2017-02-13&nbsp;18:28:05&nbsp;[23763]===print_log&nbsp;sdclient_start&nbsp;run_func&nbsp;main===<br>>&nbsp;>&nbsp;>&nbsp;docker&nbsp;daemon&nbsp;lost,&nbsp;try&nbsp;again&nbsp;in&nbsp;5&nbsp;secs.<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;If&nbsp;I&nbsp;disable&nbsp;2&nbsp;clone&nbsp;resource,&nbsp;the&nbsp;switch&nbsp;over&nbsp;test&nbsp;for&nbsp;one&nbsp;clone<br>>&nbsp;>&nbsp;>&nbsp;resource&nbsp;works&nbsp;as&nbsp;expected:&nbsp;fail&nbsp;the&nbsp;service&nbsp;->&nbsp;monitor&nbsp;fails&nbsp;->&nbsp;stop<br>>&nbsp;>&nbsp;>&nbsp;->&nbsp;start<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;Online:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;sdclient_vip&nbsp;&nbsp;&nbsp;(ocf::heartbeat:IPaddr2):&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started&nbsp;paas-<br>>&nbsp;controller-2&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;Clone&nbsp;Set:&nbsp;sdclient_rep&nbsp;[sdclient]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Started:&nbsp;[&nbsp;paas-controller-1&nbsp;paas-controller-2&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Stopped:&nbsp;[&nbsp;paas-controller-3&nbsp;]<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;>&nbsp;what's&nbsp;the&nbsp;reason&nbsp;behind????&nbsp;<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;Can&nbsp;you&nbsp;show&nbsp;the&nbsp;configuration&nbsp;of&nbsp;the&nbsp;three&nbsp;clones,&nbsp;their&nbsp;operations,<br>>&nbsp;>&nbsp;and&nbsp;any&nbsp;constraints?<br>>&nbsp;>&nbsp;<br>>&nbsp;>&nbsp;Normally,&nbsp;the&nbsp;response&nbsp;is&nbsp;controlled&nbsp;by&nbsp;the&nbsp;monitor&nbsp;operation's&nbsp;on-fail<br>>&nbsp;>&nbsp;attribute&nbsp;(which&nbsp;defaults&nbsp;to&nbsp;restart).<br><br></div><p><br></p></div></div></div></div><p><br></p> </div>