sher(new master) =>

Oct  8 18:01:20 sher kernel: [  886.867496] e1000e: eth0 NIC Link is Down
Oct  8 18:01:22 sher exportfs(res_exportfs_root)[5566]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:01:27 sher exportfs(res_exportfs_export1)[5580]: INFO: Directory /mnt/vms/export1 is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:01:30 sher kernel: [  896.771854] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Oct  8 18:01:30 sher corosync[1320]:  [TOTEM ] A new membership (192.168.0.21:3444) was formed. Members joined: 102
Oct  8 18:01:30 sher crmd[1465]:    error: pcmk_cpg_membership: Node lock[102] appears to be online even though we think it is dead
Oct  8 18:01:30 sher crmd[1465]:   notice: crm_update_peer_state: pcmk_cpg_membership: Node lock[102] - state is now member (was lost)
Oct  8 18:01:30 sher corosync[1320]:  [QUORUM] Members[2]: 101 102
Oct  8 18:01:30 sher corosync[1320]:  [MAIN  ] Completed service synchronization, ready to provide service.
Oct  8 18:01:30 sher pacemakerd[1458]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node lock[102] - state is now member (was lost)
Oct  8 18:01:32 sher exportfs(res_exportfs_root)[5652]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:01:37 sher exportfs(res_exportfs_export1)[5666]: INFO: Directory /mnt/vms/export1 is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:01:43 sher exportfs(res_exportfs_root)[5738]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:01:47 sher exportfs(res_exportfs_export1)[5779]: INFO: Directory /mnt/vms/export1 is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:01:49 sher crmd[1465]:   notice: do_state_transition: State transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Oct  8 18:01:49 sher crmd[1465]:   notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Oct  8 18:01:49 sher attrd[1463]:   notice: attrd_local_callback: Sending full refresh (origin=crmd)
Oct  8 18:01:49 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-res_exportfs_root (2)
Oct  8 18:01:49 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: master-res_drbd_export (10000)
Oct  8 18:01:49 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-res_exportfs_root (1444306659)
Oct  8 18:01:49 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Oct  8 18:01:49 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-fence_lock (INFINITY)
Oct  8 18:01:49 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-fence_lock (1444306771)
Oct  8 18:01:50 sher pengine[1464]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Oct  8 18:01:50 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op monitor for res_exportfs_root:0 on sher: not running (7)
Oct  8 18:01:50 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op start for fence_lock on sher: unknown error (1)
Oct  8 18:01:50 sher pengine[1464]:  warning: common_apply_stickiness: Forcing fence_lock away from sher after 1000000 failures (max=1000000)
Oct  8 18:01:50 sher pengine[1464]:   notice: LogActions: Start   fence_sher#011(lock)
Oct  8 18:01:50 sher pengine[1464]:   notice: LogActions: Start   res_drbd_export:1#011(lock)
Oct  8 18:01:50 sher pengine[1464]:   notice: LogActions: Start   res_exportfs_root:1#011(lock)
Oct  8 18:01:50 sher pengine[1464]:   notice: LogActions: Start   res_nfsserver:1#011(lock)
Oct  8 18:01:50 sher pengine[1464]:   notice: process_pe_message: Calculated Transition 10: /var/lib/pacemaker/pengine/pe-input-2641.bz2
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 9: monitor fence_sher_monitor_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 10: monitor fence_lock_monitor_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 11: monitor res_drbd_export:1_monitor_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 12: monitor res_fs_monitor_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 13: monitor res_ip_monitor_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 14: monitor res_exportfs_export1_monitor_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 15: monitor res_exportfs_root:1_monitor_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 16: monitor res_nfsserver:1_monitor_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 79: notify res_drbd_export_pre_notify_start_0 on sher (local)
Oct  8 18:01:50 sher crmd[1465]:   notice: process_lrm_event: LRM operation res_drbd_export_notify_0 (call=132, rc=0, cib-update=0, confirmed=true) ok
Oct  8 18:01:50 sher crmd[1465]:  warning: status_from_rc: Action 14 (res_exportfs_export1_monitor_0) on lock failed (target: 7 vs. rc: 0): Error
Oct  8 18:01:50 sher crmd[1465]:  warning: status_from_rc: Action 15 (res_exportfs_root:1_monitor_0) on lock failed (target: 7 vs. rc: 0): Error
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 8: probe_complete probe_complete on lock - no waiting
Oct  8 18:01:50 sher crmd[1465]:   notice: run_graph: Transition 10 (Complete=15, Pending=0, Fired=0, Skipped=9, Incomplete=7, Source=/var/lib/pacemaker/pengine/pe-input-2641.bz2): Stopped
Oct  8 18:01:50 sher pengine[1464]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Oct  8 18:01:50 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op monitor for res_exportfs_root:1 on sher: not running (7)
Oct  8 18:01:50 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op start for fence_lock on sher: unknown error (1)
Oct  8 18:01:50 sher pengine[1464]:  warning: common_apply_stickiness: Forcing fence_lock away from sher after 1000000 failures (max=1000000)
Oct  8 18:01:50 sher pengine[1464]:    error: native_create_actions: Resource res_exportfs_export1 (ocf::exportfs) is active on 2 nodes attempting recovery
Oct  8 18:01:50 sher pengine[1464]:  warning: native_create_actions: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information.
Oct  8 18:01:50 sher pengine[1464]:   notice: LogActions: Start   fence_sher#011(lock)
Oct  8 18:01:50 sher pengine[1464]:   notice: LogActions: Start   res_drbd_export:1#011(lock)
Oct  8 18:01:50 sher pengine[1464]:   notice: LogActions: Move    res_exportfs_export1#011(Started lock -> sher)
Oct  8 18:01:50 sher pengine[1464]:   notice: LogActions: Start   res_nfsserver:1#011(lock)
Oct  8 18:01:50 sher pengine[1464]:    error: process_pe_message: Calculated Transition 11: /var/lib/pacemaker/pengine/pe-error-303.bz2
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 10: start fence_sher_start_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 47: stop res_exportfs_export1_stop_0 on sher (local)
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 46: stop res_exportfs_export1_stop_0 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 55: monitor res_exportfs_root_monitor_10000 on lock
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 8: probe_complete probe_complete on lock - no waiting
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 73: notify res_drbd_export_pre_notify_start_0 on sher (local)
Oct  8 18:01:50 sher crmd[1465]:   notice: te_rsc_command: Initiating action 64: start res_nfsserver:1_start_0 on lock
Oct  8 18:01:50 sher exportfs(res_exportfs_export1)[5874]: INFO: Directory /mnt/vms/export1 is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:01:50 sher crmd[1465]:  warning: status_from_rc: Action 46 (res_exportfs_export1_stop_0) on lock failed (target: 0 vs. rc: 5): Error
Oct  8 18:01:50 sher crmd[1465]:  warning: update_failcount: Updating failcount for res_exportfs_export1 on lock after failed stop: rc=5 (update=INFINITY, time=1444307510)
Oct  8 18:01:50 sher exportfs(res_exportfs_export1)[5874]: INFO: Un-exporting file system ...
Oct  8 18:01:50 sher exportfs(res_exportfs_export1)[5874]: INFO: unexporting 10.105.0.0/255.255.0.0:/mnt/vms/export1
Oct  8 18:01:50 sher exportfs(res_exportfs_export1)[5874]: INFO: Sleeping 92 seconds to accommodate for NFSv4 lease expiry
Oct  8 18:01:50 sher crmd[1465]:  warning: update_failcount: Updating failcount for res_exportfs_export1 on lock after failed stop: rc=5 (update=INFINITY, time=1444307510)
Oct  8 18:01:50 sher crmd[1465]:   notice: process_lrm_event: LRM operation res_drbd_export_notify_0 (call=139, rc=0, cib-update=0, confirmed=true) ok
Oct  8 18:01:53 sher exportfs(res_exportfs_root)[5929]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:02:00 sher crmd[1465]:  warning: status_from_rc: Action 55 (res_exportfs_root_monitor_10000) on lock failed (target: 0 vs. rc: 7): Error
Oct  8 18:02:00 sher crmd[1465]:  warning: update_failcount: Updating failcount for res_exportfs_root on lock after failed monitor: rc=7 (update=value++, time=1444307520)
Oct  8 18:02:03 sher exportfs(res_exportfs_root)[6005]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:02:10 sher lrmd[1462]:  warning: child_timeout_callback: res_exportfs_export1_stop_0 process (PID 5874) timed out
Oct  8 18:02:10 sher lrmd[1462]:  warning: operation_finished: res_exportfs_export1_stop_0:5874 - timed out after 20000ms
Oct  8 18:02:00 sher crmd[1465]:  warning: update_failcount: Updating failcount for res_exportfs_root on lock after failed monitor: rc=7 (update=value++, time=1444307520)
Oct  8 18:02:10 sher crmd[1465]:    error: process_lrm_event: LRM operation res_exportfs_export1_stop_0 (136) Timed Out (timeout=20000ms)
Oct  8 18:02:10 sher crmd[1465]:  warning: status_from_rc: Action 47 (res_exportfs_export1_stop_0) on sher failed (target: 0 vs. rc: 1): Error
Oct  8 18:02:10 sher crmd[1465]:  warning: update_failcount: Updating failcount for res_exportfs_export1 on sher after failed stop: rc=1 (update=INFINITY, time=1444307530)
Oct  8 18:02:10 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-res_exportfs_export1 (INFINITY)
Oct  8 18:02:10 sher attrd[1463]:   notice: attrd_perform_update: Sent update 86: fail-count-res_exportfs_export1=INFINITY
Oct  8 18:02:10 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-res_exportfs_export1 (1444307530)
Oct  8 18:02:10 sher attrd[1463]:   notice: attrd_perform_update: Sent update 89: last-failure-res_exportfs_export1=1444307530
Oct  8 18:02:10 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-res_exportfs_export1 (INFINITY)
Oct  8 18:02:10 sher attrd[1463]:   notice: attrd_perform_update: Sent update 91: fail-count-res_exportfs_export1=INFINITY
Oct  8 18:02:10 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-res_exportfs_export1 (1444307530)
Oct  8 18:02:10 sher attrd[1463]:   notice: attrd_perform_update: Sent update 93: last-failure-res_exportfs_export1=1444307530
Oct  8 18:02:13 sher exportfs(res_exportfs_root)[6081]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:02:23 sher exportfs(res_exportfs_root)[6157]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:02:33 sher exportfs(res_exportfs_root)[6260]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:02:43 sher exportfs(res_exportfs_root)[6336]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:02:53 sher exportfs(res_exportfs_root)[6412]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:03:03 sher exportfs(res_exportfs_root)[6488]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:03:13 sher exportfs(res_exportfs_root)[6564]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).


Oct  8 18:02:10 sher crmd[1465]:  warning: update_failcount: Updating failcount for res_exportfs_export1 on sher after failed stop: rc=1 (update=INFINITY, time=1444307530)
Oct  8 18:03:22 sher crmd[1465]:   notice: run_graph: Transition 11 (Complete=11, Pending=0, Fired=0, Skipped=12, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-error-303.bz2): Stopped
Oct  8 18:03:22 sher pengine[1464]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Oct  8 18:03:22 sher pengine[1464]:   notice: unpack_rsc_op: Preventing res_exportfs_export1 from re-starting on lock: operation stop failed 'not installed' (rc=5)
Oct  8 18:03:22 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op stop for res_exportfs_export1 on lock: not installed (5)
Oct  8 18:03:22 sher pengine[1464]:  warning: pe_fence_node: Node lock will be fenced because of resource failure(s)
Oct  8 18:03:22 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op monitor for res_exportfs_root:0 on lock: not running (7)
Oct  8 18:03:22 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op stop for res_exportfs_export1 on sher: unknown error (1)
Oct  8 18:03:22 sher pengine[1464]:  warning: pe_fence_node: Node sher will be fenced because of resource failure(s)
Oct  8 18:03:22 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op monitor for res_exportfs_root:1 on sher: not running (7)
Oct  8 18:03:22 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op start for fence_lock on sher: unknown error (1)
Oct  8 18:03:22 sher pengine[1464]:  warning: common_apply_stickiness: Forcing res_exportfs_export1 away from lock after 1000000 failures (max=1000000)
Oct  8 18:03:22 sher pengine[1464]:  warning: common_apply_stickiness: Forcing fence_lock away from sher after 1000000 failures (max=1000000)
Oct  8 18:03:22 sher pengine[1464]:  warning: common_apply_stickiness: Forcing res_exportfs_export1 away from sher after 1000000 failures (max=1000000)
Oct  8 18:03:22 sher pengine[1464]:    error: native_create_actions: Resource res_exportfs_export1 (ocf::exportfs) is active on 2 nodes attempting recovery
Oct  8 18:03:22 sher pengine[1464]:  warning: native_create_actions: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information.
Oct  8 18:03:22 sher pengine[1464]:  warning: stage6: Scheduling Node lock for STONITH
Oct  8 18:03:22 sher pengine[1464]:   notice: native_stop_constraints: Stop of failed resource res_exportfs_export1 is implicit after lock is fenced
Oct  8 18:03:22 sher pengine[1464]:  warning: stage6: Scheduling Node sher for STONITH
Oct  8 18:03:22 sher pengine[1464]:   notice: native_stop_constraints: Stop of failed resource res_exportfs_export1 is implicit after sher is fenced
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Stop    fence_sher#011(lock)
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Demote  res_drbd_export:0#011(Master -> Stopped sher)
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Stop    res_fs#011(sher)
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Stop    res_ip#011(sher)
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Stop    res_exportfs_export1#011(lock)
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Stop    res_exportfs_export1#011(sher)
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Stop    res_exportfs_root:0#011(lock)
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Stop    res_exportfs_root:1#011(sher)
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Stop    res_nfsserver:0#011(lock)
Oct  8 18:03:22 sher pengine[1464]:   notice: LogActions: Stop    res_nfsserver:1#011(sher)
Oct  8 18:03:22 sher pengine[1464]:    error: process_pe_message: Calculated Transition 12: /var/lib/pacemaker/pengine/pe-error-304.bz2
Oct  8 18:03:22 sher crmd[1465]:   notice: te_fence_node: Executing reboot fencing operation (56) on lock (timeout=60000)
Oct  8 18:03:22 sher crmd[1465]:   notice: te_rsc_command: Initiating action 73: notify res_drbd_export_pre_notify_demote_0 on sher (local)
Oct  8 18:03:22 sher stonith-ng[1461]:   notice: handle_request: Client crmd.1465.e3ae1cca wants to fence (reboot) 'lock' with device '(any)'
Oct  8 18:03:22 sher stonith-ng[1461]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for lock: 375aeb28-a774-4c2f-82c4-a1c0250d8265 (0)
Oct  8 18:03:22 sher crmd[1465]:   notice: process_lrm_event: LRM operation res_drbd_export_notify_0 (call=143, rc=0, cib-update=0, confirmed=true) ok
Oct  8 18:03:22 sher stonith-ng[1461]:   notice: can_fence_host_with_device: fence_lock can fence lock: dynamic-list
Oct  8 18:03:23 sher exportfs(res_exportfs_root)[6686]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:03:23 sher kernel: [ 1009.741795] e1000e: eth0 NIC Link is Down
Oct  8 18:03:23 sher corosync[1320]:  [TOTEM ] A processor failed, forming new configuration.
Oct  8 18:03:24 sher stonith-ng[1461]:   notice: log_operation: Operation 'reboot' [6670] (call 3 from crmd.1465) for host 'lock' with device 'fence_lock' returned: 0 (OK)
Oct  8 18:03:25 sher corosync[1320]:  [TOTEM ] A new membership (192.168.0.21:3448) was formed. Members left: 102
Oct  8 18:03:25 sher corosync[1320]:  [QUORUM] Members[1]: 101
Oct  8 18:03:25 sher corosync[1320]:  [MAIN  ] Completed service synchronization, ready to provide service.
Oct  8 18:03:25 sher crmd[1465]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node lock[102] - state is now lost (was member)
Oct  8 18:03:25 sher pacemakerd[1458]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node lock[102] - state is now lost (was member)
Oct  8 18:03:25 sher kernel: [ 1011.487119] e1000e: eth0 NIC Link is Up 10 Mbps Full Duplex, Flow Control: None
Oct  8 18:03:25 sher kernel: [ 1011.487228] e1000e 0000:00:19.0 eth0: 10/100 speed: disabling TSO
Oct  8 18:03:25 sher stonith-ng[1461]:   notice: remote_op_done: Operation reboot of lock by sher for crmd.1465@sher.375aeb28: OK
Oct  8 18:03:25 sher crmd[1465]:   notice: tengine_stonith_callback: Stonith operation 3/56:12:0:6643f7d4-dd65-49dc-a9db-c9ede2838434: OK (0)
Oct  8 18:03:25 sher crmd[1465]:   notice: tengine_stonith_notify: Peer lock was terminated (reboot) by sher for sher: OK (ref=375aeb28-a774-4c2f-82c4-a1c0250d8265) by client crmd.1465
Oct  8 18:03:25 sher crmd[1465]:   notice: te_fence_node: Executing reboot fencing operation (57) on sher (timeout=60000)
Oct  8 18:03:25 sher stonith-ng[1461]:   notice: handle_request: Client crmd.1465.e3ae1cca wants to fence (reboot) 'sher' with device '(any)'
Oct  8 18:03:25 sher stonith-ng[1461]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for sher: 57357f57-d2e5-4254-ba47-716662b916a9 (0)
Oct  8 18:03:25 sher stonith-ng[1461]:   notice: can_fence_host_with_device: fence_lock can not fence sher: dynamic-list
Oct  8 18:03:25 sher stonith-ng[1461]:    error: remote_op_done: Operation reboot of sher by sher for crmd.1465@sher.57357f57: No such device
Oct  8 18:03:25 sher crmd[1465]:   notice: tengine_stonith_callback: Stonith operation 4/57:12:0:6643f7d4-dd65-49dc-a9db-c9ede2838434: No such device (-19)
Oct  8 18:03:25 sher crmd[1465]:   notice: tengine_stonith_callback: Stonith operation 4 for sher failed (No such device): aborting transition.
Oct  8 18:03:25 sher crmd[1465]:   notice: tengine_stonith_notify: Peer sher was not terminated (reboot) by sher for sher: No such device (ref=57357f57-d2e5-4254-ba47-716662b916a9) by client crmd.1465
Oct  8 18:03:25 sher crmd[1465]:   notice: run_graph: Transition 12 (Complete=9, Pending=0, Fired=0, Skipped=20, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-error-304.bz2): Stopped
Oct  8 18:03:25 sher crmd[1465]:   notice: too_many_st_failures: No devices found in cluster to fence sher, giving up
Oct  8 18:03:25 sher crmd[1465]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Oct  8 18:03:25 sher crmd[1465]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Oct  8 18:03:25 sher pengine[1464]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Oct  8 18:03:25 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op stop for res_exportfs_export1 on sher: unknown error (1)
Oct  8 18:03:25 sher pengine[1464]:  warning: pe_fence_node: Node sher will be fenced because of resource failure(s)
Oct  8 18:03:25 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op monitor for res_exportfs_root:0 on sher: not running (7)
Oct  8 18:03:25 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op start for fence_lock on sher: unknown error (1)
Oct  8 18:03:25 sher pengine[1464]:  warning: common_apply_stickiness: Forcing fence_lock away from sher after 1000000 failures (max=1000000)
Oct  8 18:03:25 sher pengine[1464]:  warning: common_apply_stickiness: Forcing res_exportfs_export1 away from sher after 1000000 failures (max=1000000)
Oct  8 18:03:25 sher pengine[1464]:  warning: stage6: Scheduling Node sher for STONITH
Oct  8 18:03:25 sher pengine[1464]:   notice: native_stop_constraints: Stop of failed resource res_exportfs_export1 is implicit after sher is fenced
Oct  8 18:03:25 sher pengine[1464]:   notice: LogActions: Demote  res_drbd_export:0#011(Master -> Stopped sher)
Oct  8 18:03:25 sher pengine[1464]:   notice: LogActions: Stop    res_fs#011(sher)
Oct  8 18:03:25 sher pengine[1464]:   notice: LogActions: Stop    res_ip#011(sher)
Oct  8 18:03:25 sher pengine[1464]:   notice: LogActions: Stop    res_exportfs_export1#011(sher)
Oct  8 18:03:25 sher pengine[1464]:   notice: LogActions: Stop    res_exportfs_root:0#011(sher)
Oct  8 18:03:25 sher pengine[1464]:   notice: LogActions: Stop    res_nfsserver:0#011(sher)
Oct  8 18:03:25 sher pengine[1464]:  warning: process_pe_message: Calculated Transition 13: /var/lib/pacemaker/pengine/pe-warn-364.bz2
Oct  8 18:03:25 sher crmd[1465]:   notice: te_fence_node: Executing reboot fencing operation (51) on sher (timeout=60000)
Oct  8 18:03:25 sher stonith-ng[1461]:   notice: handle_request: Client crmd.1465.e3ae1cca wants to fence (reboot) 'sher' with device '(any)'
Oct  8 18:03:25 sher crmd[1465]:   notice: te_rsc_command: Initiating action 67: notify res_drbd_export_pre_notify_demote_0 on sher (local)
Oct  8 18:03:25 sher stonith-ng[1461]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for sher: 8bc5d6a4-b32e-4424-80c5-83f686b4f2a0 (0)
Oct  8 18:03:25 sher stonith-ng[1461]:   notice: can_fence_host_with_device: fence_lock can not fence sher: dynamic-list
Oct  8 18:03:25 sher stonith-ng[1461]:    error: remote_op_done: Operation reboot of sher by sher for crmd.1465@sher.8bc5d6a4: No such device
Oct  8 18:03:25 sher crmd[1465]:   notice: tengine_stonith_callback: Stonith operation 5/51:13:0:6643f7d4-dd65-49dc-a9db-c9ede2838434: No such device (-19)
Oct  8 18:03:25 sher crmd[1465]:   notice: tengine_stonith_callback: Stonith operation 5 for sher failed (No such device): aborting transition.
Oct  8 18:03:25 sher crmd[1465]:   notice: tengine_stonith_notify: Peer sher was not terminated (reboot) by sher for sher: No such device (ref=8bc5d6a4-b32e-4424-80c5-83f686b4f2a0) by client crmd.1465
Oct  8 18:03:25 sher crmd[1465]:   notice: process_lrm_event: LRM operation res_drbd_export_notify_0 (call=146, rc=0, cib-update=0, confirmed=true) ok
Oct  8 18:03:25 sher crmd[1465]:   notice: run_graph: Transition 13 (Complete=5, Pending=0, Fired=0, Skipped=19, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-warn-364.bz2): Stopped
Oct  8 18:03:25 sher crmd[1465]:   notice: too_many_st_failures: No devices found in cluster to fence sher, giving up
Oct  8 18:03:25 sher crmd[1465]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Oct  8 18:03:31 sher kernel: [ 1017.803975] e1000e: eth0 NIC Link is Down
Oct  8 18:03:33 sher exportfs(res_exportfs_root)[6784]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:03:33 sher kernel: [ 1020.262569] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Oct  8 18:03:43 sher exportfs(res_exportfs_root)[6860]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:03:53 sher kernel: [ 1039.432561] e1000e: eth0 NIC Link is Down
Oct  8 18:03:53 sher exportfs(res_exportfs_root)[6936]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:03:55 sher kernel: [ 1041.895210] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Oct  8 18:04:03 sher exportfs(res_exportfs_root)[7039]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:04:13 sher exportfs(res_exportfs_root)[7115]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:04:23 sher exportfs(res_exportfs_root)[7191]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:04:33 sher exportfs(res_exportfs_root)[7267]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).


Oct  8 18:04:43 sher exportfs(res_exportfs_root)[7343]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:04:46 sher kernel: [ 1092.757491] e1000e: eth0 NIC Link is Down
Oct  8 18:04:53 sher exportfs(res_exportfs_root)[7419]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:04:56 sher kernel: [ 1102.481657] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Oct  8 18:04:56 sher corosync[1320]:  [TOTEM ] A new membership (192.168.0.21:3452) was formed. Members joined: 102
Oct  8 18:04:56 sher corosync[1320]:  [QUORUM] Members[2]: 101 102
Oct  8 18:04:56 sher corosync[1320]:  [MAIN  ] Completed service synchronization, ready to provide service.
Oct  8 18:04:56 sher crmd[1465]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node lock[102] - state is now member (was lost)
Oct  8 18:04:56 sher pacemakerd[1458]:   notice: crm_update_peer_state: pcmk_quorum_notification: Node lock[102] - state is now member (was lost)
Oct  8 18:05:03 sher exportfs(res_exportfs_root)[7495]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:05:13 sher exportfs(res_exportfs_root)[7571]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).
Oct  8 18:05:15 sher crmd[1465]:   notice: do_state_transition: State transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Oct  8 18:05:15 sher crmd[1465]:   notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Oct  8 18:05:15 sher attrd[1463]:   notice: attrd_local_callback: Sending full refresh (origin=crmd)
Oct  8 18:05:15 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-res_exportfs_export1 (INFINITY)
Oct  8 18:05:15 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-res_exportfs_root (2)
Oct  8 18:05:15 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: master-res_drbd_export (10000)
Oct  8 18:05:15 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-res_exportfs_export1 (1444307530)
Oct  8 18:05:15 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-res_exportfs_root (1444306659)
Oct  8 18:05:15 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Oct  8 18:05:15 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-fence_lock (INFINITY)
Oct  8 18:05:15 sher attrd[1463]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-fence_lock (1444306771)
Oct  8 18:05:16 sher pengine[1464]:   notice: unpack_config: On loss of CCM Quorum: Ignore
Oct  8 18:05:16 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op stop for res_exportfs_export1 on sher: unknown error (1)
Oct  8 18:05:16 sher pengine[1464]:  warning: pe_fence_node: Node sher will be fenced because of resource failure(s)
Oct  8 18:05:16 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op monitor for res_exportfs_root:0 on sher: not running (7)
Oct  8 18:05:16 sher pengine[1464]:  warning: unpack_rsc_op: Processing failed op start for fence_lock on sher: unknown error (1)
Oct  8 18:05:16 sher pengine[1464]:  warning: common_apply_stickiness: Forcing fence_lock away from sher after 1000000 failures (max=1000000)
Oct  8 18:05:16 sher pengine[1464]:  warning: common_apply_stickiness: Forcing res_exportfs_export1 away from sher after 1000000 failures (max=1000000)
Oct  8 18:05:16 sher pengine[1464]:  warning: stage6: Scheduling Node sher for STONITH
Oct  8 18:05:16 sher pengine[1464]:   notice: native_stop_constraints: Stop of failed resource res_exportfs_export1 is implicit after sher is fenced
Oct  8 18:05:16 sher pengine[1464]:   notice: LogActions: Start   fence_sher#011(lock)
Oct  8 18:05:16 sher pengine[1464]:   notice: LogActions: Demote  res_drbd_export:0#011(Master -> Slave sher - blocked)
Oct  8 18:05:16 sher pengine[1464]:   notice: LogActions: Move    res_drbd_export:0#011(Slave sher -> lock)
Oct  8 18:05:16 sher pengine[1464]:   notice: LogActions: Stop    res_fs#011(sher)
Oct  8 18:05:16 sher pengine[1464]:   notice: LogActions: Stop    res_ip#011(sher)
Oct  8 18:05:16 sher pengine[1464]:   notice: LogActions: Stop    res_exportfs_export1#011(sher)
Oct  8 18:05:16 sher pengine[1464]:   notice: LogActions: Move    res_exportfs_root:0#011(Started sher -> lock)
Oct  8 18:05:16 sher pengine[1464]:   notice: LogActions: Move    res_nfsserver:0#011(Started sher -> lock)
Oct  8 18:05:16 sher pengine[1464]:  warning: process_pe_message: Calculated Transition 14: /var/lib/pacemaker/pengine/pe-warn-365.bz2
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 8: monitor fence_sher_monitor_0 on lock
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 9: monitor fence_lock_monitor_0 on lock
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 10: monitor res_drbd_export_monitor_0 on lock
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 11: monitor res_fs_monitor_0 on lock
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 12: monitor res_ip_monitor_0 on lock
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 13: monitor res_exportfs_export1_monitor_0 on lock
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 14: monitor res_exportfs_root_monitor_0 on lock
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 15: monitor res_nfsserver_monitor_0 on lock
Oct  8 18:05:16 sher crmd[1465]:   notice: te_fence_node: Executing reboot fencing operation (68) on sher (timeout=60000)
Oct  8 18:05:16 sher stonith-ng[1461]:   notice: handle_request: Client crmd.1465.e3ae1cca wants to fence (reboot) 'sher' with device '(any)'
Oct  8 18:05:16 sher stonith-ng[1461]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for sher: a98028b6-08e1-4641-9268-06c449cfde86 (0)
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 82: notify res_drbd_export_pre_notify_demote_0 on sher (local)
Oct  8 18:05:16 sher crmd[1465]:   notice: process_lrm_event: LRM operation res_drbd_export_notify_0 (call=149, rc=0, cib-update=0, confirmed=true) ok
Oct  8 18:05:16 sher crmd[1465]:   notice: te_rsc_command: Initiating action 7: probe_complete probe_complete on lock - no waiting
Oct  8 18:05:23 sher exportfs(res_exportfs_root)[7699]: INFO: Directory /mnt/vms is exported to 10.105.0.0/255.255.0.0 (started).




rebooted!



