[ClusterLabs] Pacemaker fatal shutdown
Ken Gaillot
kgaillot at redhat.com
Tue Jul 25 13:43:39 EDT 2023
On Thu, 2023-07-20 at 12:43 +0530, Priyanka Balotra wrote:
> What I mainly want to understand is that:
> - why "fatal failure" is coming
The logs so far don't show that. The earliest sign is:
Jul 17 14:18:20.085 FILE-6 pacemaker-fenced [19411]
(remote_op_done) notice: Operation 'reboot' targeting FILE-2 by FILE-
4 for pacemaker-controld.19415 at FILE-6: OK | id=4e523b34
You'd want to figure out which node was the Designated Controller (DC)
at that time, and look at its logs before this time. The DC will have
"Calculated transition" log messages.
You want to find such messages just before the timestamp above. If you
look above the "Calculated transition" message, it will show what
actions the cluster wants to take, including fencing. The logs around
there should say why the fencing was needed.
> - why does pacemaker not start on the node after a node boots
> followed by "pacemaker fatal failure" .
A fatal failure is one where Pacemaker should stay down, so that's what
it does. In this case, fencing completed against the node, but the node
was still alive, so it shuts down and waits for manual intervention to
figure out what happened.
> - How can this be handled?
In a situation like this, figure out (1) why fencing was needed and (2)
why successful fencing did not kill the node (if you're using fabric
fencing such as SCSI fencing, that could be a reason, otherwise it
might be a misconfiguration).
Once you know that, it should be fairly obvious what to do about it,
and once it's taken care of, you can manually start Pacemaker on the
node again.
>
> Thanks
> Priyanka
>
> On Thu, Jul 20, 2023 at 12:41 PM Priyanka Balotra <
> priyanka.14balotra at gmail.com> wrote:
> > Hi,
> >
> > Here are FILE-6 logs:
> >
> > 65710:Jul 17 14:16:51.517 FILE-6 pacemaker-controld [19415]
> > (throttle_mode) debug: Current load is 0.760000 across 10
> > core(s)
> > 65711:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (throttle_update) debug: Node FILE-2 has negligible load and
> > supports at most 20 jobs; new job limit 20
> > 65712:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (handle_request) debug: The throttle changed. Trigger a graph.
> > 65713:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x00020000
> > (new_actions) for controller set by s_crmd_fsa:198
> > 65714:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_JOIN_REQUEST: [
> > state=S_INTEGRATION cause=C_HA_MESSAGE origin=route_message ]
> > 65715:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00020000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65716:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_dc_join_filter_offer) debug: Accepting join-1 request from
> > FILE-2 | ref=join_request-crmd-1689603392-8
> > 65717:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__update_peer_expected) info: do_dc_join_filter_offer:
> > Node FILE-2[2] - expected state is now member (was (null))
> > 65718:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_dc_join_filter_offer) debug: 2 nodes currently integrated in
> > join-1
> > 65719:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (check_join_state) debug: join-1: Integration of 2 peers
> > complete | state=S_INTEGRATION for=do_dc_join_filter_offer
> > 65720:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x00040000
> > (new_actions) for controller set by s_crmd_fsa:198
> > 65721:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_INTEGRATED: [
> > state=S_INTEGRATION cause=C_FSA_INTERNAL origin=check_join_state ]
> > 65722:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_state_transition) info: State transition S_INTEGRATION ->
> > S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL
> > origin=check_join_state
> > 65723:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x00000020
> > (A_INTEGRATE_TIMER_STOP) for controller set by
> > do_state_transition:559
> > 65724:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x00000040
> > (A_FINALIZE_TIMER_START) for controller set by
> > do_state_transition:563
> > 65725:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x00000200
> > (A_DC_TIMER_STOP) for controller set by do_state_transition:569
> > 65726:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_state_transition) debug: All cluster nodes (2) responded
> > to join offer
> > 65727:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00000200
> > (an_action) for controller cleared by do_fsa_action:108
> > 65728:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00000020
> > (an_action) for controller cleared by do_fsa_action:108
> > 65729:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00000040
> > (an_action) for controller cleared by do_fsa_action:108
> > 65730:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (controld_start_timer) debug: Started Finalization Timer
> > (inject I_ELECTION if pops after 1800000ms, source=119)
> > 65731:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00040000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65732:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_dc_join_finalize) debug: Finalizing join-1 for 2 nodes
> > (sync'ing from local CIB)
> > 65733:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_dc_join_finalize) debug: Requested CIB version
> > <generation_tuple crm_feature_set="3.11.0" validate-
> > with="pacemaker-3.7" epoch="24" num_updates="72" admin_epoch="0"
> > cib-last-written="Thu Jul 13 13:11:46 2023" update-origin="FILE-1"
> > update-client="cibadmin" update-user="root" have-quorum="1" dc-
> > uuid="6"/>
> > 65734:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-6=integrated
> > 65735:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-2=integrated
> > 65736:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-3=confirmed
> > 65737:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-1=none
> > 65738:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-5=confirmed
> > 65739:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-4=confirmed
> > 65740:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
> > (fsa_data->actions) for controller set by s_crmd_fsa:193
> > 65741:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags
> > 0x1000000000000000 (new_actions) for controller set by
> > s_crmd_fsa:198
> > 65742:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
> > 65743:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags
> > 0x1000000000000000 (an_action) for controller cleared by
> > do_fsa_action:108
> > 65744:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_log) info: Input I_WAIT_FOR_EVENT received in state
> > S_FINALIZE_JOIN from do_te_invoke
> > 65745:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_log) debug: do_log <create_request_adv
> > origin="do_cl_join_query" t="crmd" version="3.11.0" subt="request"
> > reference="join_announce-crmd-1689603376-2"
> > crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd"
> > src="FILE-1"/>
> > 65746:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65747:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
> > source=do_te_invoke:135 complete=false
> > 65748:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (register_fsa_input_adv) debug: Stalling the FSA pending further
> > input: source=do_te_invoke cause=C_HA_MESSAGE data=0x55c6194ed4c0
> > queue=0
> > 65749:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Exiting the FSA: queue=1,
> > fsa_actions=0x0, stalled=true
> > 65750:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (fsa_dump_queue) debug: queue[0.72]: input I_WAIT_FOR_EVENT
> > raised by do_te_invoke(0x55c619869580.1) (cause=C_HA_MESSAGE)
> > 65751:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
> > (fsa_data->actions) for controller set by s_crmd_fsa:193
> > 65752:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags
> > 0x1000000000000000 (new_actions) for controller set by
> > s_crmd_fsa:198
> > 65753:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
> > 65754:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags
> > 0x1000000000000000 (an_action) for controller cleared by
> > do_fsa_action:108
> > 65755:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_log) info: Input I_WAIT_FOR_EVENT received in state
> > S_FINALIZE_JOIN from do_te_invoke
> > 65756:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (do_log) debug: do_log <create_request_adv
> > origin="do_cl_join_query" t="crmd" version="3.11.0" subt="request"
> > reference="join_announce-crmd-1689603376-2"
> > crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd"
> > src="FILE-1"/>
> > 65757:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65758:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
> > source=do_te_invoke:135 complete=false
> > 65759:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (register_fsa_input_adv) debug: Stalling the FSA pending further
> > input: source=do_te_invoke cause=C_HA_MESSAGE data=0x55c619869580
> > queue=0
> > 65760:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Exiting the FSA: queue=1,
> > fsa_actions=0x0, stalled=true
> > 65761:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (fsa_dump_queue) debug: queue[0.73]: input I_WAIT_FOR_EVENT
> > raised by do_te_invoke(0x55c6194ed4c0.1) (cause=C_HA_MESSAGE)
> > 65762:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__execute_graph) debug: Transition 0 (Complete=33,
> > Pending=2, Fired=0, Skipped=0, Incomplete=24,
> > Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
> > 65764:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (check_join_state) debug: join-1: Still waiting on 2
> > integrated nodes | state=S_FINALIZE_JOIN for=finalize_sync_callback
> > 65765:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-6=integrated
> > 65766:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-2=integrated
> > 65767:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-3=confirmed
> > 65768:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-1=none
> > 65769:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-5=confirmed
> > 65770:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-4=confirmed
> > 65771:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (finalize_sync_callback) debug: Notifying 2 nodes of join-1
> > results
> > 65772:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (finalize_join_for) debug: Acknowledging join-1 request from
> > FILE-6
> > 65773:Jul 17 14:16:55.085 FILE-6 pacemaker-controld [19415]
> > (finalize_join_for) debug: Acknowledging join-1 request from
> > FILE-2
> > 65776:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (handle_request) debug: Raising I_JOIN_RESULT: join-1
> > 65777:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
> > (fsa_data->actions) for controller set by s_crmd_fsa:193
> > 65778:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags
> > 0x1000000000000000 (new_actions) for controller set by
> > s_crmd_fsa:198
> > 65779:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
> > 65780:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags
> > 0x1000000000000000 (an_action) for controller cleared by
> > do_fsa_action:108
> > 65781:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (do_log) info: Input I_WAIT_FOR_EVENT received in state
> > S_FINALIZE_JOIN from do_te_invoke
> > 65782:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (do_log) debug: do_log <create_request_adv
> > origin="do_cl_join_query" t="crmd" version="3.11.0" subt="request"
> > reference="join_announce-crmd-1689603376-2"
> > crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd"
> > src="FILE-1"/>
> > 65783:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65784:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
> > source=do_te_invoke:135 complete=false
> > 65785:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (register_fsa_input_adv) debug: Stalling the FSA pending further
> > input: source=do_te_invoke cause=C_HA_MESSAGE data=0x55c6194ed4c0
> > queue=1
> > 65786:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Exiting the FSA: queue=2,
> > fsa_actions=0x0, stalled=true
> > 65787:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (fsa_dump_queue) debug: queue[0.74]: input I_JOIN_RESULT raised
> > by route_message(0x55c619861a90.1) (cause=C_HA_MESSAGE)
> > 65788:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (fsa_dump_queue) debug: queue[1.75]: input I_WAIT_FOR_EVENT
> > raised by do_te_invoke(0x55c61986ed80.1) (cause=C_HA_MESSAGE)
> > 65789:Jul 17 14:16:55.093 FILE-6 pacemaker-controld [19415]
> > (pcmk__execute_graph) debug: Transition 0 (Complete=33,
> > Pending=2, Fired=0, Skipped=0, Incomplete=24,
> > Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
> > 65792:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x00880000
> > (new_actions) for controller set by s_crmd_fsa:198
> > 65793:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_JOIN_RESULT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=route_message ]
> > 65794:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00800000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65795:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource stonith-sbd after monitor op complete
> > (interval=0)
> > 65796:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource FILE_Filesystem after monitor op complete
> > (interval=0)
> > 65797:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Service_pfile after monitor op complete
> > (interval=0)
> > 65798:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Service_Postgresql after monitor op complete
> > (interval=0)
> > 65799:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Service_esm_primary after monitor op complete
> > (interval=0)
> > 65800:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Service_Postgrest after monitor op complete
> > (interval=0)
> > 65801:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource IP_Floating after monitor op complete
> > (interval=0)
> > 65802:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Shared_Cluster_Backup after monitor op complete
> > (interval=0)
> > 65803:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (do_cl_join_finalize_respond) debug: Confirming join-1:
> > sending local operation history to FILE-6
> > 65804:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00080000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65805:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (do_dc_join_ack) debug: Ignoring 'join_ack_nack' message from
> > FILE-6 while waiting for 'join_confirm'
> > 65806:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
> > (fsa_data->actions) for controller set by s_crmd_fsa:193
> > 65807:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags
> > 0x1000000000000000 (new_actions) for controller set by
> > s_crmd_fsa:198
> > 65808:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
> > 65809:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags
> > 0x1000000000000000 (an_action) for controller cleared by
> > do_fsa_action:108
> > 65810:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (do_log) info: Input I_WAIT_FOR_EVENT received in state
> > S_FINALIZE_JOIN from do_te_invoke
> > 65811:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (do_log) debug: do_log <create_request_adv
> > origin="do_cl_join_query" t="crmd" version="3.11.0" subt="request"
> > reference="join_announce-crmd-1689603376-2"
> > crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd"
> > src="FILE-1"/>
> > 65812:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65813:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
> > source=do_te_invoke:135 complete=false
> > 65814:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (register_fsa_input_adv) debug: Stalling the FSA pending further
> > input: source=do_te_invoke cause=C_HA_MESSAGE data=0x55c61986ed80
> > queue=1
> > 65815:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Exiting the FSA: queue=2,
> > fsa_actions=0x0, stalled=true
> > 65816:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (fsa_dump_queue) debug: queue[0.76]: input I_JOIN_RESULT raised
> > by route_message(0x55c619871630.1) (cause=C_HA_MESSAGE)
> > 65817:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (fsa_dump_queue) debug: queue[1.77]: input I_WAIT_FOR_EVENT
> > raised by do_te_invoke(0x55c619861a90.1) (cause=C_HA_MESSAGE)
> > 65818:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__execute_graph) debug: Transition 0 (Complete=33,
> > Pending=2, Fired=0, Skipped=0, Incomplete=24,
> > Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
> > 65821:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x00880000
> > (new_actions) for controller set by s_crmd_fsa:198
> > 65822:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_JOIN_RESULT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=route_message ]
> > 65823:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00800000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65824:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00080000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65825:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (controld_delete_node_state) info: Deleting resource history
> > for node FILE-2 (via CIB call 71) |
> > xpath=//node_state[@uname='FILE-2']/lrm
> > 65826:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (do_dc_join_ack) debug: Updating node history for FILE-2 from
> > join-1 confirmation (via CIB call 72)
> > 65827:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
> > (fsa_data->actions) for controller set by s_crmd_fsa:193
> > 65828:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags
> > 0x1000000000000000 (new_actions) for controller set by
> > s_crmd_fsa:198
> > 65829:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
> > 65830:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags
> > 0x1000000000000000 (an_action) for controller cleared by
> > do_fsa_action:108
> > 65831:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (do_log) info: Input I_WAIT_FOR_EVENT received in state
> > S_FINALIZE_JOIN from do_te_invoke
> > 65832:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (do_log) debug: do_log <create_request_adv
> > origin="do_cl_join_query" t="crmd" version="3.11.0" subt="request"
> > reference="join_announce-crmd-1689603376-2"
> > crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd"
> > src="FILE-1"/>
> > 65833:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65834:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
> > source=do_te_invoke:135 complete=false
> > 65835:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (register_fsa_input_adv) debug: Stalling the FSA pending further
> > input: source=do_te_invoke cause=C_HA_MESSAGE data=0x55c619861a90
> > queue=1
> > 65836:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Exiting the FSA: queue=2,
> > fsa_actions=0x0, stalled=true
> > 65837:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (fsa_dump_queue) debug: queue[0.78]: input I_JOIN_RESULT raised
> > by route_message(0x55c6198798d0.1) (cause=C_HA_MESSAGE)
> > 65838:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (fsa_dump_queue) debug: queue[1.79]: input I_WAIT_FOR_EVENT
> > raised by do_te_invoke(0x55c619871630.1) (cause=C_HA_MESSAGE)
> > 65839:Jul 17 14:16:55.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__execute_graph) debug: Transition 0 (Complete=33,
> > Pending=2, Fired=0, Skipped=0, Incomplete=24,
> > Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
> > 65851:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (cib_delete_callback) debug: Deletion of resource history for
> > node FILE-2 (via CIB call 71) succeeded
> > 65861:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (te_update_diff) debug: Processing (cib_modify) diff: 0.24.72 ->
> > 0.24.73 (S_FINALIZE_JOIN)
> > 65862:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (join_update_complete_callback) debug: join-1 node history
> > update (via CIB call 72) complete
> > 65863:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (check_join_state) debug: join-1: Still waiting on 1
> > finalized node | state=S_FINALIZE_JOIN
> > for=join_update_complete_callback
> > 65864:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-6=finalized
> > 65865:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-2=confirmed
> > 65866:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-3=confirmed
> > 65867:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-1=none
> > 65868:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-5=confirmed
> > 65869:Jul 17 14:16:55.109 FILE-6 pacemaker-controld [19415]
> > (crmd_join_phase_log) debug: join-1: FILE-4=confirmed
> > 65876:Jul 17 14:17:21.517 FILE-6 pacemaker-controld [19415]
> > (throttle_cib_load) debug: cib load: 0.001000 (3 ticks in
> > 30s)
> > 65877:Jul 17 14:17:21.517 FILE-6 pacemaker-controld [19415]
> > (throttle_mode) debug: Current load is 0.960000 across 10
> > core(s)
> > 65878:Jul 17 14:17:51.517 FILE-6 pacemaker-controld [19415]
> > (throttle_cib_load) debug: cib load: 0.000333 (1 ticks in
> > 30s)
> > 65879:Jul 17 14:17:51.517 FILE-6 pacemaker-controld [19415]
> > (throttle_mode) debug: Current load is 0.580000 across 10
> > core(s)
> > 65883:Jul 17 14:18:20.085 FILE-6 pacemaker-fenced [19411]
> > (process_remote_stonith_exec) debug: Finalizing action
> > 'reboot' targeting FILE-2 on behalf of
> > pacemaker-controld.19415 at FILE-6: OK | rc=0 id=4e523b34
> > 65884:Jul 17 14:18:20.085 FILE-6 pacemaker-fenced [19411]
> > (remote_op_done) notice: Operation 'reboot' targeting FILE-2 by
> > FILE-4 for pacemaker-controld.19415 at FILE-6: OK | id=4e523b34
> > 65886:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
> > (tengine_stonith_callback) notice: Stonith operation
> > 3/63:0:0:232e6505-2e98-4a79-b6ce-5f26d9cba645: OK (0)
> > 65887:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
> > (tengine_stonith_callback) info: Stonith operation 3 for
> > FILE-2 passed
> > 65888:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
> > (pcmk__update_peer_expected) info: crmd_peer_down: Node FILE-
> > 2[2] - expected state is now down (was member)
> > 65889:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
> > (send_stonith_update) debug: Sending fencing update 73 for
> > FILE-2
> > 65890:Jul 17 14:18:20.085 FILE-6 pacemaker-controld [19415]
> > (controld_delete_node_state) info: Deleting all state for
> > node FILE-2 (via CIB call 74) | xpath=//node_state[@uname='FILE-
> > 2']/*
> > 65892:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
> > (exec_alert_list) info: Sending fencing alert via pf-ha-alert to
> > (null)
> > 65896:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
> > (tengine_stonith_notify) notice: Peer FILE-2 was terminated
> > (reboot) by FILE-4 on behalf of pacemaker-controld.19415: OK |
> > initiator=FILE-6 ref=4e523b34-dcb1-40bc-a296-5e984b4e6b00
> > 65897:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
> > (send_stonith_update) debug: Sending fencing update 75 for
> > FILE-2
> > 65898:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
> > (controld_delete_node_state) info: Deleting all state for
> > node FILE-2 (via CIB call 76) | xpath=//node_state[@uname='FILE-
> > 2']/*
> > 65899:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
> > (pcmk__execute_graph) debug: Transition 0 (Complete=34,
> > Pending=1, Fired=0, Skipped=0, Incomplete=24,
> > Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
> > 65907:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
> > (te_update_diff) debug: Processing (cib_modify) diff: 0.24.73 ->
> > 0.24.74 (S_FINALIZE_JOIN)
> > 65908:Jul 17 14:18:20.089 FILE-6 pacemaker-controld [19415]
> > (cib_fencing_updated) info: Fencing update 73 for FILE-2:
> > complete
> > 65916:Jul 17 14:18:20.093 FILE-6 pacemaker-controld [19415]
> > (te_update_diff) debug: Processing (cib_delete) diff: 0.24.74 ->
> > 0.24.75 (S_FINALIZE_JOIN)
> > 65919:Jul 17 14:18:20.093 FILE-6 pacemaker-controld [19415]
> > (match_down_event) debug: Shutdown action 63 (stonith-FILE-
> > 2-reboot) found for node 2
> > 65920:Jul 17 14:18:20.093 FILE-6 pacemaker-controld [19415]
> > (cib_delete_callback) debug: Deletion of all state for node
> > FILE-2 (via CIB call 74) succeeded
> > 65921:Jul 17 14:18:20.093 FILE-6 pacemaker-controld [19415]
> > (cib_fencing_updated) info: Fencing update 75 for FILE-2:
> > complete
> > 65924:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (cib_delete_callback) debug: Deletion of all state for node
> > FILE-2 (via CIB call 76) succeeded
> > 65927:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (node_left) info: Group crmd event 5: FILE-2 (node 2 pid
> > 15962) left for unknown reason
> > 65928:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (crm_update_peer_proc) info: node_left: Node FILE-2[2] -
> > corosync-cpg is now offline
> > 65929:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (peer_update_callback) info: Node FILE-2 is no longer a peer |
> > DC=true old=0x4000000 new=0x0000000
> > 65930:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (controld_delete_node_state) info: Deleting transient
> > attributes for node FILE-2 (via CIB call 77) |
> > xpath=//node_state[@uname='FILE-2']/transient_attributes
> > 65932:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (match_down_event) debug: Shutdown action 63 (stonith-FILE-
> > 2-reboot) found for node 2
> > 65933:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk_cpg_membership) info: Group crmd event 5: FILE-3 (node 3
> > pid 19250) is member
> > 65934:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk_cpg_membership) info: Group crmd event 5: FILE-4 (node 4
> > pid 19122) is member
> > 65935:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk_cpg_membership) info: Group crmd event 5: FILE-5 (node 5
> > pid 19273) is member
> > 65936:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk_cpg_membership) info: Group crmd event 5: FILE-6 (node 6
> > pid 19415) is member
> > 65938:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x00880000
> > (new_actions) for controller set by s_crmd_fsa:198
> > 65939:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_JOIN_RESULT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=route_message ]
> > 65940:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00800000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65941:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x00080000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65942:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (controld_delete_node_state) info: Deleting resource history
> > for node FILE-6 (via CIB call 79) |
> > xpath=//node_state[@uname='FILE-6']/lrm
> > 65943:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource stonith-sbd after monitor op complete
> > (interval=0)
> > 65945:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource FILE_Filesystem after monitor op complete
> > (interval=0)
> > 65946:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Service_pfile after monitor op complete
> > (interval=0)
> > 65947:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Service_Postgresql after monitor op complete
> > (interval=0)
> > 65948:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Service_esm_primary after monitor op complete
> > (interval=0)
> > 65949:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Service_Postgrest after monitor op complete
> > (interval=0)
> > 65950:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource IP_Floating after monitor op complete
> > (interval=0)
> > 65951:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__create_history_xml) debug: build_active_RAs:
> > Updating resource Shared_Cluster_Backup after monitor op complete
> > (interval=0)
> > 65952:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (do_dc_join_ack) debug: Updating local node history for join-1
> > from query result (via CIB call 80)
> > 65954:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
> > (fsa_data->actions) for controller set by s_crmd_fsa:193
> > 65955:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags
> > 0x1000000000000000 (new_actions) for controller set by
> > s_crmd_fsa:198
> > 65956:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
> > 65957:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags
> > 0x1000000000000000 (an_action) for controller cleared by
> > do_fsa_action:108
> > 65958:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (do_log) info: Input I_WAIT_FOR_EVENT received in state
> > S_FINALIZE_JOIN from do_te_invoke
> > 65959:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (do_log) debug: do_log <create_request_adv
> > origin="do_cl_join_query" t="crmd" version="3.11.0" subt="request"
> > reference="join_announce-crmd-1689603376-2"
> > crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd"
> > src="FILE-1"/>
> > 65960:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65961:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
> > source=do_te_invoke:135 complete=false
> > 65962:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (register_fsa_input_adv) debug: Stalling the FSA pending further
> > input: source=do_te_invoke cause=C_HA_MESSAGE data=0x55c619871630
> > queue=0
> > 65963:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Exiting the FSA: queue=1,
> > fsa_actions=0x0, stalled=true
> > 65964:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (fsa_dump_queue) debug: queue[0.80]: input I_WAIT_FOR_EVENT
> > raised by do_te_invoke(0x55c6198798d0.1) (cause=C_HA_MESSAGE)
> > 65966:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__execute_graph) debug: Transition 0 (Complete=34,
> > Pending=1, Fired=0, Skipped=0, Incomplete=24,
> > Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): In progress
> > 65967:Jul 17 14:18:20.097 FILE-6 pacemaker-fenced [19411]
> > (process_remote_stonith_exec) debug: Finalizing action
> > 'reboot' targeting FILE-1 on behalf of
> > pacemaker-controld.19415 at FILE-6: OK | rc=0 id=446afc42
> > 65968:Jul 17 14:18:20.097 FILE-6 pacemaker-fenced [19411]
> > (remote_op_done) notice: Operation 'reboot' targeting FILE-1 by
> > FILE-5 for pacemaker-controld.19415 at FILE-6: OK | id=446afc42
> > 65970:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (tengine_stonith_callback) notice: Stonith operation
> > 4/62:0:0:232e6505-2e98-4a79-b6ce-5f26d9cba645: OK (0)
> > 65971:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (tengine_stonith_callback) info: Stonith operation 4 for
> > FILE-1 passed
> > 65972:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (pcmk__update_peer_expected) info: crmd_peer_down: Node FILE-
> > 1[1] - expected state is now down (was pending)
> > 65973:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (send_stonith_update) debug: Sending fencing update 81 for
> > FILE-1
> > 65974:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (controld_delete_node_state) info: Deleting all state for
> > node FILE-1 (via CIB call 82) | xpath=//node_state[@uname='FILE-
> > 1']/*
> > 65975:Jul 17 14:18:20.097 FILE-6 pacemaker-controld [19415]
> > (exec_alert_list) info: Sending fencing alert via pf-ha-alert to
> > (null)
> > 65979:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (tengine_stonith_notify) notice: Peer FILE-1 was terminated
> > (reboot) by FILE-5 on behalf of pacemaker-controld.19415: OK |
> > initiator=FILE-6 ref=446afc42-b46e-47af-9fac-0fa87c1c5e57
> > 65980:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (send_stonith_update) debug: Sending fencing update 83 for
> > FILE-1
> > 65982:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (controld_delete_node_state) info: Deleting all state for
> > node FILE-1 (via CIB call 84) | xpath=//node_state[@uname='FILE-
> > 1']/*
> > 65983:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (cib_delete_callback) debug: Deletion of transient attributes
> > for node FILE-2 (via CIB call 77) succeeded
> > 65984:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (pcmk__execute_graph) notice: Transition 0 (Complete=35,
> > Pending=0, Fired=0, Skipped=3, Incomplete=24,
> > Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
> > 65985:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (te_graph_trigger) debug: Transition 0 is now complete
> > 65986:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (notify_crmd) debug: Processing transition completion in state
> > S_FINALIZE_JOIN
> > 65987:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (notify_crmd) debug: Transition 0 status: restart - Node join
> > 65988:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x1000000000000
> > (fsa_data->actions) for controller set by s_crmd_fsa:193
> > 65989:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags
> > 0x1000000000000000 (new_actions) for controller set by
> > s_crmd_fsa:198
> > 65990:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_WAIT_FOR_EVENT: [
> > state=S_FINALIZE_JOIN cause=C_HA_MESSAGE origin=do_te_invoke ]
> > 65991:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags
> > 0x1000000000000000 (an_action) for controller cleared by
> > do_fsa_action:108
> > 65992:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (do_log) info: Input I_WAIT_FOR_EVENT received in state
> > S_FINALIZE_JOIN from do_te_invoke
> > 65993:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (do_log) debug: do_log <create_request_adv
> > origin="do_cl_join_query" t="crmd" version="3.11.0" subt="request"
> > reference="join_announce-crmd-1689603376-2"
> > crm_task="join_announce" crm_sys_to="dc" crm_sys_from="crmd"
> > src="FILE-1"/>
> > 65994:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (pcmk__clear_flags_as) debug: FSA action flags 0x1000000000000
> > (an_action) for controller cleared by do_fsa_action:108
> > 65995:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (abort_transition_graph) info: Transition 0 aborted: Peer Halt |
> > source=do_te_invoke:135 complete=true
> > 65996:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (s_crmd_fsa) debug: Processing I_PE_CALC: [
> > state=S_FINALIZE_JOIN cause=C_FSA_INTERNAL
> > origin=abort_transition_graph ]
> > 66024:Jul 17 14:18:20.101 FILE-6 pacemaker-controld [19415]
> > (cib_delete_callback) debug: Deletion of resource history for
> > node FILE-6 (via CIB call 79) succeeded
> > 66063:Jul 17 14:18:20.105 FILE-6 pacemaker-controld [19415]
> > (join_update_complete_callback) debug: join-1 node history
> > update (via CIB call 80) complete
> > 66064:Jul 17 14:18:20.105 FILE-6 pacemaker-controld [19415]
> > (check_join_state) debug: join-1: Complete |
> > state=S_FINALIZE_JOIN for=join_update_complete_callback
> > 66068:Jul 17 14:18:20.105 FILE-6 pacemaker-controld [19415]
> > (pcmk__set_flags_as) debug: FSA action flags 0x800400000000
> > (new_actions) for controller set by s_crmd_fsa:198
> >
> > Thanks
> > Priyanka
> >
> > On Thu, Jul 20, 2023 at 11:53 AM Reid Wahl <nwahl at redhat.com>
> > wrote:
> > > On Wed, Jul 19, 2023 at 8:33 PM Priyanka Balotra
> > > <priyanka.14balotra at gmail.com> wrote:
> > > >
> > > > Sure,
> > > > Here are the logs:
> > > >
> > > >
> > > > 63138:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (post_cache_update) debug: Updated cache after membership
> > > event 44.
> > > > 63139:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x200000000
> > > (A_ELECTION_CHECK) for controller set by post_cache_update:81
> > > > 63140:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000002
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63141:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (do_started) info: Delaying start, Config not read
> > > (0000000000000040)
> > > > 63142:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (register_fsa_input_adv) debug: Stalling the FSA pending
> > > further input: source=do_started cause=C_FSA_INTERNAL data=(nil)
> > > queue=0
> > > > 63143:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x00000002
> > > (with_actions) for controller set by register_fsa_input_adv:88
> > > > 63144:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (s_crmd_fsa) debug: Exiting the FSA: queue=0,
> > > fsa_actions=0x200000002, stalled=true
> > > > 63145:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (config_query_callback) debug: Call 3 : Parsing CIB options
> > > > 63146:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (config_query_callback) debug: Shutdown escalation occurs if
> > > DC has not responded to request in 1200000ms
> > > > 63147:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (config_query_callback) debug: Re-run scheduler after 900000ms
> > > of inactivity
> > > > 63148:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pe_unpack_alerts) debug: Alert pf-ha-alert:
> > > path=/usr/lib/ocf/resource.d/pacemaker/pf_ha_alert.sh
> > > timeout=30000ms tstamp-format='%H:%M:%S.%06N' 0 vars
> > > > 63149:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000002
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63150:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (do_started) debug: Init server comms
> > > > 63151:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcs_us_publish) info: server name: crmd
> > > > 63152:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (do_started) notice: Pacemaker controller successfully
> > > started and accepting connections
> > > > 63153:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x200000000
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63154:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (do_election_check) debug: Ignoring election check because
> > > we are not in an election
> > > > 63155:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags
> > > 0x1000000000100100 (new_actions) for controller set by
> > > s_crmd_fsa:198
> > > > 63156:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (s_crmd_fsa) debug: Processing I_PENDING: [
> > > state=S_STARTING cause=C_FSA_INTERNAL origin=do_started ]
> > > > 63157:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags
> > > 0x1000000000000000 (an_action) for controller cleared by
> > > do_fsa_action:108
> > > > 63158:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (do_log) info: Input I_PENDING received in state S_STARTING
> > > from do_started
> > > > 63159:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (do_state_transition) notice: State transition S_STARTING ->
> > > S_PENDING | input=I_PENDING cause=C_FSA_INTERNAL
> > > origin=do_started
> > > > 63160:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x00000020
> > > (A_INTEGRATE_TIMER_STOP) for controller set by
> > > do_state_transition:559
> > > > 63161:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x00000080
> > > (A_FINALIZE_TIMER_STOP) for controller set by
> > > do_state_transition:565
> > > > 63162:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000020
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63163:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000080
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63164:Jul 17 14:16:25.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00100000
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63165:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (do_cl_join_query) debug: Querying for a DC
> > > > 63166:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000100
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63167:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (controld_start_timer) debug: Started Election Trigger
> > > (inject I_DC_TIMEOUT if pops after 20000ms, source=18)
> > > > 63168:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (stonith_api_signon) debug: Attempting fencer connection by
> > > pacemaker-controld with mainloop
> > > > 63175:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb-
> > > >word_size:33792
> > > > 63176:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb-
> > > >word_size:33792
> > > > 63177:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:131085; real_size:135168; rb-
> > > >word_size:33792
> > > > 63178:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processing register 8 from client
> > > pacemaker-controld.15962 with call options 0x00000000
> > > > 63179:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processed register from client
> > > pacemaker-controld.15962: OK (rc=0)
> > > > 63180:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (stonith_api_signon) debug: Connection to fencer by
> > > pacemaker-controld succeeded (registration token: 5552b1b4-f725-
> > > 46ac-b239-e404cadd8d94)
> > > > 63181:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processing st_notify 9 from client
> > > pacemaker-controld.15962 with call options 0x00000000
> > > > 63182:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (handle_request) debug: Enabling st_notify_disconnect callbacks
> > > for client pacemaker-controld.15962
> > > > 63183:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processed st_notify from client
> > > pacemaker-controld.15962: OK (rc=0)
> > > > 63184:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processing st_notify 10 from client
> > > pacemaker-controld.15962 with call options 0x00000000
> > > > 63185:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (handle_request) debug: Enabling st_notify_fence callbacks for
> > > client pacemaker-controld.15962
> > > > 63186:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processed st_notify from client
> > > pacemaker-controld.15962: OK (rc=0)
> > > > 63187:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processing st_notify 11 from client
> > > pacemaker-controld.15962 with call options 0x00000000
> > > > 63188:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (handle_request) debug: Enabling st_notify_history_synced
> > > callbacks for client pacemaker-controld.15962
> > > > 63189:Jul 17 14:16:26.132 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processed st_notify from client
> > > pacemaker-controld.15962: OK (rc=0)
> > > > 63190:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (te_trigger_stonith_history_sync) info: Fence history will be
> > > synchronized cluster-wide within 30 seconds
> > > > 63191:Jul 17 14:16:26.132 FILE-2 pacemaker-controld [15962]
> > > (te_connect_stonith) notice: Fencer successfully connected
> > > > 63192:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) info: Quorum retained | membership=48
> > > members=5
> > > > 63193:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) debug: Member[0] 2
> > > > 63194:Jul 17 14:16:32.664 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) debug: Member[1] 4
> > > > 63195:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63196:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63197:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63198:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63199:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-e4qK7U/qb-request-cmap-header
> > > > 63200:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-e4qK7U/qb-response-cmap-header
> > > > 63201:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-e4qK7U/qb-event-cmap-header
> > > > 63202:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 4
> > > > 63203:Jul 17 14:16:32.668 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 4
> > > > 63204:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63205:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63206:Jul 17 14:16:32.672 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63209:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63210:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-YYxILU/qb-request-cmap-header
> > > > 63211:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-YYxILU/qb-response-cmap-header
> > > > 63212:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-YYxILU/qb-event-cmap-header
> > > > 63213:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 4
> > > > 63214:Jul 17 14:16:32.676 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) info: Obtaining name for new node 4
> > > > 63218:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63222:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63225:Jul 17 14:16:32.684 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63240:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63241:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-Cy8QVV/qb-request-cmap-header
> > > > 63242:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-Cy8QVV/qb-response-cmap-header
> > > > 63243:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-Cy8QVV/qb-event-cmap-header
> > > > 63244:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 4
> > > > 63245:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 4
> > > > 63246:Jul 17 14:16:32.688 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) debug: Member[2] 3
> > > > 63259:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63265:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63267:Jul 17 14:16:32.700 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63298:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63299:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-34-0DHKhX/qb-request-cmap-header
> > > > 63300:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-34-0DHKhX/qb-response-cmap-header
> > > > 63301:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-34-0DHKhX/qb-event-cmap-header
> > > > 63302:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 3
> > > > 63303:Jul 17 14:16:32.712 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 3
> > > > 63307:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63313:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63320:Jul 17 14:16:32.720 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63351:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63352:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-34-V0bQlV/qb-request-cmap-header
> > > > 63353:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-34-V0bQlV/qb-response-cmap-header
> > > > 63355:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-34-V0bQlV/qb-event-cmap-header
> > > > 63356:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 3
> > > > 63357:Jul 17 14:16:32.728 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) info: Obtaining name for new node 3
> > > > 63365:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63372:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63374:Jul 17 14:16:32.736 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63415:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63416:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-34-EAFzTX/qb-request-cmap-header
> > > > 63417:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-34-EAFzTX/qb-response-cmap-header
> > > > 63418:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-34-EAFzTX/qb-event-cmap-header
> > > > 63419:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 3
> > > > 63420:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 3
> > > > 63421:Jul 17 14:16:32.748 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) debug: Member[3] 6
> > > > 63425:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63426:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63427:Jul 17 14:16:32.752 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63479:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63480:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-33-q3mFYU/qb-request-cmap-header
> > > > 63481:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-33-q3mFYU/qb-response-cmap-header
> > > > 63482:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-33-q3mFYU/qb-event-cmap-header
> > > > 63483:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 6
> > > > 63484:Jul 17 14:16:32.756 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 6
> > > > 63485:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63486:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63487:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63490:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63491:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-EcEbfV/qb-request-cmap-header
> > > > 63492:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-EcEbfV/qb-response-cmap-header
> > > > 63493:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-EcEbfV/qb-event-cmap-header
> > > > 63494:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 6
> > > > 63495:Jul 17 14:16:32.760 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) info: Obtaining name for new node 6
> > > > 63499:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63502:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63505:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63508:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63509:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-fLk4xW/qb-request-cmap-header
> > > > 63510:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-fLk4xW/qb-response-cmap-header
> > > > 63511:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-fLk4xW/qb-event-cmap-header
> > > > 63512:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 6
> > > > 63513:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 6
> > > > 63514:Jul 17 14:16:32.764 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) debug: Member[4] 5
> > > > 63517:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63518:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63521:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63528:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63529:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-ushXmW/qb-request-cmap-header
> > > > 63530:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-ushXmW/qb-response-cmap-header
> > > > 63531:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-ushXmW/qb-event-cmap-header
> > > > 63532:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 5
> > > > 63533:Jul 17 14:16:32.768 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 5
> > > > 63534:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63535:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63536:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63537:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63538:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-x3qVkW/qb-request-cmap-header
> > > > 63539:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-x3qVkW/qb-response-cmap-header
> > > > 63540:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-x3qVkW/qb-event-cmap-header
> > > > 63541:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 5
> > > > 63542:Jul 17 14:16:32.772 FILE-2 pacemaker-controld [15962]
> > > (quorum_notification_cb) info: Obtaining name for new node 5
> > > > 63543:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63544:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63545:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63546:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63547:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-gUNSFU/qb-request-cmap-header
> > > > 63548:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-gUNSFU/qb-response-cmap-header
> > > > 63549:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-gUNSFU/qb-event-cmap-header
> > > > 63550:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 5
> > > > 63551:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 5
> > > > 63552:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (update_peer_state_iter) notice: Node (null) state is now lost
> > > | nodeid=1 previous=member source=pcmk__reap_unseen_nodes
> > > > 63553:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (post_cache_update) debug: Updated cache after membership
> > > event 48.
> > > > 63554:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x200000000
> > > (A_ELECTION_CHECK) for controller set by post_cache_update:81
> > > > 63555:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x200000000
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63556:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (do_election_check) debug: Ignoring election check because
> > > we are not in an election
> > > > 63557:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (pcmk_cpg_membership) info: Group crmd event 0: node 2 pid
> > > 15962 joined via cpg_join
> > > > 63558:Jul 17 14:16:32.776 FILE-2 pacemaker-controld [15962]
> > > (pcmk_cpg_membership) info: Group crmd event 0: FILE-2 (node
> > > 2 pid 15962) is member
> > > > 63559:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63560:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63561:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63564:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63565:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-5PH1gV/qb-request-cmap-header
> > > > 63566:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-5PH1gV/qb-response-cmap-header
> > > > 63567:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-5PH1gV/qb-event-cmap-header
> > > > 63568:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 3
> > > > 63569:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 3
> > > > 63570:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (pcmk_cpg_membership) info: Group crmd event 0: peer node
> > > (node 3 pid 19250) is member
> > > > 63571:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (crm_update_peer_proc) info: pcmk_cpg_membership: Node
> > > (null)[3] - corosync-cpg is now online
> > > > 63572:Jul 17 14:16:32.780 FILE-2 pacemaker-controld [15962]
> > > (peer_update_callback) debug: Sending hello to node 3 so that
> > > it learns our node name
> > > > 63573:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63574:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63575:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63576:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63577:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-QATDEV/qb-request-cmap-header
> > > > 63578:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-QATDEV/qb-response-cmap-header
> > > > 63579:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-QATDEV/qb-event-cmap-header
> > > > 63580:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 4
> > > > 63581:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 4
> > > > 63582:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (pcmk_cpg_membership) info: Group crmd event 0: peer node
> > > (node 4 pid 19122) is member
> > > > 63583:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (crm_update_peer_proc) info: pcmk_cpg_membership: Node
> > > (null)[4] - corosync-cpg is now online
> > > > 63584:Jul 17 14:16:32.784 FILE-2 pacemaker-controld [15962]
> > > (peer_update_callback) debug: Sending hello to node 4 so that
> > > it learns our node name
> > > > 63585:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63586:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63587:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63588:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63589:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-TVzR1T/qb-request-cmap-header
> > > > 63590:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-TVzR1T/qb-response-cmap-header
> > > > 63591:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-TVzR1T/qb-event-cmap-header
> > > > 63592:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 5
> > > > 63593:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 5
> > > > 63594:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (pcmk_cpg_membership) info: Group crmd event 0: peer node
> > > (node 5 pid 19273) is member
> > > > 63595:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (crm_update_peer_proc) info: pcmk_cpg_membership: Node
> > > (null)[5] - corosync-cpg is now online
> > > > 63596:Jul 17 14:16:32.788 FILE-2 pacemaker-controld [15962]
> > > (peer_update_callback) debug: Sending hello to node 5 so that
> > > it learns our node name
> > > > 63597:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63598:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63599:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_open_2) debug: shm size:1048589; real_size:1052672;
> > > rb->word_size:263168
> > > > 63600:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (qb_ipcc_disconnect) debug: qb_ipcc_disconnect()
> > > > 63601:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-8LRaoV/qb-request-cmap-header
> > > > 63602:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-8LRaoV/qb-response-cmap-header
> > > > 63603:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (qb_rb_close_helper) debug: Closing ringbuffer:
> > > /dev/shm/qb-13142-15962-31-8LRaoV/qb-event-cmap-header
> > > > 63604:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (pcmk__corosync_name) info: Unable to get node name for
> > > nodeid 6
> > > > 63605:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (get_node_name) notice: Could not obtain a node name for
> > > corosync node with id 6
> > > > 63606:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (pcmk_cpg_membership) info: Group crmd event 0: peer node
> > > (node 6 pid 19415) is member
> > > > 63607:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (crm_update_peer_proc) info: pcmk_cpg_membership: Node
> > > (null)[6] - corosync-cpg is now online
> > > > 63608:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (peer_update_callback) debug: Sending hello to node 6 so that
> > > it learns our node name
> > > > 63609:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (get_xpath_object) debug: No match for
> > > //st_notify_history_synced in /notify
> > > > 63610:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (stonith_api_del_notification) debug: Removing callback for
> > > st_notify_history_synced events
> > > > 63611:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processing st_notify 12 from client
> > > pacemaker-controld.15962 with call options 0x00000000
> > > > 63612:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958]
> > > (handle_request) debug: Disabling st_notify_history_synced
> > > callbacks for client pacemaker-controld.15962
> > > > 63613:Jul 17 14:16:32.792 FILE-2 pacemaker-fenced [15958]
> > > (stonith_command) debug: Processed st_notify from client
> > > pacemaker-controld.15962: OK (rc=0)
> > > > 63614:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (tengine_stonith_history_synced) debug: Fence-history synced -
> > > cancel all timers
> > > > 63615:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (crm_get_peer) info: Node 4 is now known as FILE-4
> > > > 63616:Jul 17 14:16:32.792 FILE-2 pacemaker-controld [15962]
> > > (update_peer_uname) warning: Node names with capitals are
> > > discouraged, consider changing 'FILE-4'
> > > > 63617:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
> > > (peer_update_callback) info: Cluster node FILE-4 is now
> > > member
> > > > 63618:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
> > > (crm_get_peer) info: Node 3 is now known as FILE-3
> > > > 63619:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
> > > (update_peer_uname) warning: Node names with capitals are
> > > discouraged, consider changing 'FILE-3'
> > > > 63620:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
> > > (peer_update_callback) info: Cluster node FILE-3 is now
> > > member
> > > > 63621:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
> > > (crm_get_peer) info: Node 5 is now known as FILE-5
> > > > 63622:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
> > > (update_peer_uname) warning: Node names with capitals are
> > > discouraged, consider changing 'FILE-5'
> > > > 63623:Jul 17 14:16:32.796 FILE-2 pacemaker-controld [15962]
> > > (peer_update_callback) info: Cluster node FILE-5 is now
> > > member
> > > > 63640:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (crm_get_peer) info: Node 6 is now known as FILE-6
> > > > 63641:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (update_peer_uname) warning: Node names with capitals are
> > > discouraged, consider changing 'FILE-6'
> > > > 63642:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (peer_update_callback) info: Cluster node FILE-6 is now
> > > member
> > > > 63643:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (handle_request) debug: Raising I_JOIN_OFFER: join-1
> > > > 63644:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x00400200
> > > (new_actions) for controller set by s_crmd_fsa:198
> > > > 63645:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (s_crmd_fsa) debug: Processing I_JOIN_OFFER: [
> > > state=S_PENDING cause=C_HA_MESSAGE origin=route_message ]
> > > > 63646:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000200
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63647:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00400000
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63648:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (update_dc) info: Set DC to FILE-6 (3.11.0)
> > > > 63649:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (pcmk__update_peer_expected) info: update_dc: Node FILE-
> > > 6[6] - expected state is now member (was (null))
> > > > 63650:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x00000200
> > > (A_DC_TIMER_STOP) for controller set by
> > > do_cl_join_offer_respond:147
> > > > 63651:Jul 17 14:16:32.880 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000200
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63788:Jul 17 14:16:32.884 FILE-2 pacemaker-controld [15962]
> > > (do_cib_replaced) debug: Updating the CIB after a replace:
> > > DC=false
> > > > 63811:Jul 17 14:16:32.892 FILE-2 pacemaker-controld [15962]
> > > (join_query_callback) debug: Respond to join offer join-1
> > > from FILE-6
> > > > 63819:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
> > > (pcmk__procfs_pid_of) info: Found pacemaker-based active as
> > > process 15957
> > > > 63820:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
> > > (throttle_cib_load) debug: Init 6 + 2 ticks at 1689603415
> > > (100 tps)
> > > > 63821:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
> > > (throttle_mode) debug: Current load is 0.980000 across 10
> > > core(s)
> > > > 63822:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
> > > (throttle_send_command) info: New throttle mode: negligible
> > > load (was undetermined)
> > > > 63823:Jul 17 14:16:55.080 FILE-2 pacemaker-controld [15962]
> > > (throttle_update) debug: Node FILE-2 has negligible load and
> > > supports at most 20 jobs; new job limit 20
> > > > 63824:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (handle_request) debug: Raising I_JOIN_RESULT: join-1
> > > > 63825:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x00800000
> > > (new_actions) for controller set by s_crmd_fsa:198
> > > > 63826:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (s_crmd_fsa) debug: Processing I_JOIN_RESULT: [
> > > state=S_PENDING cause=C_HA_MESSAGE origin=route_message ]
> > > > 63827:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00800000
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63828:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (do_cl_join_finalize_respond) debug: Confirming join-1:
> > > sending local operation history to FILE-6
> > > > 63829:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags
> > > 0x1000000000000200 (new_actions) for controller set by
> > > s_crmd_fsa:198
> > > > 63830:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (s_crmd_fsa) debug: Processing I_NOT_DC: [ state=S_PENDING
> > > cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
> > > > 63831:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags
> > > 0x1000000000000000 (an_action) for controller cleared by
> > > do_fsa_action:108
> > > > 63832:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (do_log) info: Input I_NOT_DC received in state S_PENDING from
> > > do_cl_join_finalize_respond
> > > > 63833:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (do_state_transition) notice: State transition S_PENDING ->
> > > S_NOT_DC | input=I_NOT_DC cause=C_HA_MESSAGE
> > > origin=do_cl_join_finalize_respond
> > > > 63834:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x00000020
> > > (A_INTEGRATE_TIMER_STOP) for controller set by
> > > do_state_transition:559
> > > > 63835:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (pcmk__set_flags_as) debug: FSA action flags 0x00000080
> > > (A_FINALIZE_TIMER_STOP) for controller set by
> > > do_state_transition:565
> > > > 63836:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000200
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63837:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000020
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63838:Jul 17 14:16:55.092 FILE-2 pacemaker-controld [15962]
> > > (pcmk__clear_flags_as) debug: FSA action flags 0x00000080
> > > (an_action) for controller cleared by do_fsa_action:108
> > > > 63863:Jul 17 14:17:25.073 FILE-2 pacemaker-controld [15962]
> > > (throttle_cib_load) debug: cib load: 0.000667 (2 ticks in
> > > 30s)
> > > > 63864:Jul 17 14:17:25.073 FILE-2 pacemaker-controld [15962]
> > > (throttle_mode) debug: Current load is 0.650000 across 10
> > > core(s)
> > > > 63865:Jul 17 14:17:55.073 FILE-2 pacemaker-controld [15962]
> > > (throttle_cib_load) debug: cib load: 0.000333 (1 ticks in
> > > 30s)
> > > > 63866:Jul 17 14:17:55.073 FILE-2 pacemaker-controld [15962]
> > > (throttle_mode) debug: Current load is 0.850000 across 10
> > > core(s)
> > > > 63868:Jul 17 14:18:20.085 FILE-2 pacemaker-fenced [15958]
> > > (process_remote_stonith_exec) debug: Finalizing action
> > > 'reboot' targeting FILE-2 on behalf of
> > > pacemaker-controld.19415 at FILE-6: OK | rc=0 id=4e523b34
> > > > 63869:Jul 17 14:18:20.085 FILE-2 pacemaker-fenced [15958]
> > > (remote_op_done) notice: Operation 'reboot' targeting FILE-2 by
> > > FILE-4 for pacemaker-controld.19415 at FILE-6: OK | id=4e523b34
> > > > 63872:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
> > > (exec_alert_list) info: Sending fencing alert via pf-ha-alert to
> > > (null)
> > > > 63875:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
> > > (tengine_stonith_notify) crit: We were allegedly just fenced by
> > > FILE-4 for FILE-6!
> > > > 63876:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
> > > (crm_xml_cleanup) info: Cleaning up memory from libxml2
> > > > 63877:Jul 17 14:18:20.089 FILE-2 pacemaker-controld [15962]
> > > (crm_exit) info: Exiting pacemaker-controld | with status
> > > 100
> > > > 63900:Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > > (pcmk_child_exit) warning: Shutting cluster down because
> > > pacemaker-controld[15962] had fatal failure
> > > > 63902:Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > > (pcmk_shutdown_worker) debug: pacemaker-controld confirmed
> > > stopped
> > > > 63956:Jul 17 14:18:20.101 FILE-2 pacemaker-fenced [15958]
> > > (process_remote_stonith_exec) debug: Finalizing action
> > > 'reboot' targeting FILE-1 on behalf of
> > > pacemaker-controld.19415 at FILE-6: OK | rc=0 id=446afc42
> > > > 63957:Jul 17 14:18:20.101 FILE-2 pacemaker-fenced [15958]
> > > (remote_op_done) notice: Operation 'reboot' targeting FILE-1 by
> > > FILE-5 for pacemaker-controld.19415 at FILE-6: OK | id=446afc42>
> > > > Thanks
> > > > Priyanka
> > >
> > > Hi, node FILE-6 requested that node FILE-2 be fenced by node
> > > FILE-4.
> > > FILE-2's controller daemon received notification that it was
> > > being
> > > fenced, and it shut down. You'd want to check the logs on FILE-6
> > > to
> > > determine why FILE-2 was fenced.
> > >
> > > >
> > > > On Thu, Jul 20, 2023 at 12:07 AM Ken Gaillot <
> > > kgaillot at redhat.com> wrote:
> > > >>
> > > >> On Wed, 2023-07-19 at 23:49 +0530, Priyanka Balotra wrote:
> > > >> > Hi All,
> > > >> > I am using SLES 15 SP4. One of the nodes of the cluster is
> > > brought
> > > >> > down and boot up after sometime. Pacemaker service came up
> > > first but
> > > >> > later it faced a fatal shutdown. Due to that crm service is
> > > down.
> > > >> >
> > > >> > The logs from /var/log/pacemaker.pacemaker.log are as
> > > follows:
> > > >> >
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > > >> > (pcmk_child_exit) warning: Shutting cluster down
> > > because
> > > >> > pacemaker-controld[15962] had fatal failure
> > > >>
> > > >> The interesting messages will be before this. The ones with
> > > "pacemaker-
> > > >> controld" will be the most relevant, at least initially.
> > > >>
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > > >> > (pcmk_shutdown_worker) notice: Shutting down Pacemaker
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > > >> > (pcmk_shutdown_worker) debug: pacemaker-controld confirmed
> > > stopped
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > > (stop_child)
> > > >> > notice: Stopping pacemaker-schedulerd | sent signal 15 to
> > > process
> > > >> > 15961
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> > > >> > (crm_signal_dispatch) notice: Caught 'Terminated' signal
> > > | 15
> > > >> > (invoking handler)
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> > > >> > (qb_ipcs_us_withdraw) info: withdrawing server sockets
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> > > >> > (qb_ipcs_unref) debug: qb_ipcs_unref() - destroying
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> > > >> > (crm_xml_cleanup) info: Cleaning up memory from
> > > libxml2
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-schedulerd[15961]
> > > (crm_exit)
> > > >> > info: Exiting pacemaker-schedulerd | with status 0
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
> > > >> > (qb_ipcs_event_sendv) debug: new_event_notification
> > > (/dev/shm/qb-
> > > >> > 15957-15962-12-RDPw6O/qb): Broken pipe (32)
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
> > > >> > (cib_notify_send_one) warning: Could not notify client
> > > crmd:
> > > >> > Broken pipe | id=e29d175e-7e91-4b6a-bffb-fabfdd7a33bf
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
> > > >> > (cib_process_request) info: Completed cib_delete
> > > operation for
> > > >> > section //node_state[@uname='FILE-2']/*: OK (rc=0,
> > > origin=FILE-
> > > >> > 6/crmd/74, version=0.24.75)
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-fenced [15958]
> > > >> > (xml_patch_version_check) debug: Can apply patch
> > > 0.24.75 to
> > > >> > 0.24.74
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > > >> > (pcmk_child_exit) info: pacemaker-schedulerd[15961]
> > > exited
> > > >> > with status 0 (OK)
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-based [15957]
> > > >> > (cib_process_request) info: Completed cib_modify
> > > operation for
> > > >> > section status: OK (rc=0, origin=FILE-6/crmd/75,
> > > version=0.24.75)
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > > >> > (pcmk_shutdown_worker) debug: pacemaker-schedulerd
> > > confirmed
> > > >> > stopped
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemakerd [15956]
> > > (stop_child)
> > > >> > notice: Stopping pacemaker-attrd | sent signal 15 to
> > > process 15960
> > > >> > Jul 17 14:18:20.093 FILE-2 pacemaker-attrd [15960]
> > > >> > (crm_signal_dispatch) notice: Caught 'Terminated' signal
> > > | 15
> > > >> > (invoking handler)
> > > >> >
> > > >> > Could you please help me understand the issue here.
> > > >> >
> > > >> > Regards
> > > >> > Priyanka
> > > >> > _______________________________________________
> > > >> > Manage your subscription:
> > > >> > https://lists.clusterlabs.org/mailman/listinfo/users
> > > >> >
> > > >> > ClusterLabs home: https://www.clusterlabs.org/
> > > >> --
> > > >> Ken Gaillot <kgaillot at redhat.com>
> > > >>
> > > >> _______________________________________________
> > > >> Manage your subscription:
> > > >> https://lists.clusterlabs.org/mailman/listinfo/users
> > > >>
> > > >> ClusterLabs home: https://www.clusterlabs.org/
> > > >
> > > > _______________________________________________
> > > > Manage your subscription:
> > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > >
> > > > ClusterLabs home: https://www.clusterlabs.org/
> > >
> > >
> > >
> > > _______________________________________________
> > > Manage your subscription:
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > >
> > > ClusterLabs home: https://www.clusterlabs.org/
--
Ken Gaillot <kgaillot at redhat.com>
More information about the Users
mailing list