Edition 13
Abstract
Table of Contents
List of Figures
List of Tables
lrm_rsc_op jobList of Examples
OCF_CHECK_LEVEL.resource_setresource-stickiness during working hoursreloadreload Operationapcstonith resourceapcstonith resource)pingd clone with multiple jobsticketA is revokedrsc1 if ticketA is revokedTable of Contents
Mono-spaced Bold
To see the contents of the filemy_next_bestselling_novelin your current working directory, enter thecat my_next_bestselling_novelcommand at the shell prompt and press Enter to execute the command.
Press Enter to execute the command.Press Ctrl+Alt+F2 to switch to a virtual terminal.
mono-spaced bold. For example:
File-related classes includefilesystemfor file systems,filefor files, anddirfor directories. Each class has its own associated set of permissions.
Choose → → from the main menu bar to launch Mouse Preferences. In the Buttons tab, select the Left-handed mouse check box and click to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand).To insert a special character into a gedit file, choose → → from the main menu bar. Next, choose → from the Character Map menu bar, type the name of the character in the Search field and click . The character you sought will be highlighted in the Character Table. Double-click this highlighted character to place it in the Text to copy field and then click the button. Now switch back to your document and choose → from the gedit menu bar.
Mono-spaced Bold Italic or Proportional Bold Italic
To connect to a remote machine using ssh, typessh username@domain.nameat a shell prompt. If the remote machine isexample.comand your username on that machine is john, typessh john@example.com.Themount -o remount file-systemcommand remounts the named file system. For example, to remount the/homefile system, the command ismount -o remount /home.To see the version of a currently installed package, use therpm -q packagecommand. It will return a result as follows:package-version-release.
Publican is a DocBook publishing system.
mono-spaced roman and presented thus:
books Desktop documentation drafts mss photos stuff svn books_tests Desktop1 downloads images notes scripts svgs
mono-spaced roman but add syntax highlighting as follows:
package org.jboss.book.jca.ex1;
import javax.naming.InitialContext;
public class ExClient
{
public static void main(String args[])
throws Exception
{
InitialContext iniCtx = new InitialContext();
Object ref = iniCtx.lookup("EchoBean");
EchoHome home = (EchoHome) ref;
Echo echo = home.create();
System.out.println("Created Echo");
System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));
}
}Note
Important
Warning
Table of Contents
Fencing
Old daemon names
| Old name | New name |
|---|---|
|
attrd
|
pacemaker-attrd
|
|
cib
|
pacemaker-based
|
|
crmd
|
pacemaker-controld
|
|
lrmd
|
pacemaker-execd
|
|
stonithd
|
pacemaker-fenced
|
|
pacemaker_remoted
|
pacemaker-remoted
|
Table of Contents
Example 2.1. An empty configuration
<cib crm_feature_set="3.0.7" validate-with="pacemaker-1.2" admin_epoch="1" epoch="0" num_updates="0"> <configuration> <crm_config/> <nodes/> <resources/> <constraints/> </configuration> <status/> </cib>
cib: The entire CIB is enclosed with a cib tag. Certain fundamental settings are defined as attributes of this tag.
configuration: This section — the primary focus of this document — contains traditional configuration information such as what resources the cluster serves and the relationships among them.
crm_config: cluster-wide configuration options
nodes: the machines that host the cluster
resources: the services run by the cluster
constraints: indications of how resources should be placed
status: This section contains the history of each resource on each node. Based on this data, the cluster can construct the complete current state of the cluster. The authoritative source for this section is the local executor (pacemaker-execd process) on each cluster node, and the cluster will occasionally repopulate the entire section. For this reason, it is never written to disk, and administrators are advised against modifying it in any way.
nvpair child elements of an XML element.
cib tag) rather than with the rest of the cluster configuration in the configuration section.
Table 2.1. CIB Properties
crm_config section, and, in advanced configurations, there may be more than one set. (This will be described later in the section on Chapter 8, Rules where we will show how to have the cluster use different sets of options during working hours than during weekends.) For now, we will describe the simple case where each option is present at most once.
man pacemaker-schedulerd and man pacemaker-controld commands.
Table 2.2. Cluster Options
| Option | Default | Description |
|---|---|---|
cluster-name
|
|
An (optional) name for the cluster as a whole. This is mostly for users' convenience for use as desired in administration, but this can be used in the Pacemaker configuration in rules (as the
#cluster-name node attribute). It may also be used by higher-level tools when displaying cluster information, and by certain resource agents (for example, the ocf:heartbeat:GFS2 agent stores the cluster name in filesystem meta-data).
|
dc-version
|
| |
cluster-infrastructure
|
| |
no-quorum-policy
|
stop
|
|
batch-limit
|
0
| |
migration-limit
|
-1
|
The number of live migration actions that the cluster is allowed to execute in parallel on a node. A value of -1 means unlimited.
|
symmetric-cluster
|
TRUE
| |
stop-all-resources
|
FALSE
| |
stop-orphan-resources
|
TRUE
| |
stop-orphan-actions
|
TRUE
| |
start-failure-is-fatal
|
TRUE
|
Should a failure to start a resource on a particular node prevent further start attempts on that node? If FALSE, the cluster will decide whether the same node is still eligible based on the resource’s current failure count and
migration-threshold (see Section 9.2, “Handling Resource Failure”).
|
enable-startup-probes
|
TRUE
| |
maintenance-mode
|
FALSE
| |
stonith-enabled
|
TRUE
|
Should failed nodes and nodes with resources that can’t be stopped be shot? If you value your data, set up a STONITH device and enable this.
If true, or unset, the cluster will refuse to start resources unless one or more STONITH resources have been configured. If false, unresponsive nodes are immediately assumed to be running no resources, and resource takeover to online nodes starts without any further protection (which means data loss if the unresponsive node still accesses shared storage, for example). See also the
requires meta-attribute in Section 4.4, “Resource Options”.
|
stonith-action
|
reboot
| |
stonith-timeout
|
60s
| |
stonith-max-attempts
|
10
| |
stonith-watchdog-timeout
|
0
|
If nonzero, along with
have-watchdog=true automatically set by the cluster, when fencing is required, watchdog-based self-fencing will be performed via SBD without requiring a fencing resource explicitly configured. If stonith-watchdog-timeout is set to a positive value, unseen nodes are assumed to self-fence within this much time. WARNING: It must be ensured that this value is larger than the SBD_WATCHDOG_TIMEOUT environment variable on all nodes. Pacemaker verifies the settings individually on all nodes and prevents startup or shuts down if configured wrongly on the fly. It’s strongly recommended that SBD_WATCHDOG_TIMEOUT is set to the same value on all nodes. If stonith-watchdog-timeout is set to a negative value, and SBD_WATCHDOG_TIMEOUT is set, twice that value will be used. WARNING: In this case, it’s essential (currently not verified by pacemaker) that SBD_WATCHDOG_TIMEOUT is set to the same value on all nodes.
|
concurrent-fencing
|
FALSE
| |
fence-reaction
|
stop
|
How should a cluster node react if notified of its own fencing? A cluster node may receive notification of its own fencing if fencing is misconfigured, or if fabric fencing is in use that doesn’t cut cluster communication. Allowed values are
stop to attempt to immediately stop pacemaker and stay stopped, or panic to attempt to immediately reboot the local node, falling back to stop on failure. The default is likely to be changed to panic in a future release. (since 2.0.3)
|
priority-fencing-delay
|
0
|
Apply specified delay for the fencings that are targeting the lost nodes with the highest total resource priority in case we don’t have the majority of the nodes in our cluster partition, so that the more significant nodes potentially win any fencing match, which is especially meaningful under split-brain of 2-node cluster. A promoted resource instance takes the base priority + 1 on calculation if the base priority is not 0. Any static/random delays that are introduced by
pcmk_delay_base/max configured for the corresponding fencing resources will be added to this delay. This delay should be significantly greater than, safely twice, the maximum pcmk_delay_base/max. By default, priority fencing delay is disabled. (since 2.0.4)
|
cluster-delay
|
60s
|
Estimated maximum round-trip delay over the network (excluding action execution). If the DC requires an action to be executed on another node, it will consider the action failed if it does not get a response from the other node in this time (after considering the action’s own timeout). The "correct" value will depend on the speed and load of your network and cluster nodes.
|
dc-deadtime
|
20s
|
The "correct" value will depend on the speed/load of your network and the type of switches used.
|
cluster-ipc-limit
|
500
|
The maximum IPC message backlog before one cluster daemon will disconnect another. This is of use in large clusters, for which a good value is the number of resources in the cluster multiplied by the number of nodes. The default of 500 is also the minimum. Raise this if you see "Evicting client" messages for cluster daemon PIDs in the logs.
|
pe-error-series-max
|
-1
| |
pe-warn-series-max
|
-1
| |
pe-input-series-max
|
-1
| |
placement-strategy
|
default
|
How the cluster should allocate resources to nodes (see Chapter 12, Utilization and Placement Strategy). Allowed values are
default, utilization, balanced, and minimal.
|
node-health-strategy
|
none
|
How the cluster should react to node health attributes (see Section 9.4, “Tracking Node Health”). Allowed values are
none, migrate-on-red, only-green, progressive, and custom.
|
enable-acl
|
FALSE
|
Whether access control lists (ACLs) (see Chapter 13, ACLs) can be used to authorize modifications to the CIB.
|
node-health-base
|
0
| |
node-health-green
|
0
| |
node-health-yellow
|
0
| |
node-health-red
|
0
| |
cluster-recheck-interval
|
15min
|
Pacemaker is primarily event-driven, and looks ahead to know when to recheck the cluster for failure timeouts and most time-based rules. However, it will also recheck the cluster after this amount of inactivity. This has two goals: rules with
date_spec are only guaranteed to be checked this often, and it also serves as a fail-safe for certain classes of scheduler bugs. A value of 0 disables this polling; positive values are a time interval.
|
shutdown-lock
|
false
|
The default of false allows active resources to be recovered elsewhere when their node is cleanly shut down, which is what the vast majority of users will want. However, some users prefer to make resources highly available only for failures, with no recovery for clean shutdowns. If this option is true, resources active on a node when it is cleanly shut down are kept "locked" to that node (not allowed to run elsewhere) until they start again on that node after it rejoins (or for at most shutdown-lock-limit, if set). Stonith resources and Pacemaker Remote connections are never locked. Clone and bundle instances and the master role of promotable clones are currently never locked, though support could be added in a future release. Locks may be manually cleared using the
--refresh option of crm_resource (both the resource and node must be specified; this works with remote nodes if their connection resource’s target-role is set to Stopped, but not if Pacemaker Remote is stopped on the remote node without disabling the connection resource). (since 2.0.4)
|
shutdown-lock-limit
|
0
|
If shutdown-lock is true, and this is set to a nonzero time duration, locked resources will be allowed to start after this much time has passed since the node shutdown was initiated, even if the node has not rejoined. (This works with remote nodes only if their connection resource’s target-role is set to Stopped.) (since 2.0.4)
|
remove-after-stop
|
FALSE
| |
startup-fencing
|
TRUE
| |
election-timeout
|
2min
| |
shutdown-escalation
|
20min
| |
join-integration-timeout
|
3min
| |
join-finalization-timeout
|
30min
| |
transition-delay
|
0s
|
Table of Contents
uname -n. This can be problematic for services that require the uname -n to be a specific value (e.g. for a licence file).
corosync.conf under ring0_addr in the nodelist, if it does not contain an IP address; otherwise
corosync.conf under name in the nodelist; otherwise
uname -n
crm_node -n command which displays the name used by a running cluster.
crm_node --name-for-id number is also available to display the name used by the node with the corosync nodeid of number, for example: crm_node --name-for-id 2.
crm_attribute and attrd_updater commands, so that the user does not have to deal with XML configuration directly.
Example 3.2. Result of using crm_attribute to specify which kernel pcmk-1 is running
# crm_attribute --type nodes --node pcmk-1 --name kernel --update $(uname -r)
<node id="1" uname="pcmk-1"> <instance_attributes id="nodes-1-attributes"> <nvpair id="nodes-1-kernel" name="kernel" value="3.10.0-862.14.4.el7.x86_64"/> </instance_attributes> </node>
# crm_attribute --type nodes --node pcmk-1 --name kernel --query scope=nodes name=kernel value=3.10.0-862.14.4.el7.x86_64
--type nodes the admin tells the cluster that this attribute is persistent across reboots. There are also transient attributes which are kept in the status section and are "forgotten" whenever the node leaves the cluster. Administrators can use this section by specifying --type status.
Table 3.1. Node attributes with special significance
| Name | Description |
|---|---|
fail-count-*
| |
last-failure-*
| |
maintenance
|
Similar to the
maintenance-mode cluster option, but for a single node. If true, resources will not be started or stopped on the node, resources and individual clone instances running on the node will become unmanaged, and any recurring operations for those will be cancelled.
|
probe_complete
| |
resource-discovery-enabled
| |
shutdown
| |
site-name
| |
standby
| |
terminate
| |
#digests-*
|
Attributes whose names start with
#digests- are managed by the cluster to detect when unfencing needs to be redone, and should never be used directly.
|
#node-unfenced
|
Warning
maintenance is set as a transient attribute, it will be erased when pacemaker is stopped, which will immediately take the node out of maintenance mode and likely get it fenced. Even if permanent, if pacemaker is restarted, any resources active on the node will have their local history erased when the node rejoins, so the cluster will no longer consider them running on the node and thus will consider them managed again, leading them to be started elsewhere. This behavior might be improved in a future release.
Table of Contents
start, stop or monitor command. For this reason, it is crucial that resource agents are well-tested.
OCF_RESKEY_. So, a parameter which the user thinks of as ip will be passed to the resource agent as OCF_RESKEY_ip. The number and purpose of the parameters is left to the resource agent; however, the resource agent should use the meta-data command to advertise any that it supports.
/etc/init.d.
Warning
status operation at all
start/stop/status actions
Important
Important
Important
systemd, upstart, and lsb), Pacemaker supports a special service alias which intelligently figures out which one applies to a given cluster node.
systemd, upstart, and lsb.
crm_resource tool. For example:
# crm_resource --resource Email --query-xml
Note
Example 4.2. An OCF resource definition
<primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat"> <instance_attributes id="Public-IP-params"> <nvpair id="Public-IP-ip" name="ip" value="192.0.2.2"/> </instance_attributes> </primitive>
--meta option of the crm_resource command.
Table 4.2. Meta-attributes of a Primitive Resource
| Field | Default | Description |
|---|---|---|
priority
|
0
| |
target-role
|
Started
|
What state should the cluster attempt to keep this resource in? Allowed values:
|
is-managed
|
TRUE
| |
maintenance
|
FALSE
|
Similar to the
maintenance-mode cluster option, but for a single resource. If true, the resource will not be started, stopped, or monitored on any node. This differs from is-managed in that monitors will not be run. Allowed values: true, false
|
resource-stickiness
|
1 for individual clone instances, 0 for all other resources
| |
requires
|
quorum for resources with a class of stonith, otherwise unfencing if unfencing is active in the cluster, otherwise fencing if stonith-enabled is true, otherwise quorum
|
Conditions under which the resource can be started Allowed values:
|
migration-threshold
|
INFINITY
|
How many failures may occur for this resource on a node, before this node is marked ineligible to host this resource. A value of 0 indicates that this feature is disabled (the node will never be marked ineligible); by constrast, the cluster treats INFINITY (the default) as a very large but finite number. This option has an effect only if the failed operation specifies
on-fail as restart (the default), and additionally for failed start operations, if the cluster property start-failure-is-fatal is false.
|
failure-timeout
|
0
| |
multiple-active
|
stop_start
|
What should the cluster do if it ever finds the resource active on more than one node? Allowed values:
|
allow-migrate
|
TRUE for ocf:pacemaker:remote resources, FALSE otherwise
|
Whether the cluster should try to "live migrate" this resource when it needs to be moved (see Section 9.3.3, “Migrating Resources”)
|
container-attribute-target
|
|
Specific to bundle resources; see Section 10.3.7, “Bundle Node Attributes”
|
remote-node
|
|
The name of the Pacemaker Remote guest node this resource is associated with, if any. If specified, this both enables the resource as a guest node and defines the unique name used to identify the guest node. The guest must be configured to run the Pacemaker Remote daemon when it is started.
WARNING: This value cannot overlap with any resource or node IDs.
|
remote-port
|
3121
|
If
remote-node is specified, the port on the guest used for its Pacemaker Remote connection. The Pacemaker Remote daemon on the guest must be configured to listen on this port.
|
remote-addr
|
value of
remote-node
|
If
remote-node is specified, the IP address or hostname used to connect to the guest via Pacemaker Remote. The Pacemaker Remote daemon on the guest must be configured to accept connections on this address.
|
remote-connect-timeout
|
60s
|
If
remote-node is specified, how long before a pending guest connection will time out.
|
# crm_resource --meta --resource Email --set-parameter priority --parameter-value 100 # crm_resource -m -r Email -p multiple-active -v block
Example 4.3. An LSB resource with cluster options
<primitive id="Email" class="lsb" type="exim"> <meta_attributes id="Email-meta_attributes"> <nvpair id="Email-meta_attributes-priority" name="priority" value="100"/> <nvpair id="Email-meta_attributes-multiple-active" name="multiple-active" value="block"/> </meta_attributes> </primitive>
rsc_defaults section with crm_attribute. For example,
# crm_attribute --type rsc_defaults --name is-managed --update false
is-managed set to true).
crm_resource command. For example,
# crm_resource --resource Public-IP --set-parameter ip --parameter-value 192.0.2.2
Example 4.4. An example OCF resource with instance attributes
<primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat"> <instance_attributes id="params-public-ip"> <nvpair id="public-ip-addr" name="ip" value="192.0.2.2"/> </instance_attributes> </primitive>
OCF_RESKEY_ip with a value of 192.0.2.2.
meta-data command. The output contains an XML description of all the supported attributes, their purpose and default values.
Example 4.5. Displaying the metadata for the Dummy resource agent template
# export OCF_ROOT=/usr/lib/ocf # $OCF_ROOT/resource.d/pacemaker/Dummy meta-data
<?xml version="1.0"?> <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd"> <resource-agent name="Dummy" version="1.0"> <version>1.0</version> <longdesc> This is a Dummy Resource Agent. It does absolutely nothing except keep track of whether its running or not. Its purpose in life is for testing and to serve as a template for RA writers. NB: Please pay attention to the timeouts specified in the actions section below. They should be meaningful for the kind of resource the agent manages. They should be the minimum advised timeouts, but they shouldn't/cannot cover _all_ possible resource instances. So, try to be neither overly generous nor too stingy, but moderate. The minimum timeouts should never be below 10 seconds. </longdesc> <shortdesc>Example stateless resource agent</shortdesc> <parameters> <parameter name="state" unique="1"> <longdesc> Location to store the resource state in. </longdesc> <shortdesc>State file</shortdesc> <content type="string" default="/var/run/Dummy-default.state" /> </parameter> <parameter name="fake" unique="0"> <longdesc> Fake attribute that can be changed to cause a reload </longdesc> <shortdesc>Fake attribute that can be changed to cause a reload</shortdesc> <content type="string" default="dummy" /> </parameter> <parameter name="op_sleep" unique="1"> <longdesc> Number of seconds to sleep during operations. This can be used to test how the cluster reacts to operation timeouts. </longdesc> <shortdesc>Operation sleep duration in seconds.</shortdesc> <content type="string" default="0" /> </parameter> </parameters> <actions> <action name="start" timeout="20" /> <action name="stop" timeout="20" /> <action name="monitor" timeout="20" interval="10" depth="0"/> <action name="reload" timeout="20" /> <action name="migrate_to" timeout="20" /> <action name="migrate_from" timeout="20" /> <action name="validate-all" timeout="20" /> <action name="meta-data" timeout="5" /> </actions> </resource-agent>
Example 4.6. An OCF resource with a non-default start timeout
<primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat"> <operations> <op id="Public-IP-start" name="start" timeout="60s"/> </operations> <instance_attributes id="params-public-ip"> <nvpair id="public-ip-addr" name="ip" value="192.0.2.2"/> </instance_attributes> </primitive>
op element as XML attributes, or in a separate meta_attributes block as nvpair elements. XML attributes take precedence over nvpair elements if both are specified.
Table 4.3. Properties of an Operation
| Field | Default | Description |
|---|---|---|
id
|
| |
name
|
| |
interval
|
0
|
How frequently (in seconds) to perform the operation. A value of 0 means "when needed". A positive value defines a recurring action, which is typically used with monitor.
|
timeout
|
| |
on-fail
|
Varies by action:
|
The action to take if this action ever fails. Allowed values:
|
enabled
|
TRUE
|
If
false, ignore this operation definition. This is typically used to pause a particular recurring monitor operation; for instance, it can complement the respective resource being unmanaged (is-managed=false), as this alone will not block any configured monitoring. Disabling the operation does not suppress all actions of the given type. Allowed values: true, false.
|
record-pending
|
TRUE
|
If
true, the intention to perform the operation is recorded so that GUIs and CLI tools can indicate that an operation is in progress. This is best set as an operation default (see Section 4.5.4, “Setting Global Defaults for Operations”). Allowed values: true, false.
|
role
|
|
Run the operation only on node(s) that the cluster thinks should be in the specified role. This only makes sense for recurring
monitor operations. Allowed (case-sensitive) values: Stopped, Started, and in the case of promotable clone resources, Slave and Master.
|
Note
on-fail is set to demote, recovery from failure by a successful demote causes the cluster to recalculate whether and where a new instance should be promoted. The node with the failure is eligible, so if master scores have not changed, it will be promoted again.
migration-threshold for the master role, but the same effect can be achieved with a location constraint using a rule with a node attribute expression for the resource’s fail count.
<rsc_location id="loc1" rsc="my_primitive"> <rule id="rule1" score="-INFINITY" role="Master" boolean-op="or"> <expression id="expr1" attribute="fail-count-my_primitive#promote_0" operation="gte" value="1"/> <expression id="expr2" attribute="fail-count-my_primitive#monitor_10000" operation="gte" value="1"/> </rule> </rsc_location>
my_primitive resource (note that the primitive name, not the clone name, is used in the rule), and that there is a recurring 10-second-interval monitor configured for the master role (fail count attributes specify the interval in milliseconds).
monitor operations (referred to as probes) to ensure the resource is running where it’s supposed to be, and not running where it’s not supposed to be. (This behavior can be affected by the resource-discovery location constraint property.)
monitor operations explicitly to perform these checks.
Example 4.7. An OCF resource with a recurring health check
<primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat"> <operations> <op id="Public-IP-start" name="start" timeout="60s"/> <op id="Public-IP-monitor" name="monitor" interval="60s"/> </operations> <instance_attributes id="params-public-ip"> <nvpair id="public-ip-addr" name="ip" value="192.0.2.2"/> </instance_attributes> </primitive>
monitor operation will ensure that the resource is running where it is supposed to. The target-role property can be used for further checking.
monitor operation with interval=10 role=Started and a second monitor operation with interval=11 role=Stopped, the cluster will run the first monitor on any nodes it thinks should be running the resource, and the second monitor on any nodes that it thinks should not be running the resource (for the truly paranoid, who want to know when an administrator manually starts a service by mistake).
Note
role=Stopped are not implemented for clone resources.
monitor operations behave differently under various administrative settings:
is-managed=false): No monitors will be stopped.
target-role to be set to Stopped then Started to be recovered.
monitor operations will be stopped on the node, except those specifying role as Stopped (which will be newly initiated if appropriate).
role as Stopped (which will be newly initiated if appropriate). As with single unmanaged resources, starting a resource on a node other than where the cluster expects it to be will cause problems.
op_defaults section of the CIB’s configuration section, and can be set with crm_attribute. For example,
# crm_attribute --type op_defaults --name timeout --update 20s
timeout to 20 seconds. If an operation’s definition also includes a value for timeout, then that value would be used for that operation instead.
start, stop and a non-recurring monitor operation used at startup to check whether the resource is already active. If one of these is taking too long, then you can create an entry for them and specify a longer timeout.
Example 4.8. An OCF resource with custom timeouts for its implicit actions
<primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat"> <operations> <op id="public-ip-startup" name="monitor" interval="0" timeout="90s"/> <op id="public-ip-start" name="start" interval="0" timeout="180s"/> <op id="public-ip-stop" name="stop" interval="0" timeout="15min"/> </operations> <instance_attributes id="params-public-ip"> <nvpair id="public-ip-addr" name="ip" value="192.0.2.2"/> </instance_attributes> </primitive>
monitor operations as you like. In this way, you can do a superficial health check every minute and progressively more intense ones at higher intervals.
OCF_CHECK_LEVEL for this purpose and dictates that it is "made available to the resource agent without the normal OCF_RESKEY prefix".
instance_attributes block to the op tag. It is up to each resource agent to look for the parameter and decide how to use it.
Example 4.9. An OCF resource with two recurring health checks, performing different levels of checks specified via OCF_CHECK_LEVEL.
<primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat"> <operations> <op id="public-ip-health-60" name="monitor" interval="60"> <instance_attributes id="params-public-ip-depth-60"> <nvpair id="public-ip-depth-60" name="OCF_CHECK_LEVEL" value="10"/> </instance_attributes> </op> <op id="public-ip-health-300" name="monitor" interval="300"> <instance_attributes id="params-public-ip-depth-300"> <nvpair id="public-ip-depth-300" name="OCF_CHECK_LEVEL" value="20"/> </instance_attributes> </op> </operations> <instance_attributes id="params-public-ip"> <nvpair id="public-ip-level" name="ip" value="192.0.2.2"/> </instance_attributes> </primitive>
enabled=false to the operation’s definition.
Example 4.10. Example of an OCF resource with a disabled health check
<primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat"> <operations> <op id="public-ip-check" name="monitor" interval="60s" enabled="false"/> </operations> <instance_attributes id="params-public-ip"> <nvpair id="public-ip-addr" name="ip" value="192.0.2.2"/> </instance_attributes> </primitive>
# cibadmin --modify --xml-text '<op id="public-ip-check" enabled="false"/>'
# cibadmin --modify --xml-text '<op id="public-ip-check" enabled="true"/>'
ocf-tester script, which can be useful in this regard.
Table of Contents
INFINITY (or equivalently, +INFINITY) internally as a score of 1,000,000. Addition and subtraction with it follow these three basic rules:
INFINITY = INFINITY
INFINITY = -INFINITY
INFINITY - INFINITY = -INFINITY
Note
Table 5.1. Attributes of a rsc_location Element
| Attribute | Default | Description |
|---|---|---|
id
|
| |
rsc
|
| |
rsc-pattern
|
|
A pattern matching the names of resources to which this constraint applies. The syntax is the same as POSIX extended regular expressions, with the addition of an initial ! indicating that resources not matching the pattern are selected. If the regular expression contains submatches, and the constraint is governed by a rule, the submatches can be referenced as
%0 through %9 in the rule’s score-attribute or a rule expression’s attribute. A location constraint must either have a rsc, have a rsc-pattern, or contain at least one resource set.
|
node
|
| |
score
|
|
Positive values indicate a preference for running the affected resource(s) on
node — the higher the value, the stronger the preference. Negative values indicate the resource(s) should avoid this node (a value of -INFINITY changes "should" to "must"). A location constraint must either have a node and score, or contain at least one rule.
|
resource-discovery
|
always
|
Whether Pacemaker should perform resource discovery (that is, check whether the resource is already running) for this resource on this node. This should normally be left as the default, so that rogue instances of a service can be stopped when they are running where they are not supposed to be. However, there are two situations where disabling resource discovery is a good idea: when a service is not installed on a node, discovery might return an error (properly written OCF agents will not, so this is usually only seen with other agent types); and when Pacemaker Remote is used to scale a cluster to hundreds of nodes, limiting resource discovery to allowed nodes can significantly boost performance.
|
Warning
never or exclusive removes Pacemaker’s ability to detect and stop unwanted instances of a service running where it’s not supposed to be. It is up to the system administrator (you!) to make sure that the service can never be active on nodes without resource-discovery (such as by leaving the relevant software uninstalled).
# crm_attribute --name symmetric-cluster --update false
Example 5.1. Opt-in location constraints for two resources
<constraints> <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="200"/> <rsc_location id="loc-2" rsc="Webserver" node="sles-3" score="0"/> <rsc_location id="loc-3" rsc="Database" node="sles-2" score="200"/> <rsc_location id="loc-4" rsc="Database" node="sles-3" score="0"/> </constraints>
# crm_attribute --name symmetric-cluster --update true
Example 5.2. Opt-out location constraints for two resources
<constraints> <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="200"/> <rsc_location id="loc-2-do-not-run" rsc="Webserver" node="sles-2" score="-INFINITY"/> <rsc_location id="loc-3-do-not-run" rsc="Database" node="sles-1" score="-INFINITY"/> <rsc_location id="loc-4" rsc="Database" node="sles-2" score="200"/> </constraints>
Example 5.3. Constraints where a resource prefers two nodes equally
<constraints> <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="INFINITY"/> <rsc_location id="loc-2" rsc="Webserver" node="sles-2" score="INFINITY"/> <rsc_location id="loc-3" rsc="Database" node="sles-1" score="500"/> <rsc_location id="loc-4" rsc="Database" node="sles-2" score="300"/> <rsc_location id="loc-5" rsc="Database" node="sles-2" score="200"/> </constraints>
Webserver would probably be placed on sles-1 and Database on sles-2. It would likely have placed Webserver based on the node’s uname and Database based on the desire to spread the resource load evenly across the cluster. However other factors can also be involved in more complex configurations.
Important
Table 5.2. Attributes of a rsc_order Element
| Field | Default | Description |
|---|---|---|
id
|
| |
first
|
| |
then
|
| |
first-action
|
start
| |
then-action
|
value of
first-action
| |
kind
|
Mandatory
|
How to enforce the constraint. Allowed values:
|
symmetrical
|
TRUE for
Mandatory and Optional kinds. FALSE for Serialize kind.
|
Database must start before Webserver, and IP should start before Webserver if they both need to be started:
Example 5.4. Optional and mandatory ordering constraints
<constraints> <rsc_order id="order-1" first="IP" then="Webserver" kind="Optional"/> <rsc_order id="order-2" first="Database" then="Webserver" kind="Mandatory" /> </constraints>
symmetrical default to TRUE, Webserver must be stopped before Database can be stopped, and Webserver should be stopped before IP if they both need to be stopped.
Important
Table 5.3. Attributes of a rsc_colocation Constraint
| Field | Default | Description |
|---|---|---|
id
|
| |
rsc
|
| |
with-rsc
|
| |
node-attribute
|
#uname
|
The node attribute that must be the same on the node running
rsc and the node running with-rsc for the constraint to be satisfied. (For details, see Section 5.4.4, “Colocation by Node Attribute”.)
|
score
|
|
+INFINITY or -INFINITY. In such cases, if the constraint can’t be satisfied, then the rsc resource is not permitted to run. For score=INFINITY, this includes cases where the with-rsc resource is not active.
A to always run on the same machine as resource B, you would add the following constraint:
Example 5.5. Mandatory colocation constraint for two resources
<rsc_colocation id="colocate" rsc="A" with-rsc="B" score="INFINITY"/>
INFINITY was used, if B can’t run on any of the cluster nodes (for whatever reason) then A will not be allowed to run. Whether A is running or not has no effect on B.
A cannot run on the same machine as B. In this case, use score="-INFINITY".
Example 5.6. Mandatory anti-colocation constraint for two resources
<rsc_colocation id="anti-colocate" rsc="A" with-rsc="B" score="-INFINITY"/>
-INFINITY, the constraint is binding. So if the only place left to run is where B already is, then A may not run anywhere.
INFINITY, B can run even if A is stopped. However, in this case A also can run if B is stopped, because it still meets the constraint of A and B not running on the same node.
-INFINITY and less than INFINITY, the cluster will try to accommodate your wishes but may ignore them if the alternative is to stop some of the cluster resources.
Example 5.7. Advisory colocation constraint for two resources
<rsc_colocation id="colocate-maybe" rsc="A" with-rsc="B" score="500"/>
node-attribute property of a colocation constraints allows you to express the requirement, "these resources must be on similar nodes".
r1 and r2 such that r2 needs to use the same SAN as r1, but doesn’t necessarily have to be on the same exact node. In such a case, you could define a node attribute named san, with the value san1 or san2 on each node as appropriate. Then, you could colocate r2 with r1 using node-attribute set to san.
Example 5.8. A set of 3 resources
<resource_set id="resource-set-example"> <resource_ref id="A"/> <resource_ref id="B"/> <resource_ref id="C"/> </resource_set>
rsc_location, rsc_order (see Section 5.6, “Ordering Sets of Resources”), rsc_colocation (see Section 5.7, “Colocating Sets of Resources”), and rsc_ticket (see Section 15.3, “Configuring Ticket Dependencies”) constraints.
Table 5.4. Attributes of a resource_set Element
Example 5.9. A chain of ordered resources
<constraints> <rsc_order id="order-1" first="A" then="B" /> <rsc_order id="order-2" first="B" then="C" /> <rsc_order id="order-3" first="C" then="D" /> </constraints>
Example 5.10. A chain of ordered resources expressed as a set
<constraints> <rsc_order id="order-1"> <resource_set id="ordered-set-example" sequential="true"> <resource_ref id="A"/> <resource_ref id="B"/> <resource_ref id="C"/> <resource_ref id="D"/> </resource_set> </rsc_order> </constraints>
Important
A B may be equivalent to A then B, or B then A.
sequential property). In the example below, A and B can both start in parallel, as can C and D, however C and D can only start once both A and B are active.
Example 5.11. Ordered sets of unordered resources
<constraints> <rsc_order id="order-1"> <resource_set id="ordered-set-1" sequential="false"> <resource_ref id="A"/> <resource_ref id="B"/> </resource_set> <resource_set id="ordered-set-2" sequential="false"> <resource_ref id="C"/> <resource_ref id="D"/> </resource_set> </rsc_order> </constraints>
sequential="true") and there is no limit to the number of sets that can be specified.
Example 5.12. Advanced use of set ordering - Three ordered sets, two of which are internally unordered
<constraints> <rsc_order id="order-1"> <resource_set id="ordered-set-1" sequential="false"> <resource_ref id="A"/> <resource_ref id="B"/> </resource_set> <resource_set id="ordered-set-2" sequential="true"> <resource_ref id="C"/> <resource_ref id="D"/> </resource_set> <resource_set id="ordered-set-3" sequential="false"> <resource_ref id="E"/> <resource_ref id="F"/> </resource_set> </rsc_order> </constraints>
Important
sequential=false makes sense only if there is another set in the constraint. Otherwise, the constraint has no effect.
(A and B) then (C) then (D) then (E and F).
(A and B), to use "OR" logic so the sets look like this: (A or B) then (C) then (D) then (E and F). This functionality can be achieved through the use of the require-all option. This option defaults to TRUE which is why the "AND" logic is used by default. Setting require-all=false means only one resource in the set needs to be started before continuing on to the next set.
Example 5.13. Resource Set "OR" logic: Three ordered sets, where the first set is internally unordered with "OR" logic
<constraints> <rsc_order id="order-1"> <resource_set id="ordered-set-1" sequential="false" require-all="false"> <resource_ref id="A"/> <resource_ref id="B"/> </resource_set> <resource_set id="ordered-set-2" sequential="true"> <resource_ref id="C"/> <resource_ref id="D"/> </resource_set> <resource_set id="ordered-set-3" sequential="false"> <resource_ref id="E"/> <resource_ref id="F"/> </resource_set> </rsc_order> </constraints>
Important
require-all=false makes sense only in conjunction with sequential=false. Think of it like this: sequential=false modifies the set to be an unordered set using "AND" logic by default, and adding require-all=false flips the unordered set’s "AND" logic to "OR" logic.
Example 5.14. Colocation chain as individual constraints, where A is placed first, then B, then C, then D
<constraints> <rsc_colocation id="coloc-1" rsc="D" with-rsc="C" score="INFINITY"/> <rsc_colocation id="coloc-2" rsc="C" with-rsc="B" score="INFINITY"/> <rsc_colocation id="coloc-3" rsc="B" with-rsc="A" score="INFINITY"/> </constraints>
Example 5.15. Equivalent colocation chain expressed using resource_set
<constraints> <rsc_colocation id="coloc-1" score="INFINITY" > <resource_set id="colocated-set-example" sequential="true"> <resource_ref id="A"/> <resource_ref id="B"/> <resource_ref id="C"/> <resource_ref id="D"/> </resource_set> </rsc_colocation> </constraints>
Note
resource_set, the resources are listed in the order they are placed, which is the reverse of the order in which they are colocated. In the above example, resource A is placed before resource B, which is the same as saying resource B is colocated with resource A.
B is unable to run, then both C and by inference D must remain stopped.
Important
A B may be equivalent to A with B, or B with A.
sequential property).
B, C, and D will each be colocated with A (which will be placed first). A must be able to run in order for any of the resources to run, but any of B, C, or D may be stopped without affecting any of the others.
Example 5.16. Using colocated sets to specify a shared dependency
<constraints> <rsc_colocation id="coloc-1" score="INFINITY" > <resource_set id="colocated-set-2" sequential="false"> <resource_ref id="B"/> <resource_ref id="C"/> <resource_ref id="D"/> </resource_set> <resource_set id="colocated-set-1" sequential="true"> <resource_ref id="A"/> </resource_set> </rsc_colocation> </constraints>
Note
Important
sequential="false" makes sense only if there is another set in the constraint. Otherwise, the constraint has no effect.
sequential="true", then in order for one member of that set to be active, all members listed before it must also be active.
role property.
Example 5.17. Colocation in which the members of the middle set have no interdependencies, and the last set listed applies only to instances in the master role
<constraints> <rsc_colocation id="coloc-1" score="INFINITY" > <resource_set id="colocated-set-1" sequential="true"> <resource_ref id="F"/> <resource_ref id="G"/> </resource_set> <resource_set id="colocated-set-2" sequential="false"> <resource_ref id="C"/> <resource_ref id="D"/> <resource_ref id="E"/> </resource_set> <resource_set id="colocated-set-3" sequential="true" role="Master"> <resource_ref id="A"/> <resource_ref id="B"/> </resource_set> </rsc_colocation> </constraints>
Note
require-all option.
Table of Contents
on-fail property of any resource operation to fencing, in which case the cluster will fence the resource’s node if the operation fails.
stonith-class resource agent.
off and reboot) that the cluster can use to fence nodes. As with other resource agent classes, this allows a layer of abstraction so that Pacemaker doesn’t need any knowledge about specific fencing technologies — that knowledge is isolated in the agent.
target-role to Stopped, then no node can use that device. If mandatory location constraints prevent a specific node from "running" a fence device, then that node will never be chosen to execute fencing using the device. A node may fence itself, but the cluster will choose that only if no other nodes can do the fencing.
-INFINITY), so that the node can fence itself if no other nodes can.
pacemaker-fenced.
Table 6.1. Additional Properties of Fencing Resources
| Field | Type | Default | Description |
|---|---|---|---|
stonith-timeout
|
NA
|
NA
| |
provides
|
string
|
|
Any special capability provided by the fence device. Currently, only one such capability is meaningful:
unfencing (see Section 6.8, “Unfencing”).
|
pcmk_host_map
|
string
|
| |
pcmk_host_list
|
string
|
| |
pcmk_host_check
|
string
|
A value appropriate to other configuration options and device capabilities (see note below)
|
How to determine which machines are controlled by the device. Allowed values:
|
pcmk_delay_max
|
time
|
0s
|
Enable a random delay of up to the time specified before executing fencing actions. This is sometimes used in two-node clusters to ensure that the nodes don’t fence each other at the same time. The overall delay introduced by pacemaker is derived from this random delay value adding a static delay so that the sum is kept below the maximum delay.
|
pcmk_delay_base
|
time
|
0s
|
Enable a static delay before executing fencing actions. This can be used e.g. in two-node clusters to ensure that the nodes don’t fence each other, by having separate fencing resources with different values. The node that is fenced with the shorter delay will lose a fencing race. The overall delay introduced by pacemaker is derived from this value plus a random delay such that the sum is kept below the maximum delay.
|
pcmk_action_limit
|
integer
|
1
|
The maximum number of actions that can be performed in parallel on this device, if the cluster option
concurrent-fencing is true. -1 is unlimited.
|
pcmk_host_argument
|
string
|
port otherwise plug if supported according to the metadata of the fence agent
|
Advanced use only. Which parameter should be supplied to the fence agent to identify the node to be fenced. Some devices support neither the standard
plug nor the deprecated port parameter, or may provide additional ones. Use this to specify an alternate, device-specific parameter. A value of none tells the cluster not to supply any additional parameters.
|
pcmk_reboot_action
|
string
|
reboot
| |
pcmk_reboot_timeout
|
time
|
60s
| |
pcmk_reboot_retries
|
integer
|
2
|
Advanced use only. The maximum number of times to retry the
reboot command within the timeout period. Some devices do not support multiple connections, and operations may fail if the device is busy with another task, so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries before giving up.
|
pcmk_off_action
|
string
|
off
| |
pcmk_off_timeout
|
time
|
60s
| |
pcmk_off_retries
|
integer
|
2
|
Advanced use only. The maximum number of times to retry the
off command within the timeout period. Some devices do not support multiple connections, and operations may fail if the device is busy with another task, so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries before giving up.
|
pcmk_list_action
|
string
|
list
| |
pcmk_list_timeout
|
time
|
60s
| |
pcmk_list_retries
|
integer
|
2
|
Advanced use only. The maximum number of times to retry the
list command within the timeout period. Some devices do not support multiple connections, and operations may fail if the device is busy with another task, so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries before giving up.
|
pcmk_monitor_action
|
string
|
monitor
| |
pcmk_monitor_timeout
|
time
|
60s
| |
pcmk_monitor_retries
|
integer
|
2
|
Advanced use only. The maximum number of times to retry the
monitor command within the timeout period. Some devices do not support multiple connections, and operations may fail if the device is busy with another task, so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries before giving up.
|
pcmk_status_action
|
string
|
status
| |
pcmk_status_timeout
|
time
|
60s
| |
pcmk_status_retries
|
integer
|
2
|
Advanced use only. The maximum number of times to retry the
status command within the timeout period. Some devices do not support multiple connections, and operations may fail if the device is busy with another task, so Pacemaker will automatically retry the operation, if there is time remaining. Use this option to alter the number of times Pacemaker retries before giving up.
|
Note
pcmk_host_check is static-list if either pcmk_host_list or pcmk_host_map is configured. If neither of those are configured, the default is dynamic-list if the fence device supports the list action, or status if the fence device supports the status action but not the list action. If none of those conditions apply, the default is none.
on command.
requires set to unfencing, then that resource will not be probed or started on a node until that node has been unfenced.
concurrent-fencing cluster property should be set to true. Otherwise, if both the node running the depended-on resource and some node targeted by the dependent fence device need to be fenced, the fencing of the node running the depended-on resource might be ordered first, making the second fencing impossible and blocking further recovery. With concurrent fencing, the dependent fence device might fail at first due to the depended-on resource being unavailable, but it will be retried and eventually succeed once the resource is brought back up.
on-fail set to fencing fails on the DC.
# stonith_admin --list-installed
# stonith_admin --metadata --agent $AGENT_NAME
stonith.xml containing a primitive resource with a class of stonith, a type equal to the agent name obtained earlier, and a parameter for each of the values returned in the previous step.
pcmk_host_map parameter. See man pacemaker-fenced for details.
list command, you may also need to set the special pcmk_host_list and/or pcmk_host_check parameters. See man pacemaker-fenced for details.
port parameter, you may also need to set the special pcmk_host_argument parameter. See man pacemaker-fenced for details.
# cibadmin -C -o resources --xml-file stonith.xml
stonith-enabled to true:
# crm_attribute -t crm_config -n stonith-enabled -v true
# stonith_admin --reboot nodename
fence_ipmilan driver, and obtain the following list of parameters:
Example 6.1. Obtaining a list of Fence Agent Parameters
# stonith_admin --metadata -a fence_ipmilan
<resource-agent name="fence_ipmilan" shortdesc="Fence agent for IPMI over LAN"> <symlink name="fence_ilo3" shortdesc="Fence agent for HP iLO3"/> <symlink name="fence_ilo4" shortdesc="Fence agent for HP iLO4"/> <symlink name="fence_idrac" shortdesc="Fence agent for Dell iDRAC"/> <symlink name="fence_imm" shortdesc="Fence agent for IBM Integrated Management Module"/> <longdesc> </longdesc> <vendor-url> </vendor-url> <parameters> <parameter name="auth" unique="0" required="0"> <getopt mixed="-A"/> <content type="string"/> <shortdesc> </shortdesc> </parameter> <parameter name="ipaddr" unique="0" required="1"> <getopt mixed="-a"/> <content type="string"/> <shortdesc> </shortdesc> </parameter> <parameter name="passwd" unique="0" required="0"> <getopt mixed="-p"/> <content type="string"/> <shortdesc> </shortdesc> </parameter> <parameter name="passwd_script" unique="0" required="0"> <getopt mixed="-S"/> <content type="string"/> <shortdesc> </shortdesc> </parameter> <parameter name="lanplus" unique="0" required="0"> <getopt mixed="-P"/> <content type="boolean"/> <shortdesc> </shortdesc> </parameter> <parameter name="login" unique="0" required="0"> <getopt mixed="-l"/> <content type="string"/> <shortdesc> </shortdesc> </parameter> <parameter name="action" unique="0" required="0"> <getopt mixed="-o"/> <content type="string" default="reboot"/> <shortdesc> </shortdesc> </parameter> <parameter name="timeout" unique="0" required="0"> <getopt mixed="-t"/> <content type="string"/> <shortdesc> </shortdesc> </parameter> <parameter name="cipher" unique="0" required="0"> <getopt mixed="-C"/> <content type="string"/> <shortdesc> </shortdesc> </parameter> <parameter name="method" unique="0" required="0"> <getopt mixed="-M"/> <content type="string" default="onoff"/> <shortdesc> </shortdesc> </parameter> <parameter name="power_wait" unique="0" required="0"> <getopt mixed="-T"/> <content type="string" default="2"/> <shortdesc> </shortdesc> </parameter> <parameter name="delay" unique="0" required="0"> <getopt mixed="-f"/> <content type="string"/> <shortdesc> </shortdesc> </parameter> <parameter name="privlvl" unique="0" required="0"> <getopt mixed="-L"/> <content type="string"/> <shortdesc> </shortdesc> </parameter> <parameter name="verbose" unique="0" required="0"> <getopt mixed="-v"/> <content type="boolean"/> <shortdesc> </shortdesc> </parameter> </parameters> <actions> <action name="on"/> <action name="off"/> <action name="reboot"/> <action name="status"/> <action name="diag"/> <action name="list"/> <action name="monitor"/> <action name="metadata"/> <action name="stop" timeout="20s"/> <action name="start" timeout="20s"/> </actions> </resource-agent>
Example 6.2. An IPMI-based Fencing Resource
<primitive id="Fencing" class="stonith" type="fence_ipmilan" > <instance_attributes id="Fencing-params" > <nvpair id="Fencing-passwd" name="passwd" value="testuser" /> <nvpair id="Fencing-login" name="login" value="abc123" /> <nvpair id="Fencing-ipaddr" name="ipaddr" value="192.0.2.1" /> <nvpair id="Fencing-pcmk_host_list" name="pcmk_host_list" value="pcmk-1 pcmk-2" /> </instance_attributes> <operations > <op id="Fencing-monitor-10m" interval="10m" name="monitor" timeout="300s" /> </operations> </primitive>
# crm_attribute -t crm_config -n stonith-enabled -v true
fencing-level entries in the fencing-topology section of the configuration.
index. Allowed values are 1 through 9.
Table 6.2. Properties of Fencing Levels
| Field | Description |
|---|---|
id
| |
target
| |
target-pattern
|
An extended regular expression (as defined in POSIX) matching the names of nodes to which this level applies
|
target-attribute
| |
target-value
| |
index
| |
devices
|
Example 6.3. Fencing topology with different devices for different nodes
<cib crm_feature_set="3.0.6" validate-with="pacemaker-1.2" admin_epoch="1" epoch="0" num_updates="0"> <configuration> ... <fencing-topology> <!-- For pcmk-1, try poison-pill and fail back to power --> <fencing-level id="f-p1.1" target="pcmk-1" index="1" devices="poison-pill"/> <fencing-level id="f-p1.2" target="pcmk-1" index="2" devices="power"/> <!-- For pcmk-2, try disk and network, and fail back to power --> <fencing-level id="f-p2.1" target="pcmk-2" index="1" devices="disk,network"/> <fencing-level id="f-p2.2" target="pcmk-2" index="2" devices="power"/> </fencing-topology> ... <configuration> <status/> </cib>
fencing-topology in a cluster with the following properties:
fence_ipmi agent
fence_apc_snmp agent targetting 2 fencing devices (one per PSU, either port 10 or 11)
fence_ipmi to try to kill the faulty node. Using a fencing topology, if that first method fails, STONITH will then move on to selecting fence_apc_snmp twice:
fence_ipmi, and so on until the node is fenced or fencing action is cancelled.
<primitive class="stonith" id="fence_prod-mysql1_ipmi" type="fence_ipmilan"> <instance_attributes id="fence_prod-mysql1_ipmi-instance_attributes"> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-ipaddr" name="ipaddr" value="192.0.2.1"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-verbose" name="verbose" value="true"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/> </instance_attributes> </primitive> <primitive class="stonith" id="fence_prod-mysql2_ipmi" type="fence_ipmilan"> <instance_attributes id="fence_prod-mysql2_ipmi-instance_attributes"> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-ipaddr" name="ipaddr" value="192.0.2.2"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-verbose" name="verbose" value="true"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/> </instance_attributes> </primitive>
<primitive class="stonith" id="fence_prod-mysql1_apc1" type="fence_apc_snmp"> <instance_attributes id="fence_prod-mysql1_apc1-instance_attributes"> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-ipaddr" name="ipaddr" value="198.51.100.1"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-port" name="port" value="10"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-passwd" name="passwd" value="fencing"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/> </instance_attributes> </primitive> <primitive class="stonith" id="fence_prod-mysql1_apc2" type="fence_apc_snmp"> <instance_attributes id="fence_prod-mysql1_apc2-instance_attributes"> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-ipaddr" name="ipaddr" value="203.0.113.1"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-port" name="port" value="10"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-passwd" name="passwd" value="fencing"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/> </instance_attributes> </primitive> <primitive class="stonith" id="fence_prod-mysql2_apc1" type="fence_apc_snmp"> <instance_attributes id="fence_prod-mysql2_apc1-instance_attributes"> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-ipaddr" name="ipaddr" value="198.51.100.1"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-port" name="port" value="11"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-passwd" name="passwd" value="fencing"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/> </instance_attributes> </primitive> <primitive class="stonith" id="fence_prod-mysql2_apc2" type="fence_apc_snmp"> <instance_attributes id="fence_prod-mysql2_apc2-instance_attributes"> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-ipaddr" name="ipaddr" value="203.0.113.1"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-port" name="port" value="11"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-passwd" name="passwd" value="fencing"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/> </instance_attributes> </primitive>
<constraints> <rsc_location id="l_fence_prod-mysql1_ipmi" node="prod-mysql1" rsc="fence_prod-mysql1_ipmi" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql2_ipmi" node="prod-mysql2" rsc="fence_prod-mysql2_ipmi" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql1_apc2" node="prod-mysql1" rsc="fence_prod-mysql1_apc2" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql1_apc1" node="prod-mysql1" rsc="fence_prod-mysql1_apc1" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql2_apc1" node="prod-mysql2" rsc="fence_prod-mysql2_apc1" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql2_apc2" node="prod-mysql2" rsc="fence_prod-mysql2_apc2" score="-INFINITY"/> </constraints>
<fencing-topology> <fencing-level devices="fence_prod-mysql1_ipmi" id="fencing-2" index="1" target="prod-mysql1"/> <fencing-level devices="fence_prod-mysql1_apc1,fence_prod-mysql1_apc2" id="fencing-3" index="2" target="prod-mysql1"/> <fencing-level devices="fence_prod-mysql2_ipmi" id="fencing-0" index="1" target="prod-mysql2"/> <fencing-level devices="fence_prod-mysql2_apc1,fence_prod-mysql2_apc2" id="fencing-1" index="2" target="prod-mysql2"/> </fencing-topology>
fencing-topology, the lowest index value determines the priority of the first fencing method.
<cib admin_epoch="0" crm_feature_set="3.0.7" epoch="292" have-quorum="1" num_updates="29" validate-with="pacemaker-1.2"> <configuration> <crm_config> <cluster_property_set id="cib-bootstrap-options"> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="true"/> <nvpair id="cib-bootstrap-options-stonith-action" name="stonith-action" value="off"/> <nvpair id="cib-bootstrap-options-expected-quorum-votes" name="expected-quorum-votes" value="3"/> ... </cluster_property_set> </crm_config> <nodes> <node id="prod-mysql1" uname="prod-mysql1"> <node id="prod-mysql2" uname="prod-mysql2"/> <node id="prod-mysql-rep1" uname="prod-mysql-rep1"/> <instance_attributes id="prod-mysql-rep1"> <nvpair id="prod-mysql-rep1-standby" name="standby" value="on"/> </instance_attributes> </node> </nodes> <resources> <primitive class="stonith" id="fence_prod-mysql1_ipmi" type="fence_ipmilan"> <instance_attributes id="fence_prod-mysql1_ipmi-instance_attributes"> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-ipaddr" name="ipaddr" value="192.0.2.1"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-verbose" name="verbose" value="true"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/> <nvpair id="fence_prod-mysql1_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/> </instance_attributes> </primitive> <primitive class="stonith" id="fence_prod-mysql2_ipmi" type="fence_ipmilan"> <instance_attributes id="fence_prod-mysql2_ipmi-instance_attributes"> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-ipaddr" name="ipaddr" value="192.0.2.2"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-passwd" name="passwd" value="finishme"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-verbose" name="verbose" value="true"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/> <nvpair id="fence_prod-mysql2_ipmi-instance_attributes-lanplus" name="lanplus" value="true"/> </instance_attributes> </primitive> <primitive class="stonith" id="fence_prod-mysql1_apc1" type="fence_apc_snmp"> <instance_attributes id="fence_prod-mysql1_apc1-instance_attributes"> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-ipaddr" name="ipaddr" value="198.51.100.1"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-port" name="port" value="10"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-passwd" name="passwd" value="fencing"/> <nvpair id="fence_prod-mysql1_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/> </instance_attributes> </primitive> <primitive class="stonith" id="fence_prod-mysql1_apc2" type="fence_apc_snmp"> <instance_attributes id="fence_prod-mysql1_apc2-instance_attributes"> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-ipaddr" name="ipaddr" value="203.0.113.1"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-port" name="port" value="10"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-passwd" name="passwd" value="fencing"/> <nvpair id="fence_prod-mysql1_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql1"/> </instance_attributes> </primitive> <primitive class="stonith" id="fence_prod-mysql2_apc1" type="fence_apc_snmp"> <instance_attributes id="fence_prod-mysql2_apc1-instance_attributes"> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-ipaddr" name="ipaddr" value="198.51.100.1"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-port" name="port" value="11"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-passwd" name="passwd" value="fencing"/> <nvpair id="fence_prod-mysql2_apc1-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/> </instance_attributes> </primitive> <primitive class="stonith" id="fence_prod-mysql2_apc2" type="fence_apc_snmp"> <instance_attributes id="fence_prod-mysql2_apc2-instance_attributes"> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-ipaddr" name="ipaddr" value="203.0.113.1"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-action" name="action" value="off"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-port" name="port" value="11"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-login" name="login" value="fencing"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-passwd" name="passwd" value="fencing"/> <nvpair id="fence_prod-mysql2_apc2-instance_attributes-pcmk_host_list" name="pcmk_host_list" value="prod-mysql2"/> </instance_attributes> </primitive> </resources> <constraints> <rsc_location id="l_fence_prod-mysql1_ipmi" node="prod-mysql1" rsc="fence_prod-mysql1_ipmi" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql2_ipmi" node="prod-mysql2" rsc="fence_prod-mysql2_ipmi" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql1_apc2" node="prod-mysql1" rsc="fence_prod-mysql1_apc2" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql1_apc1" node="prod-mysql1" rsc="fence_prod-mysql1_apc1" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql2_apc1" node="prod-mysql2" rsc="fence_prod-mysql2_apc1" score="-INFINITY"/> <rsc_location id="l_fence_prod-mysql2_apc2" node="prod-mysql2" rsc="fence_prod-mysql2_apc2" score="-INFINITY"/> </constraints> <fencing-topology> <fencing-level devices="fence_prod-mysql1_ipmi" id="fencing-2" index="1" target="prod-mysql1"/> <fencing-level devices="fence_prod-mysql1_apc1,fence_prod-mysql1_apc2" id="fencing-3" index="2" target="prod-mysql1"/> <fencing-level devices="fence_prod-mysql2_ipmi" id="fencing-0" index="1" target="prod-mysql2"/> <fencing-level devices="fence_prod-mysql2_apc1,fence_prod-mysql2_apc2" id="fencing-1" index="2" target="prod-mysql2"/> </fencing-topology> ... </configuration> </cib>
stonith-action is reboot or because a reboot was manually requested (such as by stonith_admin --reboot), it will remap that to other commands in two cases:
reboot command, the cluster will ask it to perform off instead.
off, then ask the devices to perform on.
off commands succeed, because then it is safe for the cluster to recover any resources that were on the node. Timeouts and errors in the on phase will be logged but ignored.
pcmk_off_timeout will be used when executing the off command, not pcmk_reboot_timeout).
Table of Contents
Example 7.1. Simple alert configuration
<configuration> <alerts> <alert id="my-alert" path="/path/to/my-script.sh" /> </alerts> </configuration>
my-script.sh for each event.
Example 7.2. Alert configuration with recipient
<configuration> <alerts> <alert id="my-alert" path="/path/to/my-script.sh"> <recipient id="my-alert-recipient" value="some-address"/> </alert> </alerts> </configuration>
my-script.sh for each event, passing the recipient some-address as an environment variable.
Table 7.1. Meta-Attributes of an Alert
Example 7.3. Alert configuration with meta-attributes
<configuration> <alerts> <alert id="my-alert" path="/path/to/my-script.sh"> <meta_attributes id="my-alert-attributes"> <nvpair id="my-alert-attributes-timeout" name="timeout" value="15s"/> </meta_attributes> <recipient id="my-alert-recipient1" value="someuser@example.com"> <meta_attributes id="my-alert-recipient1-attributes"> <nvpair id="my-alert-recipient1-timestamp-format" name="timestamp-format" value="%D %H:%M"/> </meta_attributes> </recipient> <recipient id="my-alert-recipient2" value="otheruser@example.com"> <meta_attributes id="my-alert-recipient2-attributes"> <nvpair id="my-alert-recipient2-timestamp-format" name="timestamp-format" value="%c"/> </meta_attributes> </recipient> </alert> </alerts> </configuration>
my-script.sh will get called twice for each event, with each call using a 15-second timeout. One call will be passed the recipient someuser@example.com and a timestamp in the format %D %H:%M, while the other call will be passed the recipient otheruser@example.com and a timestamp in the format %c.
Example 7.4. Alert configuration with instance attributes
<configuration> <alerts> <alert id="my-alert" path="/path/to/my-script.sh"> <meta_attributes id="my-alert-attributes"> <nvpair id="my-alert-attributes-timeout" name="timeout" value="15s"/> </meta_attributes> <instance_attributes id="my-alert-options"> <nvpair id="my-alert-options-debug" name="debug" value="false"/> </instance_attributes> <recipient id="my-alert-recipient1" value="someuser@example.com"/> </alert> </alerts> </configuration>
Example 7.5. Alert configuration to receive only node events and fencing events
<configuration> <alerts> <alert id="my-alert" path="/path/to/my-script.sh"> <select> <select_nodes /> <select_fencing /> </select> <recipient id="my-alert-recipient1" value="someuser@example.com"/> </alert> </alerts> </configuration>
<select> are <select_nodes>, <select_fencing>, <select_resources>, and <select_attributes>.
<select_attributes> (the only event type not enabled by default), the agent will receive alerts when a node attribute changes. If you wish the agent to be called only when certain attributes change, you can configure that as well.
Example 7.6. Alert configuration to be called when certain node attributes change
<configuration> <alerts> <alert id="my-alert" path="/path/to/my-script.sh"> <select> <select_attributes> <attribute id="alert-standby" name="standby" /> <attribute id="alert-shutdown" name="shutdown" /> </select_attributes> </select> <recipient id="my-alert-recipient1" value="someuser@example.com"/> </alert> </alerts> </configuration>
/usr/share/pacemaker/alerts by default.
Example 7.7. Sending cluster events as SNMP traps
<configuration> <alerts> <alert id="snmp_alert" path="/path/to/alert_snmp.sh"> <instance_attributes id="config_for_alert_snmp"> <nvpair id="trap_node_states" name="trap_node_states" value="all"/> </instance_attributes> <meta_attributes id="config_for_timestamp"> <nvpair id="ts_fmt" name="timestamp-format" value="%Y-%m-%d,%H:%M:%S.%01N"/> </meta_attributes> <recipient id="snmp_destination" value="192.168.1.2"/> </alert> </alerts> </configuration>
Example 7.8. Sending cluster events as e-mails
<configuration> <alerts> <alert id="smtp_alert" path="/path/to/alert_smtp.sh"> <instance_attributes id="config_for_alert_smtp"> <nvpair id="email_sender" name="email_sender" value="donotreply@example.com"/> </instance_attributes> <recipient id="smtp_destination" value="admin@example.com"/> </alert> </alerts> </configuration>
Table 7.2. Environment variables passed to alert agents
hacluster user, which has a minimal set of permissions. If an agent requires additional privileges, it is recommended to configure sudo to allow the agent to run the necessary commands as another user with the appropriate privileges.
Note
ocf:pacemaker:ClusterMon resource, which is now deprecated. To preserve this compatibility, the environment variables passed to alert agents are available prepended with CRM_notify_ as well as CRM_alert_. One break in compatibility is that ClusterMon ran external scripts as the root user, while alert agents are run as the hacluster user.
Table of Contents
resource-stickiness during working hours, to minimize downtime, and a lower value on weekends, to allow resources to move to their most preferred locations when people aren’t around to notice.
rule elements specifying conditions under which the constraint or properties take effect. Examples later in this section will make this clearer.
Table 8.1. Attributes of a rule Element
| Attribute | Default | Description |
|---|---|---|
id
|
| |
role
|
Started
| |
score
|
| |
score-attribute
|
| |
boolean-op
|
and
|
rule element must contain one or more conditions. A condition may be an expression element, a date_expression element, or another rule element.
Table 8.2. Attributes of an expression Element
| Field | Default | Description |
|---|---|---|
id
|
| |
attribute
|
| |
type
|
The default type for
lt, gt, lte, and gte operations is number if either value contains a decimal point character, or integer otherwise. The default type for all other operations is string. If a numeric parse fails for either value, then the values compared as type string.
| |
operation
|
|
The comparison to perform (required). Allowed values:
|
value
|
| |
value-source
|
literal
|
How the
value is derived. Allowed values:
|
Table 8.3. Built-in Node Attributes
| Name | Value |
|---|---|
#uname
|
Node name
|
#id
|
Node ID
|
#kind
|
Node type. Possible values are
cluster, remote, and container. Kind is remote for Pacemaker Remote nodes created with the ocf:pacemaker:remote resource, and container for Pacemaker Remote guest nodes and bundle nodes
|
#is_dc
|
"true" if this node is a Designated Controller (DC), "false" otherwise
|
#cluster-name
|
The value of the
cluster-name cluster property, if set
|
#site-name
|
The value of the
site-name node attribute, if set, otherwise identical to #cluster-name
|
#role
|
The role the relevant promotable clone resource has on this node. Valid only within a rule for a location constraint for a promotable clone resource.
|
date_expression element may optionally contain a date_spec or duration element depending on the context.
Table 8.4. Attributes of a date_expression Element
| Field | Description |
|---|---|
id
| |
start
|
A date/time conforming to the ISO8601 specification. May be used when
operation is in_range (in which case at least one of start or end must be specified) or gt (in which case start is required).
|
end
|
A date/time conforming to the ISO8601 specification. May be used when
operation is in_range (in which case at least one of start or end must be specified) or lt (in which case end is required).
|
operation
|
Compares the current date/time with the start and/or end date, depending on the context. Allowed values:
|
Note
eq, neq, gte, or lte operation, since they would be valid only for a single second.
date_spec element is used to create a cron-like expression relating to time. Each field can contain a single number or range. Any field not supplied is ignored.
Table 8.5. Attributes of a date_spec Element
| Field | Description |
|---|---|
id
| |
hours
| |
monthdays
| |
weekdays
| |
yeardays
| |
months
| |
weeks
| |
years
| |
weekyears
| |
moon
|
monthdays="1" matches the first day of every month, and hours="09-17" matches the hours between 9 a.m. and 5 p.m. (inclusive).
weekdays="1,2" or weekdays="1-2,5-6") are not supported.
Note
date_expression with an operation of gt, lt, or in_range will next change, and schedule a cluster re-check for that time. However, it does not do this for date_spec. Instead, it evaluates the date_spec whenever a cluster re-check naturally happens via a cluster event or the cluster-recheck-interval cluster option. For example, if you have a date_spec enabling a resource from 9 a.m. to 5 p.m., and cluster-recheck-interval has been set to 5 minutes, then sometime between 9 a.m. and 9:05 a.m. the cluster would notice that it needs to start the resource, and sometime between 5 p.m. and 5:05 p.m. it would realize that it needs to stop the resource. The timing of the actual start and stop actions will further depend on factors such as any other actions the cluster may need to perform first, and the load of the machine.
duration is used to calculate a value for end when one is not supplied to in_range operations. It contains one or more attributes each containing a single number. Any attribute not supplied is ignored.
Table 8.6. Attributes of a duration Element
| Field | Description |
|---|---|
id
| |
seconds
| |
minutes
| |
hours
| |
weeks
| |
months
| |
years
|
Example 8.1. True if now is any time in the year 2005
<rule id="rule1" score="INFINITY"> <date_expression id="date_expr1" start="2005-001" operation="in_range"> <duration id="duration1" years="1"/> </date_expression> </rule>
Example 8.2. Equivalent expression
<rule id="rule2" score="INFINITY"> <date_expression id="date_expr2" operation="date_spec"> <date_spec id="date_spec2" years="2005"/> </date_expression> </rule>
Example 8.3. 9am-5pm Monday-Friday
<rule id="rule3" score="INFINITY"> <date_expression id="date_expr3" operation="date_spec"> <date_spec id="date_spec3" hours="9-16" weekdays="1-5"/> </date_expression> </rule>
16 matches up to 16:59:59, as the numeric value (hour) still matches!
Example 8.4. 9am-6pm Monday through Friday or anytime Saturday
<rule id="rule4" score="INFINITY" boolean-op="or"> <date_expression id="date_expr4-1" operation="date_spec"> <date_spec id="date_spec4-1" hours="9-16" weekdays="1-5"/> </date_expression> <date_expression id="date_expr4-2" operation="date_spec"> <date_spec id="date_spec4-2" weekdays="6"/> </date_expression> </rule>
Example 8.5. 9am-5pm or 9pm-12am Monday through Friday
<rule id="rule5" score="INFINITY" boolean-op="and"> <rule id="rule5-nested1" score="INFINITY" boolean-op="or"> <date_expression id="date_expr5-1" operation="date_spec"> <date_spec id="date_spec5-1" hours="9-16"/> </date_expression> <date_expression id="date_expr5-2" operation="date_spec"> <date_spec id="date_spec5-2" hours="21-23"/> </date_expression> </rule> <date_expression id="date_expr5-3" operation="date_spec"> <date_spec id="date_spec5-3" weekdays="1-5"/> </date_expression> </rule>
Example 8.6. Mondays in March 2005
<rule id="rule6" score="INFINITY" boolean-op="and"> <date_expression id="date_expr6-1" operation="date_spec"> <date_spec id="date_spec6" weekdays="1"/> </date_expression> <date_expression id="date_expr6-2" operation="in_range" start="2005-03-01" end="2005-04-01"/> </rule>
Note
end="2005-03-31T23:59:59" to avoid confusion.
Example 8.7. A full moon on Friday the 13th
<rule id="rule7" score="INFINITY" boolean-op="and"> <date_expression id="date_expr7" operation="date_spec"> <date_spec id="date_spec7" weekdays="5" monthdays="13" moon="4"/> </date_expression> </rule>
rsc_expression is a rule condition based on a resource agent’s properties. This rule is only valid within an rsc_defaults or op_defaults context. None of the matching attributes of class, provider, and type are required. If one is omitted, all values of that attribute will match. For instance, omitting type means every type will match.
Table 8.7. Attributes of an rsc_expression Element
Example 8.8. True for all ocf:heartbeat:IPaddr2 resources
<rule id="rule1" score="INFINITY"> <rsc_expression id="rule_expr1" class="ocf" provider="heartbeat" type="IPaddr2"/> </rule>
Example 8.9. Provider doesn’t apply to non-OCF resources
<rule id="rule2" score="INFINITY"> <rsc_expression id="rule_expr2" class="stonith" type="fence_xvm"/> </rule>
op_expression is a rule condition based on an action of some resource agent. This rule is only valid within an op_defaults context.
Example 8.10. True for all monitor actions
<rule id="rule1" score="INFINITY"> <op_expression id="rule_expr1" name="monitor"/> </rule>
Example 8.11. True for all monitor actions with a 10 second interval
<rule id="rule2" score="INFINITY"> <op_expression id="rule_expr2" name="monitor" interval="10s"/> </rule>
Example 8.12. Prevent resource "webserver" from running on node3
<rsc_location id="ban-apache-on-node3" rsc="webserver" score="-INFINITY" node="node3"/>
Example 8.13. Prevent resource "webserver" from running on node3 using rule
<rsc_location id="ban-apache-on-node3" rsc="webserver"> <rule id="ban-apache-rule" score="-INFINITY"> <expression id="ban-apache-expr" attribute="#uname" operation="eq" value="node3"/> </rule> </rsc_location>
Example 8.14. A sample nodes section for use with score-attribute
<nodes> <node id="uuid1" uname="c001n01" type="normal"> <instance_attributes id="uuid1-custom_attrs"> <nvpair id="uuid1-cpu_mips" name="cpu_mips" value="1234"/> </instance_attributes> </node> <node id="uuid2" uname="c001n02" type="normal"> <instance_attributes id="uuid2-custom_attrs"> <nvpair id="uuid2-cpu_mips" name="cpu_mips" value="5678"/> </instance_attributes> </node> </nodes>
<rule id="need-more-power-rule" score="-INFINITY"> <expression id="need-more-power-expr" attribute="cpu_mips" operation="lt" value="3000"/> </rule>
score-attribute Instead of scorescore-attribute instead of score, each node matched by the rule has its score adjusted differently, according to its value for the named node attribute. Thus, in the previous example, if a rule used score-attribute="cpu_mips", c001n01 would have its preference to run the resource increased by 1234 whereas c001n02 would have its preference increased by 5678.
cluster_property_set elements)
utilization, meta_attributes, or instance_attributes elements inside a resource definition element or op , rsc_defaults, op_defaults, or template element)
instance_attributes objects for the resource and adding a rule to each, we can easily handle these special cases.
mySpecialRsc will use eth1 and port 9999 when run on node1, eth2 and port 8888 on node2 and default to eth0 and port 9999 for all other nodes.
Example 8.15. Defining different resource options based on the node name
<primitive id="mySpecialRsc" class="ocf" type="Special" provider="me"> <instance_attributes id="special-node1" score="3"> <rule id="node1-special-case" score="INFINITY" > <expression id="node1-special-case-expr" attribute="#uname" operation="eq" value="node1"/> </rule> <nvpair id="node1-interface" name="interface" value="eth1"/> </instance_attributes> <instance_attributes id="special-node2" score="2" > <rule id="node2-special-case" score="INFINITY"> <expression id="node2-special-case-expr" attribute="#uname" operation="eq" value="node2"/> </rule> <nvpair id="node2-interface" name="interface" value="eth2"/> <nvpair id="node2-port" name="port" value="8888"/> </instance_attributes> <instance_attributes id="defaults" score="1" > <nvpair id="default-interface" name="interface" value="eth0"/> <nvpair id="default-port" name="port" value="9999"/> </instance_attributes> </primitive>
instance_attributes objects are evaluated is determined by their score (highest to lowest). If not supplied, score defaults to zero, and objects with an equal score are processed in listed order. If the instance_attributes object has no rule or a rule that evaluates to true, then for any parameter the resource does not yet have a value for, the resource will use the parameter values defined by the instance_attributes.
special-node1 has the highest score (3) and so is evaluated first; its rule evaluates to true, so interface is set to eth1.
special-node2 is evaluated next with score 2, but its rule evaluates to false, so it is ignored.
defaults is evaluated last with score 1, and has no rule, so its values are examined; interface is already defined, so the value here is not used, but port is not yet defined, so port is set to 9999.
resource-stickiness value during and outside work hours. This allows resources to automatically move back to their most preferred hosts, but at a time that (in theory) does not interfere with business activities.
Example 8.16. Change resource-stickiness during working hours
<rsc_defaults> <meta_attributes id="core-hours" score="2"> <rule id="core-hour-rule" score="0"> <date_expression id="nine-to-five-Mon-to-Fri" operation="date_spec"> <date_spec id="nine-to-five-Mon-to-Fri-spec" hours="9-16" weekdays="1-5"/> </date_expression> </rule> <nvpair id="core-stickiness" name="resource-stickiness" value="INFINITY"/> </meta_attributes> <meta_attributes id="after-hours" score="1" > <nvpair id="after-stickiness" name="resource-stickiness" value="0"/> </meta_attributes> </rsc_defaults>
instance_attributes or utilization blocks.
rsc_expression and op_expression blocks may additionally be used to set defaults on either a single resource or across an entire class of resources with a single rule. rsc_expression may be used to select resource agents within both rsc_defaults and op_defaults, while op_expression may only be used within op_defaults. If multiple rules succeed for a given resource agent, the last one specified will be the one that takes effect. As with any other rule, boolean operations may be used to make more complicated expressions.
Example 8.17. Set all IPaddr2 resources to stopped
<rsc_defaults> <meta_attributes id="op-target-role"> <rule id="op-target-role-rule" score="INFINITY"> <rsc_expression id="op-target-role-expr" class="ocf" provider="heartbeat" type="IPaddr2"/> </rule> <nvpair id="op-target-role-nvpair" name="target-role" value="Stopped"/> </meta_attributes> </rsc_defaults>
Example 8.18. Set all monitor action timeouts to 7 seconds
<op_defaults> <meta_attributes id="op-monitor-defaults"> <rule id="op-monitor-default-rule" score="INFINITY"> <op_expression id="op-monitor-default-expr" name="monitor"/> </rule> <nvpair id="op-monitor-timeout" name="timeout" value="7s"/> </meta_attributes> </op_defaults>
Example 8.19. Set the monitor action timeout on all IPaddr2 resources with a given monitor interval to 8 seconds
<op_defaults> <meta_attributes id="op-monitor-and"> <rule id="op-monitor-and-rule" score="INFINITY"> <rsc_expression id="op-monitor-and-rsc-expr" class="ocf" provider="heartbeat" type="IPaddr2"/> <op_expression id="op-monitor-and-op-expr" name="monitor" interval="10s"/> </rule> <nvpair id="op-monitor-and-timeout" name="timeout" value="8s"/> </meta_attributes> </op_defaults>
maintenance_mode during a scheduled maintenance window. This will keep the cluster running but not monitor, start, or stop resources during this time.
Example 8.20. Schedule a maintenance window for 9 to 11 p.m. CDT Sept. 20, 2019
<crm_config> <cluster_property_set id="cib-bootstrap-options"> <nvpair id="bootstrap-stonith-enabled" name="stonith-enabled" value="1"/> </cluster_property_set> <cluster_property_set id="normal-set" score="10"> <nvpair id="normal-maintenance-mode" name="maintenance-mode" value="false"/> </cluster_property_set> <cluster_property_set id="maintenance-window-set" score="1000"> <nvpair id="maintenance-nvpair1" name="maintenance-mode" value="true"/> <rule id="maintenance-rule1" score="INFINITY"> <date_expression id="maintenance-date1" operation="in_range" start="2019-09-20 21:00:00 -05:00" end="2019-09-20 23:00:00 -05:00"/> </rule> </cluster_property_set> </crm_config>
Important
cluster_property_set with an id set to "cib-bootstrap-options" will always have the highest priority, regardless of any scores. Therefore, rules in another cluster_property_set can never take effect for any properties listed in the bootstrap set.
Table of Contents
interval-origin. The cluster uses this point to calculate the correct start-delay such that the operation will occur at origin + (interval * N).
interval and interval-origin can be any date/time conforming to the ISO8601 standard. By way of example, to specify an operation that would run on the first Monday of 2009 and every Monday after that, you would add:
Example 9.1. Specifying a Base for Recurring Action Intervals
<op id="my-weekly-action" name="custom-action" interval="P7D" interval-origin="2009-W01-1"/>
crm_failcount command. For example, to see how many times the 10-second monitor for myrsc has failed on node1, run:
# crm_failcount --query -r myrsc -N node1 -n monitor -I 10s
crm_failcount will use the local node. If you omit the operation and interval, crm_failcount will display the sum of the fail counts for all operations on the resource.
crm_resource --cleanup or crm_failcount --delete to clear fail counts. For example, to clear the above monitor failures, run:
# crm_resource --cleanup -r myrsc -N node1 -n monitor -I 10s
crm_resource --cleanup will clear failures for all resources. If you omit the node, it will clear failures on all nodes. If you omit the operation and interval, it will clear the failures for all operations on the resource.
Note
crm_mon tool shows the current cluster status, including any failed operations. To see the current fail counts for any failed resources, call crm_mon with the --failcounts option. This shows the fail counts per resource (that is, the sum of any operation fail counts for the resource).
migration-threshold resource meta-attribute.
[13]
migration-threshold=N for a resource, it will be banned from the original node after N failures.
Note
migration-threshold is per resource, even though fail counts are tracked per operation. The operation fail counts are added together to compare against the migration-threshold.
crm_resource --cleanup or crm_failcount --delete (hopefully after first fixing the failure’s cause). It is possible to have fail counts expire automatically by setting the failure-timeout resource meta-attribute.
Important
migration-threshold=2 and failure-timeout=60s would cause the resource to move to a new node after 2 failures, and allow it to move back (depending on stickiness and constraint scores) after one minute.
Note
failure-timeout is measured since the most recent failure. That is, older failures do not individually time out and lower the fail count. Instead, all failures are timed out simultaneously (and the fail count is reset to 0) if there is no new failure for the timeout period.
start-failure-is-fatal is set to true (which is the default), start failures cause the fail count to be set to INFINITY and thus always cause the resource to move immediately.
crm_standby. To check the standby status of the current machine, run:
# crm_standby -G
on indicates that the node is not able to host any resources, while a value of off says that it can.
--node option:
# crm_standby -G --node sles-2
-v instead of -G:
# crm_standby -v on
--node.
crm_resource command, which creates and modifies the extra constraints for you. If Email were running on sles-1 and you wanted it moved to a specific location, the command would look something like:
# crm_resource -M -r Email -H sles-2
<rsc_location rsc="Email" node="sles-2" score="INFINITY"/>
crm_resource -M are not cumulative. So, if you ran these commands
# crm_resource -M -r Email -H sles-2 # crm_resource -M -r Email -H sles-3
# crm_resource -U -r Email
resource-stickiness, it might stay where it is. To be absolutely certain that it moves back to sles-1, move it there before issuing the call to crm_resource -U:
# crm_resource -M -r Email -H sles-1 # crm_resource -U -r Email
# crm_resource -B -r Email
<rsc_location rsc="Email" node="sles-1" score="-INFINITY"/>
-INFINITY constraint will prevent the resource from running on that node until crm_resource -U is used. This includes the situation where every other cluster node is no longer available!
resource-stickiness is set to INFINITY, it is possible that you will end up with the problem described in Section 5.2.4, “What if Two Nodes Have the Same Score”. The tool can detect some of these cases and deals with them by creating both positive and negative constraints. E.g.
Email prefers sles-1 with a score of -INFINITY
Email prefers sles-2 with a score of INFINITY
pingd by default.
[14]
Note
Example 9.2. An example ping cluster resource that checks node connectivity once every minute
<clone id="Connected"> <primitive id="ping" provider="pacemaker" class="ocf" type="ping"> <instance_attributes id="ping-attrs"> <nvpair id="pingd-dampen" name="dampen" value="5s"/> <nvpair id="pingd-multiplier" name="multiplier" value="1000"/> <nvpair id="pingd-hosts" name="host_list" value="my.gateway.com www.bigcorp.com"/> </instance_attributes> <operations> <op id="ping-monitor-60s" interval="60s" name="monitor"/> </operations> </primitive> </clone>
Important
ocf:pacemaker:ping is recording.
Important
Example 9.3. Don’t run a resource on unconnected nodes
<rsc_location id="WebServer-no-connectivity" rsc="Webserver"> <rule id="ping-exclude-rule" score="-INFINITY" > <expression id="ping-exclude" attribute="pingd" operation="not_defined"/> </rule> </rsc_location>
Example 9.4. Run only on nodes connected to three or more ping targets.
<primitive id="ping" provider="pacemaker" class="ocf" type="ping"> ... <!-- omitting some configuration to highlight important parts --> <nvpair id="pingd-multiplier" name="multiplier" value="1000"/> ... </primitive> ... <rsc_location id="WebServer-connectivity" rsc="Webserver"> <rule id="ping-prefer-rule" score="-INFINITY" > <expression id="ping-prefer" attribute="pingd" operation="lt" value="3000"/> </rule> </rsc_location>
multiplier to a value higher than that of resource-stickiness (and don’t set either of them to INFINITY).
Example 9.5. Prefer the node with the most connected ping nodes
<rsc_location id="WebServer-connectivity" rsc="Webserver"> <rule id="ping-prefer-rule" score-attribute="pingd" > <expression id="ping-prefer" attribute="pingd" operation="defined"/> </rule> </rsc_location>
Example 9.6. How the cluster translates the above location constraint
<rsc_location id="ping-1" rsc="Webserver" node="sles-1" score="5000"/> <rsc_location id="ping-2" rsc="Webserver" node="sles-2" score="2000"/>
multiplier is set to 1000).
Example 9.7. A more complex example of choosing a location based on connectivity
<rsc_location id="WebServer-connectivity" rsc="Webserver"> <rule id="ping-exclude-rule" score="-INFINITY" > <expression id="ping-exclude" attribute="pingd" operation="lt" value="3000"/> </rule> <rule id="ping-prefer-rule" score-attribute="pingd" > <expression id="ping-prefer" attribute="pingd" operation="defined"/> </rule> </rsc_location>
migrate_to (performed on the current location) and migrate_from (performed on the destination).
migrate_to action and, if anything, the activation would occur during migrate_from.
migrate_to action is practically empty and migrate_from does most of the work, extracting the relevant resource state from the old location and activating it.
Migration Checklist
migrate_to and migrate_from actions, and advertise them in its metadata.
allow-migrate meta-attribute set to true (which is not the default).
#health as an indicator of node health. Node health attributes may have one of the following values:
Table 9.2. Allowed Values for Node Health Attributes
node-health-strategy cluster option controls how Pacemaker responds to changes in node health attributes, and how it translates red, yellow, and green to scores.
Table 9.3. Node Health Strategies
| Value | Effect |
|---|---|
none
| |
migrate-on-red
| |
only-green
| |
progressive
|
Assign the value of the
node-health-red cluster option to red, the value of node-health-yellow to yellow, and the value of node-health-green to green. Each node is additionally assigned a score of node-health-base (this allows resources to start even if some attributes are yellow). This strategy gives the administrator finer control over how important each value is.
|
custom
|
Track node health attributes using the same values as
progressive for red, yellow, and green, but do not take them into account. The administrator is expected to implement a policy by defining rules (see Chapter 8, Rules) referencing node health attributes.
|
ocf:pacemaker:HealthCPU and ocf:pacemaker:HealthSMART resource agents set node health attributes based on CPU and disk parameters. The ipmiservicelogd daemon sets node health attributes based on IPMI values (the ocf:pacemaker:SystemHealth resource agent can be used to manage the daemon as a cluster resource).
<clone id="resHealthIOWait-clone"> <primitive class="ocf" id="HealthIOWait" provider="pacemaker" type="HealthIOWait"> <instance_attributes id="resHealthIOWait-instance_attributes"> <nvpair id="resHealthIOWait-instance_attributes-red_limit" name="red_limit" value="30"/> <nvpair id="resHealthIOWait-instance_attributes-yellow_limit" name="yellow_limit" value="10"/> </instance_attributes> <operations> <op id="resHealthIOWait-monitor-interval-5" interval="5" name="monitor" timeout="5"/> <op id="resHealthIOWait-start-interval-0s" interval="0s" name="start" timeout="10s"/> <op id="resHealthIOWait-stop-interval-0s" interval="0s" name="stop" timeout="10s"/> </operations> </primitive> </clone>
# attrd_updater -n "#health-iowait" -U "green" --node="<nodename>" -d "60s"
reload operation and perform any required actions. The actions here depend completely on your application!
Example 9.8. The DRBD agent’s logic for supporting reload
case $1 in start) drbd_start ;; stop) drbd_stop ;; reload) drbd_reload ;; monitor) drbd_monitor ;; *) drbd_usage exit $OCF_ERR_UNIMPLEMENTED ;; esac exit $?
reload operation in the actions section of its metadata
Example 9.9. The DRBD Agent Advertising Support for the reload Operation
<?xml version="1.0"?> <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd"> <resource-agent name="drbd"> <version>1.1</version> <longdesc> Master/Slave OCF Resource Agent for DRBD </longdesc> ... <actions> <action name="start" timeout="240" /> <action name="reload" timeout="240" /> <action name="promote" timeout="90" /> <action name="demote" timeout="90" /> <action name="notify" timeout="90" /> <action name="stop" timeout="100" /> <action name="meta-data" timeout="5" /> <action name="validate-all" timeout="30" /> </actions> </resource-agent>
reload.
unique set to 0 is eligible to be used in this way.
Example 9.10. Parameter that can be changed using reload
<parameter name="drbdconf" unique="0"> <longdesc>Full path to the drbd.conf file.</longdesc> <shortdesc>Path to drbd.conf</shortdesc> <content type="string" default="${OCF_RESKEY_drbdconf_default}"/> </parameter>
Note
unique=0.
Note
Table of Contents
Example 10.1. A group of two primitive resources
<group id="shortcut"> <primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat"> <instance_attributes id="params-public-ip"> <nvpair id="public-ip-addr" name="ip" value="192.0.2.2"/> </instance_attributes> </primitive> <primitive id="Email" class="lsb" type="exim"/> </group>
Public-IP first, then Email)
Email first, then Public-IP)
Public-IP can’t run anywhere, neither can Email;
Email can’t run anywhere, this does not affect Public-IP in any way
Example 10.2. How the cluster sees a group resource
<configuration> <resources> <primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat"> <instance_attributes id="params-public-ip"> <nvpair id="public-ip-addr" name="ip" value="192.0.2.2"/> </instance_attributes> </primitive> <primitive id="Email" class="lsb" type="exim"/> </resources> <constraints> <rsc_colocation id="xxx" rsc="Email" with-rsc="Public-IP" score="INFINITY"/> <rsc_order id="yyy" first="Public-IP" then="Email"/> </constraints> </configuration>
priority, target-role, and is-managed properties from primitive resources. See Section 4.4, “Resource Options” for information about those properties.
id instead of the group’s.
Example 10.3. Some constraints involving groups
<constraints> <rsc_location id="group-prefers-node1" rsc="shortcut" node="node1" score="500"/> <rsc_colocation id="webserver-with-group" rsc="Webserver" with-rsc="shortcut"/> <rsc_order id="start-group-then-webserver" first="Webserver" then="shortcut"/> </constraints>
resource-stickiness is 100, and a group has seven members, five of which are active, then the group as a whole will prefer its current location with a score of 500.
promote and demote actions of the resource agent.
promote and demote actions to manage entering and exiting the special role.
priority, target-role, is-managed
Table 10.3. Clone-specific configuration options
| Field | Default | Description |
|---|---|---|
globally-unique
|
false
| |
clone-max
|
number of nodes in cluster
| |
clone-node-max
|
1
| |
clone-min
|
0
| |
notify
|
false
| |
ordered
|
false
| |
interleave
|
false
| |
promotable
|
false
| |
promoted-max
|
1
| |
promoted-node-max
|
1
|
master-max and master-node-max are accepted as aliases for promoted-max and promoted-node-max, but are deprecated since 2.0.0, and support for them will be removed in a future version.
Example 10.4. A clone that runs a web server on all nodes
<clone id="apache-clone"> <primitive id="apache" class="lsb" type="apache"> <operations> <op id="apache-monitor" name="monitor" interval="30"/> </operations> </primitive> </clone>
Warning
id is used.
Example 10.5. Some constraints involving clones
<constraints> <rsc_location id="clone-prefers-node1" rsc="apache-clone" node="node1" score="500"/> <rsc_colocation id="stats-with-clone" rsc="apache-stats" with="apache-clone"/> <rsc_order id="start-clone-then-stats" first="apache-clone" then="apache-stats"/> </constraints>
apache-stats will wait until all copies of apache-clone that need to be started have done so before being started itself. Only if no copies can be started will apache-stats be prevented from being active. Additionally, the clone will wait for apache-stats to be stopped before stopping itself.
A is colocated with another clone B, the set of allowed locations for A is limited to nodes on which B is (or will be) active. Placement is then performed normally.
first-action and/or then-action fields for ordering constraints may be set to promote or demote to constrain the master role, and colocation constraints may contain rsc-role and/or with-rsc-role fields.
Table 10.4. Additional colocation constraint options for promotable clone resources
| Field | Default | Description |
|---|---|---|
rsc-role
|
Started
| |
with-rsc-role
|
Started
|
Example 10.6. Constraints involving promotable clone resources
<constraints> <rsc_location id="db-prefers-node1" rsc="database" node="node1" score="500"/> <rsc_colocation id="backup-with-db-slave" rsc="backup" with-rsc="database" with-rsc-role="Slave"/> <rsc_colocation id="myapp-with-db-master" rsc="myApp" with-rsc="database" with-rsc-role="Master"/> <rsc_order id="start-db-before-backup" first="database" then="backup"/> <rsc_order id="promote-db-then-app" first="database" first-action="promote" then="myApp" then-action="start"/> </constraints>
myApp will wait until one of the database copies has been started and promoted to master before being started itself on the same node. Only if no copies can be promoted will myApp be prevented from being active. Additionally, the cluster will wait for myApp to be stopped before demoting the database.
master or slave). In the example above, the cluster will choose a location based on where database is running as a master, and if there are multiple master instances it will also factor in myApp's own location preferences when deciding which location to choose.
rsc clone is (after role filtering) limited to nodes on which the with-rsc promotable clone resource is (or will be) in the specified role. Placement is then performed as normal.
Table 10.5. Additional colocation set options relevant to promotable clone resources
B's master must be located on the same node as A's master. Additionally resources C and D must be located on the same node as A's and B's masters.
Example 10.7. Colocate C and D with A’s and B’s master instances
<constraints> <rsc_colocation id="coloc-1" score="INFINITY" > <resource_set id="colocated-set-example-1" sequential="true" role="Master"> <resource_ref id="A"/> <resource_ref id="B"/> </resource_set> <resource_set id="colocated-set-example-2" sequential="true"> <resource_ref id="C"/> <resource_ref id="D"/> </resource_set> </rsc_colocation> </constraints>
Table 10.6. Additional ordered set options relevant to promotable clone resources
Example 10.8. Start C and D after first promoting A and B
<constraints> <rsc_order id="order-1" score="INFINITY" > <resource_set id="ordered-set-1" sequential="true" action="promote"> <resource_ref id="A"/> <resource_ref id="B"/> </resource_set> <resource_set id="ordered-set-2" sequential="true" action="start"> <resource_ref id="C"/> <resource_ref id="D"/> </resource_set> </rsc_order> </constraints>
B cannot be promoted to a master role until A has been promoted. Additionally, resources C and D must wait until A and B have been promoted before they can start.
resource-stickiness is provided, the clone will use a value of 1. Being a small value, it causes minimal disturbance to the score calculations of other resources but is enough to prevent Pacemaker from needlessly moving copies around the cluster.
Note
resource-stickiness of 0 for the clone temporarily and let the cluster adjust, then set it back to 1 if you want the default behavior to apply again.
Important
resource-stickiness is set in the rsc_defaults section, it will apply to clone instances as well. This means an explicit resource-stickiness of 0 in rsc_defaults works differently from the implicit default used when resource-stickiness is not specified.
${OCF_SUCCESS} if the node has that exact instance active. All other probes for instances of the clone should result in ${OCF_NOT_RUNNING} (or one of the other OCF error codes if they are failed).
apache:2.
OCF_RESKEY_CRM_meta_clone_max environment variable and which instance it is by examining OCF_RESKEY_CRM_meta_clone.
OCF_RESKEY_CRM_meta_clone) about which numerical instances are active. In particular, the list of active copies will not always be an unbroken sequence, nor always start at 0.
demote and promote, which are responsible for changing the state of the resource. Like start and stop, they should return ${OCF_SUCCESS} if they completed successfully or a relevant error code if they did not.
slave. From there the cluster will decide which instances to promote to master.
Table 10.7. Role implications of OCF return codes
notify meta-attribute set to true, and the resource agent supports the notify action, Pacemaker will call the action when appropriate, passing a number of extra variables which, when combined with additional context, can be used to calculate the current state of the cluster and what is about to happen to it.
Table 10.8. Environment variables supplied with Clone notify actions
| Variable | Description |
|---|---|
|
OCF_RESKEY_CRM_meta_notify_type
| |
|
OCF_RESKEY_CRM_meta_notify_operation
| |
|
OCF_RESKEY_CRM_meta_notify_start_resource
| |
|
OCF_RESKEY_CRM_meta_notify_stop_resource
| |
|
OCF_RESKEY_CRM_meta_notify_active_resource
| |
|
OCF_RESKEY_CRM_meta_notify_inactive_resource
| |
|
OCF_RESKEY_CRM_meta_notify_start_uname
| |
|
OCF_RESKEY_CRM_meta_notify_stop_uname
| |
|
OCF_RESKEY_CRM_meta_notify_active_uname
|
OCF_RESKEY_CRM_meta_notify_start_resource and OCF_RESKEY_CRM_meta_notify_start_uname, and should be treated as an array of whitespace-separated elements.
OCF_RESKEY_CRM_meta_notify_inactive_resource is an exception, as the matching uname variable does not exist since inactive resources are not running on any node.
clone:0 will be started on sles-1, clone:2 will be started on sles-3, and clone:3 will be started on sles-2, the cluster would set:
Example 10.9. Notification variables
OCF_RESKEY_CRM_meta_notify_start_resource="clone:0 clone:2 clone:3" OCF_RESKEY_CRM_meta_notify_start_uname="sles-1 sles-3 sles-2"
Note
Pre-notification (stop):
$OCF_RESKEY_CRM_meta_notify_active_resource
$OCF_RESKEY_CRM_meta_notify_inactive_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
Post-notification (stop) / Pre-notification (start):
$OCF_RESKEY_CRM_meta_notify_active_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_inactive_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
Post-notification (start):
$OCF_RESKEY_CRM_meta_notify_active_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_inactive_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
Table 10.9. Extra environment variables supplied for promotable clones
| Variable | Description |
|---|---|
|
OCF_RESKEY_CRM_meta_notify_master_resource
| |
|
OCF_RESKEY_CRM_meta_notify_slave_resource
| |
|
OCF_RESKEY_CRM_meta_notify_promote_resource
| |
|
OCF_RESKEY_CRM_meta_notify_demote_resource
| |
|
OCF_RESKEY_CRM_meta_notify_promote_uname
| |
|
OCF_RESKEY_CRM_meta_notify_demote_uname
| |
|
OCF_RESKEY_CRM_meta_notify_master_uname
| |
|
OCF_RESKEY_CRM_meta_notify_slave_uname
|
Pre-notification (demote):
Active resources: $OCF_RESKEY_CRM_meta_notify_active_resource
Master resources: $OCF_RESKEY_CRM_meta_notify_master_resource
Slave resources: $OCF_RESKEY_CRM_meta_notify_slave_resource
$OCF_RESKEY_CRM_meta_notify_inactive_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_promote_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
Post-notification (demote) / Pre-notification (stop):
Active resources: $OCF_RESKEY_CRM_meta_notify_active_resource
Master resources:
$OCF_RESKEY_CRM_meta_notify_master_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
Slave resources: $OCF_RESKEY_CRM_meta_notify_slave_resource
$OCF_RESKEY_CRM_meta_notify_inactive_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_promote_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
Post-notification (stop) / Pre-notification (start)
Active resources:
$OCF_RESKEY_CRM_meta_notify_active_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
Master resources:
$OCF_RESKEY_CRM_meta_notify_master_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
Slave resources:
$OCF_RESKEY_CRM_meta_notify_slave_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_inactive_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_promote_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
Post-notification (start) / Pre-notification (promote)
Active resources:
$OCF_RESKEY_CRM_meta_notify_active_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
Master resources:
$OCF_RESKEY_CRM_meta_notify_master_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
Slave resources:
$OCF_RESKEY_CRM_meta_notify_slave_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_inactive_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_promote_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
Post-notification (promote)
Active resources:
$OCF_RESKEY_CRM_meta_notify_active_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
Master resources:
$OCF_RESKEY_CRM_meta_notify_master_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
$OCF_RESKEY_CRM_meta_notify_promote_resource
Slave resources:
$OCF_RESKEY_CRM_meta_notify_slave_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_promote_resource
$OCF_RESKEY_CRM_meta_notify_inactive_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_promote_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
$OCF_RESKEY_CRM_meta_notify_start_resource
$OCF_RESKEY_CRM_meta_notify_promote_resource
$OCF_RESKEY_CRM_meta_notify_demote_resource
$OCF_RESKEY_CRM_meta_notify_stop_resource
role="master" will cover the master role.
Example 10.10. Monitoring both states of a promotable clone resource
<clone id="myMasterRsc"> <meta_attributes id="myMasterRsc-meta"> <nvpair name="promotable" value="true"/> </meta_attributes> <primitive id="myRsc" class="ocf" type="myApp" provider="myCorp"> <operations> <op id="public-ip-slave-check" name="monitor" interval="60"/> <op id="public-ip-master-check" name="monitor" interval="61" role="Master"/> </operations> </primitive> </clone>
Important
crm_master utility, which generally would be called by the resource agent’s start action if it supports promotable clones. This tool automatically detects both the resource and host, and should be used to set a preference for being promoted. Based on this, promoted-max, and promoted-node-max, the instance(s) with the highest preference will be promoted.
Example 10.11. Explicitly preferring node1 to be promoted to master
<rsc_location id="master-location" rsc="myMasterRsc"> <rule id="master-rule" score="100" role="Master"> <expression id="master-exp" attribute="#uname" operation="eq" value="node1"/> </rule> </rsc_location>
Example 10.12. A bundle for a containerized web server
<bundle id="httpd-bundle"> <podman image="pcmk:http" replicas="3"/> <network ip-range-start="192.168.122.131" host-netmask="24" host-interface="eth0"> <port-mapping id="httpd-port" port="80"/> </network> <storage> <storage-mapping id="httpd-syslog" source-dir="/dev/log" target-dir="/dev/log" options="rw"/> <storage-mapping id="httpd-root" source-dir="/srv/html" target-dir="/var/www/html" options="rw,Z"/> <storage-mapping id="httpd-logs" source-dir-root="/var/log/pacemaker/bundles" target-dir="/etc/httpd/logs" options="rw,Z"/> </storage> <primitive class="ocf" id="httpd" provider="heartbeat" type="apache"/> </bundle>
ocf:heartbeat:docker, ocf:heartbeat:podman, or ocf:heartbeat:rkt to manage a bundle’s container. The user must ensure that the appropriate resource agent is installed on every node allowed to run the bundle.
Table 10.10. XML Attributes of a bundle Element
docker, podman, or rkt element.
Table 10.11. XML attributes of a docker, podman, or rkt Element
| Attribute | Default | Description |
|---|---|---|
image
|
| |
replicas
|
Value of
promoted-max if that is positive, else 1
| |
replicas-per-host
|
1
| |
promoted-max
|
0
| |
network
|
| |
run-command
|
/usr/sbin/pacemaker-remoted if bundle contains a primitive, otherwise none
| |
options
|
|
Note
run-command to /usr/sbin/pacemaker_remoted (note the underbar instead of dash).
masters is accepted as an alias for promoted-max, but is deprecated since 2.0.0, and support for it will be removed in a future version.
Table 10.12. XML attributes of a network Element
| Attribute | Default | Description |
|---|---|---|
add-host
|
TRUE
|
If TRUE, and
ip-range-start is used, Pacemaker will automatically ensure that /etc/hosts inside the containers has entries for each replica name and its assigned IP.
|
ip-range-start
|
|
If specified, Pacemaker will create an implicit
ocf:heartbeat:IPaddr2 resource for each container instance, starting with this IP address, using up to replicas sequential addresses. These addresses can be used from the host’s network to reach the service inside the container, though it is not visible within the container itself. Only IPv4 addresses are currently supported.
|
host-netmask
|
32
| |
host-interface
|
| |
control-port
|
3121
|
If the bundle contains a
primitive, the cluster will use this integer TCP port for communication with Pacemaker Remote inside the container. Changing this is useful when the container is unable to listen on the default port, for example, when the container uses the host’s network rather than ip-range-start (in which case replicas-per-host must be 1), or when the bundle may run on a Pacemaker Remote node that is already listening on the default port. Any PCMK_remote_port environment variable set on the host or in the container is ignored for bundle connections.
|
Note
httpd-bundle has replicas=2, its containers will be named httpd-bundle-0 and httpd-bundle-1.
Table 10.13. Attributes of a port-mapping Element
Note
primitive, Pacemaker will automatically map the control-port, so it is not necessary to specify that port in a port-mapping.
storage element. A storage element has no properties of its own, but may contain one or more storage-mapping elements.
Table 10.14. Attributes of a storage-mapping Element
| Attribute | Default | Description |
|---|---|---|
id
|
| |
source-dir
|
| |
source-dir-root
|
|
The start of a path on the host’s filesystem that will be mapped into the container, using a different subdirectory on the host for each container instance. The subdirectory will be named the same as the replica name. Exactly one of
source-dir and source-dir-root must be specified in a storage-mapping.
|
target-dir
|
| |
options
|
|
Note
Note
primitive, Pacemaker will automatically map the equivalent of source-dir=/etc/pacemaker/authkey target-dir=/etc/pacemaker/authkey and source-dir-root=/var/log/pacemaker/bundles target-dir=/var/log into the container, so it is not necessary to specify those paths in a storage-mapping.
Important
PCMK_authkey_location environment variable must not be set to anything other than the default of /etc/pacemaker/authkey on any node in the cluster.
Important
ip-range-start or control-port must be configured in the bundle. Pacemaker will create an implicit ocf:pacemaker:remote resource for the connection, launch Pacemaker Remote within the container, and monitor and manage the primitive resource via Pacemaker Remote.
masters greater than zero.
Note
options attribute.
storage-mapping element to map a file on the host as /etc/pacemaker/pcmk-init.env in the container. Pacemaker Remote will parse this file as a shell-like format, with variables set as NAME=VALUE, ignoring blank lines and comments starting with "#".
Important
primitive, Pacemaker on all cluster nodes must be able to contact Pacemaker Remote inside the bundle’s containers.
network should not be set to "none" with a primitive).
ip-range-start. Any firewall must allow access from all cluster nodes to the control-port on the container IPs.
network to "host"), a unique control-port should be specified for each bundle. Any firewall must allow access from all cluster nodes to the control-port on all cluster and remote node IPs.
primitive, the primitive’s resource agent may want to set node attributes such as promotion scores. However, with containers, it is not apparent which node should get the attribute.
container-attribute-target resource meta-attribute allows the user to specify which approach to use. If it is set to host, then user-defined node attributes will be checked on the underlying host. If it is anything else, the local node (in this case the bundle node) is used as usual.
#uname.
container-attribute-target is host, the cluster will pass additional environment variables to the primitive’s resource agent that allow it to set node attributes appropriately: CRM_meta_container_attribute_target (identical to the meta-attribute value) and CRM_meta_physical_host (the name of the underlying host).
Note
attrd_updater and crm_attribute commands will automatically check those environment variables and set attributes appropriately.
priority, target-role, and is-managed. See Section 4.4, “Resource Options” for more information.
replicas is greater than 1, the bundle will behave like a clone implicitly.)
control-port.
master tag instead of a clone tag with the promotable meta-attribute set to true; the master tag is supported, but deprecated, and will be removed in a future version. You may also see such services referred to as multi-state or stateful; these means the same thing as promotable.
Table of Contents
Example 11.1. Resource template for a migratable Xen virtual machine
<template id="vm-template" class="ocf" provider="heartbeat" type="Xen"> <meta_attributes id="vm-template-meta_attributes"> <nvpair id="vm-template-meta_attributes-allow-migrate" name="allow-migrate" value="true"/> </meta_attributes> <utilization id="vm-template-utilization"> <nvpair id="vm-template-utilization-memory" name="memory" value="512"/> </utilization> <operations> <op id="vm-template-monitor-15s" interval="15s" name="monitor" timeout="60s"/> <op id="vm-template-start-0" interval="0" name="start" timeout="60s"/> </operations> </template>
template property.
Example 11.2. Xen primitive resource using a resource template
<primitive id="vm1" template="vm-template"> <instance_attributes id="vm1-instance_attributes"> <nvpair id="vm1-instance_attributes-name" name="name" value="vm1"/> <nvpair id="vm1-instance_attributes-xmfile" name="xmfile" value="/etc/xen/shared-vm/vm1"/> </instance_attributes> </primitive>
vm1 will inherit everything from vm-template. For example, the equivalent of the above two examples would be:
Example 11.3. Equivalent Xen primitive resource not using a resource template
<primitive id="vm1" class="ocf" provider="heartbeat" type="Xen"> <meta_attributes id="vm-template-meta_attributes"> <nvpair id="vm-template-meta_attributes-allow-migrate" name="allow-migrate" value="true"/> </meta_attributes> <utilization id="vm-template-utilization"> <nvpair id="vm-template-utilization-memory" name="memory" value="512"/> </utilization> <operations> <op id="vm-template-monitor-15s" interval="15s" name="monitor" timeout="60s"/> <op id="vm-template-start-0" interval="0" name="start" timeout="60s"/> </operations> <instance_attributes id="vm1-instance_attributes"> <nvpair id="vm1-instance_attributes-name" name="name" value="vm1"/> <nvpair id="vm1-instance_attributes-xmfile" name="xmfile" value="/etc/xen/shared-vm/vm1"/> </instance_attributes> </primitive>
Example 11.4. Xen resource overriding template values
<primitive id="vm2" template="vm-template"> <meta_attributes id="vm2-meta_attributes"> <nvpair id="vm2-meta_attributes-allow-migrate" name="allow-migrate" value="false"/> </meta_attributes> <utilization id="vm2-utilization"> <nvpair id="vm2-utilization-memory" name="memory" value="1024"/> </utilization> <instance_attributes id="vm2-instance_attributes"> <nvpair id="vm2-instance_attributes-name" name="name" value="vm2"/> <nvpair id="vm2-instance_attributes-xmfile" name="xmfile" value="/etc/xen/shared-vm/vm2"/> </instance_attributes> <operations> <op id="vm2-monitor-30s" interval="30s" name="monitor" timeout="120s"/> <op id="vm2-stop-0" interval="0" name="stop" timeout="60s"/> </operations> </primitive>
vm2 has special attribute values. Its monitor operation has a longer timeout and interval, and the primitive has an additional stop operation.
# crm_resource --query-xml --resource vm2
# crm_resource --query-xml-raw --resource vm2
order constraints (see Section 5.3, “Specifying the Order in which Resources Should Start/Stop”)
colocation constraints (see Section 5.4, “Placing Resources Relative to other Resources”)
rsc_ticket constraints (for multi-site clusters as described in Section 15.3, “Configuring Ticket Dependencies”)
<rsc_colocation id="vm-template-colo-base-rsc" rsc="vm-template" rsc-role="Started" with-rsc="base-rsc" score="INFINITY"/>
base-rsc and is the equivalent of the following constraint configuration:
<rsc_colocation id="vm-colo-base-rsc" score="INFINITY"> <resource_set id="vm-colo-base-rsc-0" sequential="false" role="Started"> <resource_ref id="vm1"/> <resource_ref id="vm2"/> </resource_set> <resource_set id="vm-colo-base-rsc-1"> <resource_ref id="base-rsc"/> </resource_set> </rsc_colocation>
Note
rsc or with-rsc; the other reference must be a regular resource.
<rsc_order id="order1" score="INFINITY"> <resource_set id="order1-0"> <resource_ref id="base-rsc"/> <resource_ref id="vm-template"/> <resource_ref id="top-rsc"/> </resource_set> </rsc_order>
<rsc_order id="order1" score="INFINITY"> <resource_set id="order1-0"> <resource_ref id="base-rsc"/> <resource_ref id="vm1"/> <resource_ref id="vm2"/> <resource_ref id="top-rsc"/> </resource_set> </rsc_order>
<rsc_order id="order2" score="INFINITY"> <resource_set id="order2-0"> <resource_ref id="base-rsc"/> </resource_set> <resource_set id="order2-1" sequential="false"> <resource_ref id="vm-template"/> </resource_set> <resource_set id="order2-2"> <resource_ref id="top-rsc"/> </resource_set> </rsc_order>
<rsc_order id="order2" score="INFINITY"> <resource_set id="order2-0"> <resource_ref id="base-rsc"/> </resource_set> <resource_set id="order2-1" sequential="false"> <resource_ref id="vm1"/> <resource_ref id="vm2"/> </resource_set> <resource_set id="order2-2"> <resource_ref id="top-rsc"/> </resource_set> </rsc_order>
id-ref instead of an id.
<rsc_location id="WebServer-connectivity" rsc="Webserver"> <rule id="ping-prefer-rule" score-attribute="pingd" > <expression id="ping-prefer" attribute="pingd" operation="defined"/> </rule> </rsc_location>
Example 11.5. Referencing rules from other constraints
<rsc_location id="WebDB-connectivity" rsc="WebDB"> <rule id-ref="ping-prefer-rule"/> </rsc_location>
Important
rule exists somewhere. Attempting to add a reference to a non-existing rule will cause a validation failure, as will attempting to remove a rule that is referenced elsewhere.
meta_attributes and instance_attributes as illustrated in the example below:
Example 11.6. Referencing attributes, options, and operations from other resources
<primitive id="mySpecialRsc" class="ocf" type="Special" provider="me"> <instance_attributes id="mySpecialRsc-attrs" score="1" > <nvpair id="default-interface" name="interface" value="eth0"/> <nvpair id="default-port" name="port" value="9999"/> </instance_attributes> <meta_attributes id="mySpecialRsc-options"> <nvpair id="failure-timeout" name="failure-timeout" value="5m"/> <nvpair id="migration-threshold" name="migration-threshold" value="1"/> <nvpair id="stickiness" name="resource-stickiness" value="0"/> </meta_attributes> <operations id="health-checks"> <op id="health-check" name="monitor" interval="60s"/> <op id="health-check" name="monitor" interval="30min"/> </operations> </primitive> <primitive id="myOtherlRsc" class="ocf" type="Other" provider="me"> <instance_attributes id-ref="mySpecialRsc-attrs"/> <meta_attributes id-ref="mySpecialRsc-options"/> <operations id-ref="health-checks"/> </primitive>
id-ref can similarly be used with resource_set (in any constraint type), nvpair, and operations.
Example 11.7. Tag referencing three resources
<tags> <tag id="all-vms"> <obj_ref id="vm1"/> <obj_ref id="vm2"/> <obj_ref id="vm3"/> </tag> </tags>
Example 11.8. Constraint using a tag
<rsc_order id="order1" first="storage" then="all-vms" kind="Mandatory" />
all-vms tag is defined as in the previous example, the constraint will behave the same as:
Example 11.9. Equivalent constraints without tags
<rsc_order id="order1-1" first="storage" then="vm1" kind="Mandatory" /> <rsc_order id="order1-2" first="storage" then="vm2" kind="Mandatory" /> <rsc_order id="order1-3" first="storage" then="vm2" kind="Mandatory" />
sequential property.
Table of Contents
node and resource objects. You can name utilization attributes according to your preferences and define as many name/value pairs as your configuration needs. However, the attributes' values must be integers.
Example 12.1. Specifying CPU and RAM capacities of two nodes
<node id="node1" type="normal" uname="node1"> <utilization id="node1-utilization"> <nvpair id="node1-utilization-cpu" name="cpu" value="2"/> <nvpair id="node1-utilization-memory" name="memory" value="2048"/> </utilization> </node> <node id="node2" type="normal" uname="node2"> <utilization id="node2-utilization"> <nvpair id="node2-utilization-cpu" name="cpu" value="4"/> <nvpair id="node2-utilization-memory" name="memory" value="4096"/> </utilization> </node>
Example 12.2. Specifying CPU and RAM consumed by several resources
<primitive id="rsc-small" class="ocf" provider="pacemaker" type="Dummy"> <utilization id="rsc-small-utilization"> <nvpair id="rsc-small-utilization-cpu" name="cpu" value="1"/> <nvpair id="rsc-small-utilization-memory" name="memory" value="1024"/> </utilization> </primitive> <primitive id="rsc-medium" class="ocf" provider="pacemaker" type="Dummy"> <utilization id="rsc-medium-utilization"> <nvpair id="rsc-medium-utilization-cpu" name="cpu" value="2"/> <nvpair id="rsc-medium-utilization-memory" name="memory" value="2048"/> </utilization> </primitive> <primitive id="rsc-large" class="ocf" provider="pacemaker" type="Dummy"> <utilization id="rsc-large-utilization"> <nvpair id="rsc-large-utilization-cpu" name="cpu" value="3"/> <nvpair id="rsc-large-utilization-memory" name="memory" value="3072"/> </utilization> </primitive>
placement-strategy in the global cluster options, otherwise the capacity configurations have no effect.
placement-strategy:
defaultutilizationbalancedminimalplacement-strategy with crm_attribute:
# crm_attribute --name placement-strategy --update balanced
placement-strategy is default or utilization, the node that has the least number of allocated resources gets consumed first.
placement-strategy is balanced, the node that has the most free capacity gets consumed first.
placement-strategy is minimal, the first eligible node listed in the CIB gets consumed first.
nodeA has more free cpus, and nodeB has more free memory, then their free capacities are equal.
nodeA has more free cpus, while nodeB has more free memory and storage, then nodeB has more free capacity.
priority (see Section 4.4, “Resource Options”) gets allocated first.
rsc-small would be allocated to node1
rsc-medium would be allocated to node2
rsc-large would remain inactive
Table of Contents
root user or any user in the haclient group can modify Pacemaker’s CIB without restriction. Pacemaker offers access control lists (ACLs) to provide more fine-grained authorization.
pacemakerd --features contains acls, your installation supports ACLs.
haclient group on all nodes in the cluster.
cibadmin --upgrade in order to use the syntax documented here.
enable-acl cluster option must be set to true.
acls element of the CIB. The acls element may contain any number of acl_role, acl_target, and acl_group elements.
Table 13.1. Properties of an ACL Role
acl_role element may contain any number of acl_permission elements.
Table 13.2. Properties of an ACL Permission
| Attribute | Description |
|---|---|
id
| |
description
| |
kind
| |
object-type
| |
attribute
| |
reference
| |
xpath
|
An XPath specification selecting an XML element in the CIB to which the permission applies. Attributes may be specified in the XPath to select particular elements, but the permissions apply to the entire element. (Exactly one of
object-type, xpath, and reference must be specified for a permission.)
|
Important
deny permission takes precedence, then write, then lastly read.
Table 13.3. Properties of an ACL Target
Table 13.4. Properties of an ACL Group
acl_target and acl_group element may contain any number of role elements.
Table 13.5. Properties of an ACL Role Reference
Important
root and hacluster user accounts always have full access to the CIB, regardless of ACLs. For other user accounts, when enable-acl is true, permission to all parts of the CIB is denied by default (permissions must be explicitly granted).
<acls> <acl_role id="read_all"> <acl_permission id="read_all-cib" kind="read" xpath="/cib" /> </acl_role> <acl_role id="operator"> <acl_permission id="operator-maintenance-mode" kind="write" xpath="//crm_config//nvpair[@name='maintenance-mode']" /> <acl_permission id="operator-maintenance-attr" kind="write" xpath="//nvpair[@name='maintenance']" /> <acl_permission id="operator-target-role" kind="write" xpath="//resources//meta_attributes/nvpair[@name='target-role']" /> <acl_permission id="operator-is-managed" kind="write" xpath="//resources//nvpair[@name='is-managed']" /> <acl_permission id="operator-rsc_location" kind="write" object-type="rsc_location" /> </acl_role> <acl_role id="administrator"> <acl_permission id="administrator-cib" kind="write" xpath="/cib" /> </acl_role> <acl_role id="minimal"> <acl_permission id="minimal-standby" kind="read" description="allow reading standby node attribute (permanent or transient)" xpath="//instance_attributes/nvpair[@name='standby']"/> <acl_permission id="minimal-maintenance" kind="read" description="allow reading maintenance node attribute (permanent or transient)" xpath="//nvpair[@name='maintenance']"/> <acl_permission id="minimal-target-role" kind="read" description="allow reading resource target roles" xpath="//resources//meta_attributes/nvpair[@name='target-role']"/> <acl_permission id="minimal-is-managed" kind="read" description="allow reading resource managed status" xpath="//resources//meta_attributes/nvpair[@name='is-managed']"/> <acl_permission id="minimal-deny-instance-attributes" kind="deny" xpath="//instance_attributes"/> <acl_permission id="minimal-deny-meta-attributes" kind="deny" xpath="//meta_attributes"/> <acl_permission id="minimal-deny-operations" kind="deny" xpath="//operations"/> <acl_permission id="minimal-deny-utilization" kind="deny" xpath="//utilization"/> <acl_permission id="minimal-nodes" kind="read" description="allow reading node names/IDs (attributes are denied separately)" xpath="/cib/configuration/nodes"/> <acl_permission id="minimal-resources" kind="read" description="allow reading resource names/agents (parameters are denied separately)" xpath="/cib/configuration/resources"/> <acl_permission id="minimal-deny-constraints" kind="deny" xpath="/cib/configuration/constraints"/> <acl_permission id="minimal-deny-topology" kind="deny" xpath="/cib/configuration/fencing-topology"/> <acl_permission id="minimal-deny-op_defaults" kind="deny" xpath="/cib/configuration/op_defaults"/> <acl_permission id="minimal-deny-rsc_defaults" kind="deny" xpath="/cib/configuration/rsc_defaults"/> <acl_permission id="minimal-deny-alerts" kind="deny" xpath="/cib/configuration/alerts"/> <acl_permission id="minimal-deny-acls" kind="deny" xpath="/cib/configuration/acls"/> <acl_permission id="minimal-cib" kind="read" description="allow reading cib element and crm_config/status sections" xpath="/cib"/> </acl_role> <acl_target id="alice"> <role id="minimal"/> </acl_target> <acl_target id="bob"> <role id="read_all"/> </acl_target> <acl_target id="carol"> <role id="read_all"/> <role id="operator"/> </acl_target> <acl_target id="dave"> <role id="administrator"/> </acl_target> </acls>
alice has the minimal permissions necessary to run basic Pacemaker CLI tools, including using crm_mon to view the cluster status, without being able to modify anything. The user bob can view the entire configuration and status of the cluster, but not make any changes. The user carol can read everything, and change selected cluster properties as well as resource roles and location constraints. Finally, dave has full read and write access to the entire CIB.
minimal role in more depth, it is designed to allow read access to the cib tag itself, while denying access to particular portions of its subtree (which is the entire CIB).
cib tag, so crm_mon will not be able to report the DC otherwise. However, this does change the security model to allow by default, since any portions of the CIB not explicitly denied will be readable. The cib read access could be removed and replaced with read access to just the crm_config and status sections, for a safer approach at the cost of not seeing the DC in status output.
minimal role allows read access to the entire crm_config section, which contains cluster properties. It would be possible to allow read access to specific properties instead (such as stonith-enabled, dc-uuid, have-quorum, and cluster-name) to restrict access further while still allowing status output, but cluster properties are unlikely to be considered sensitive.
Table of Contents
crm_mon.
status section.
Example 14.1. A bare-bones status entry for a healthy node cl-virt-1
<node_state id="1" uname="cl-virt-1" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member"> <transient_attributes id="1"/> <lrm id="1"/> </node_state>
Table 14.1. Authoritative Sources for State Information
| CIB Object | Authoritative Source |
|---|---|
node_state
|
pacemaker-controld
|
transient_attributes
|
pacemaker-attrd
|
lrm
|
pacemaker-execd
|
node_state objects are named as they are largely for historical reasons and are rooted in Pacemaker’s origins as the resource manager for the older Heartbeat project. They have remained unchanged to preserve compatibility with older versions.
Table 14.2. Node Status Fields
| Field | Description |
|---|---|
id
| |
uname
| |
in_ccm
| |
crmd
| |
crm-debug-origin
| |
join
| |
expected
|
transient_attributes section help to describe the node. However they are forgotten by the cluster when the node goes offline. This can be useful, for instance, when you want a node to be in standby mode (not able to run resources) just until the next reboot.
Example 14.2. A set of transient node attributes for node cl-virt-1
<transient_attributes id="cl-virt-1"> <instance_attributes id="status-cl-virt-1"> <nvpair id="status-cl-virt-1-pingd" name="pingd" value="3"/> <nvpair id="status-cl-virt-1-probe_complete" name="probe_complete" value="true"/> <nvpair id="status-cl-virt-1-fail-count-pingd:0.monitor_30000" name="fail-count-pingd:0#monitor_30000" value="1"/> <nvpair id="status-cl-virt-1-last-failure-pingd:0" name="last-failure-pingd:0" value="1239009742"/> </instance_attributes> </transient_attributes>
lrm_resources tag (a child of the lrm tag). The information stored here includes enough information for the cluster to stop the resource safely if it is removed from the configuration section. Specifically, the resource’s id, class, type and provider are stored.
Example 14.3. A record of the apcstonith resource
<lrm_resource id="apcstonith" type="apcmastersnmp" class="stonith"/>
resource, action and interval. The concatenation of the values in this tuple are used to create the id of the lrm_rsc_op object.
Table 14.3. Contents of an lrm_rsc_op job
| Field | Description |
|---|---|
id
|
Identifier for the job constructed from the resource’s
id, operation and interval.
|
call-id
|
The job’s ticket number. Used as a sort key to determine the order in which the jobs were executed.
|
operation
|
The action the resource agent was invoked with.
|
interval
|
The frequency, in milliseconds, at which the operation will be repeated. A one-off job is indicated by 0.
|
op-status
|
The job’s status. Generally this will be either 0 (done) or -1 (pending). Rarely used in favor of
rc-code.
|
rc-code
|
The job’s result. Refer to the Resource Agents section of Pacemaker Administration for details on what the values here mean and how they are interpreted.
|
last-run
|
Machine-local date/time, in seconds since epoch, at which the job was executed. For diagnostic purposes.
|
last-rc-change
|
Machine-local date/time, in seconds since epoch, at which the job first returned the current value of
rc-code. For diagnostic purposes.
|
exec-time
|
Time, in milliseconds, that the job was running for. For diagnostic purposes.
|
queue-time
|
Time, in seconds, that the job was queued for in the LRMd. For diagnostic purposes.
|
crm_feature_set
|
The version which this job description conforms to. Used when processing
op-digest.
|
transition-key
|
A concatenation of the job’s graph action number, the graph number, the expected result and the UUID of the controller instance that scheduled it. This is used to construct
transition-magic (below).
|
transition-magic
|
A concatenation of the job’s
op-status, rc-code and transition-key. Guaranteed to be unique for the life of the cluster (which ensures it is part of CIB update notifications) and contains all the information needed for the controller to correctly analyze and process the completed job. Most importantly, the decomposed elements tell the controller if the job entry was expected and whether it failed.
|
op-digest
|
An MD5 sum representing the parameters passed to the job. Used to detect changes to the configuration, to restart resources if necessary.
|
crm-debug-origin
|
The origin of the current values. For diagnostic purposes.
|
Example 14.4. A monitor operation (determines current state of the apcstonith resource)
<lrm_resource id="apcstonith" type="apcmastersnmp" class="stonith"> <lrm_rsc_op id="apcstonith_monitor_0" operation="monitor" call-id="2" rc-code="7" op-status="0" interval="0" crm-debug-origin="do_update_resource" crm_feature_set="3.0.1" op-digest="2e3da9274d3550dc6526fb24bfcbcba0" transition-key="22:2:7:2668bbeb-06d5-40f9-936d-24cb7f87006a" transition-magic="0:7;22:2:7:2668bbeb-06d5-40f9-936d-24cb7f87006a" last-run="1239008085" last-rc-change="1239008085" exec-time="10" queue-time="0"/> </lrm_resource>
apcstonith resource.
transition-key, we can see that this was the 22nd action of the 2nd graph produced by this instance of the controller (2668bbeb-06d5-40f9-936d-24cb7f87006a).
transition-key contains a 7, which indicates that the job expects to find the resource inactive. By looking at the rc-code property, we see that this was the case.
Example 14.5. Resource history of a pingd clone with multiple jobs
<lrm_resource id="pingd:0" type="pingd" class="ocf" provider="pacemaker"> <lrm_rsc_op id="pingd:0_monitor_30000" operation="monitor" call-id="34" rc-code="0" op-status="0" interval="30000" crm-debug-origin="do_update_resource" crm_feature_set="3.0.1" transition-key="10:11:0:2668bbeb-06d5-40f9-936d-24cb7f87006a" ... last-run="1239009741" last-rc-change="1239009741" exec-time="10" queue-time="0"/> <lrm_rsc_op id="pingd:0_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.0.1" call-id="32" rc-code="0" op-status="0" interval="0" transition-key="11:11:0:2668bbeb-06d5-40f9-936d-24cb7f87006a" ... last-run="1239009741" last-rc-change="1239009741" exec-time="10" queue-time="0"/> <lrm_rsc_op id="pingd:0_start_0" operation="start" call-id="33" rc-code="0" op-status="0" interval="0" crm-debug-origin="do_update_resource" crm_feature_set="3.0.1" transition-key="31:11:0:2668bbeb-06d5-40f9-936d-24cb7f87006a" ... last-run="1239009741" last-rc-change="1239009741" exec-time="10" queue-time="0" /> <lrm_rsc_op id="pingd:0_monitor_0" operation="monitor" call-id="3" rc-code="0" op-status="0" interval="0" crm-debug-origin="do_update_resource" crm_feature_set="3.0.1" transition-key="23:2:7:2668bbeb-06d5-40f9-936d-24cb7f87006a" ... last-run="1239008085" last-rc-change="1239008085" exec-time="20" queue-time="0"/> </lrm_resource>
call-id before interpreting them.
call-id of 3
call-id of 32
call-id of 33
call-id of 34
stop operation with a lower call-id than that of the start operation, we can conclude that the resource has been restarted. Specifically this occurred as part of actions 11 and 31 of transition 11 from the controller instance with the key 2668bbeb…. This information can be helpful for locating the relevant section of the logs when looking for the source of a failure.
date command to print a human-readable version of any seconds-since-epoch value, for example date -d @1239009742.
Table of Contents
rsc_ticket constraints. Only if the ticket is available at a site can the respective resources be started there. Vice versa, if the ticket is revoked, the resources depending on that ticket must be stopped.
have-quorum flag as a special, cluster-wide ticket that is granted in case of node majority.)
true (the site has the ticket) or false (the site does not have the ticket). The absence of a certain ticket (during the initial state of the multi-site cluster) is the same as the value false.
loss-policy="fence" in rsc_ticket constraints.
rsc_ticket constraint lets you specify the resources depending on a certain ticket. Together with the constraint, you can set a loss-policy that defines what should happen to the respective resources if the ticket is revoked.
loss-policy can have the following values:
fence: Fence the nodes that are running the relevant resources.
stop: Stop the relevant resources.
freeze: Do nothing to the relevant resources.
demote: Demote relevant resources that are running in master mode to slave mode.
Example 15.1. Constraint that fences node if ticketA is revoked
<rsc_ticket id="rsc1-req-ticketA" rsc="rsc1" ticket="ticketA" loss-policy="fence"/>
rsc1-req-ticketA. It defines that the resource rsc1 depends on ticketA and that the node running the resource should be fenced if ticketA is revoked.
rsc1 were a promotable resource (i.e. it could run in master or slave mode), you might want to configure that only master mode depends on ticketA. With the following configuration, rsc1 will be demoted to slave mode if ticketA is revoked:
Example 15.2. Constraint that demotes rsc1 if ticketA is revoked
<rsc_ticket id="rsc1-req-ticketA" rsc="rsc1" rsc-role="Master" ticket="ticketA" loss-policy="demote"/>
rsc_ticket constraints to let multiple resources depend on the same ticket. However, rsc_ticket also supports resource sets (see Section 5.5, “Resource Sets”), so one can easily list all the resources in one rsc_ticket constraint instead.
Example 15.3. Ticket constraint for multiple resources
<rsc_ticket id="resources-dep-ticketA" ticket="ticketA" loss-policy="fence"> <resource_set id="resources-dep-ticketA-0" role="Started"> <resource_ref id="rsc1"/> <resource_ref id="group1"/> <resource_ref id="clone1"/> </resource_set> <resource_set id="resources-dep-ticketA-1" role="Master"> <resource_ref id="ms1"/> </resource_set> </rsc_ticket>
rsc_ticket constraint. There’s no dependency between the two resource sets, and there’s no dependency among the resources within a resource set. Each of the resources just depends on ticketA.
rsc_ticket constraints, and even referencing them within resource sets, is also supported.
rsc_ticket.
crm_ticket command line tool to grant and revoke tickets.
# crm_ticket --ticket ticketA --grant
# crm_ticket --ticket ticketA --revoke
Important
crm_ticket command with great care, because it cannot check whether the same ticket is already granted elsewhere.
boothd.
Boothd at each site connects to its peers running at the other sites and exchanges connectivity details. Once a ticket is granted to a site, the booth mechanism will manage the ticket automatically: If the site which holds the ticket is out of service, the booth daemons will vote which of the other sites will get the ticket. To protect against brief connection failures, sites that lose the vote (either explicitly or implicitly by being disconnected from the voting body) need to relinquish the ticket after a time-out. Thus, it is made sure that a ticket will only be re-distributed after it has been relinquished by the previous site. The resources that depend on that ticket will fail over to the new site holding the ticket. The nodes that have run the resources before will be treated according to the loss-policy you set within the rsc_ticket constraint.
booth command-line tool. After you have initially granted a ticket to a site, boothd will take over and manage the ticket automatically.
Important
booth command-line tool can be used to grant, list, or revoke tickets and can be run on any machine where boothd is running. If you are managing tickets via Booth, use only booth for manual intervention, not crm_ticket. That ensures the same ticket will only be owned by one cluster site at a time.
# crm_ticket --info
# crm_mon --tickets
rsc_ticket constraints that apply to a ticket:
# crm_ticket --ticket ticketA --constraints
loss-policy="fence", the dependent resources could not be gracefully stopped/demoted, and other unrelated resources could even be affected.
# crm_ticket --ticket ticketA --standby
# crm_ticket --ticket ticketA --activate
Table of Contents
Example A.1. An Empty Configuration
<cib crm_feature_set="3.0.7" validate-with="pacemaker-1.2" admin_epoch="1" epoch="0" num_updates="0"> <configuration> <crm_config/> <nodes/> <resources/> <constraints/> </configuration> <status/> </cib>
Example A.2. A simple configuration with two nodes, some cluster options and a resource
<cib crm_feature_set="3.0.7" validate-with="pacemaker-1.2" admin_epoch="1" epoch="0" num_updates="0"> <configuration> <crm_config> <cluster_property_set id="cib-bootstrap-options"> <nvpair id="option-1" name="symmetric-cluster" value="true"/> <nvpair id="option-2" name="no-quorum-policy" value="stop"/> <nvpair id="option-3" name="stonith-enabled" value="0"/> </cluster_property_set> </crm_config> <nodes> <node id="xxx" uname="c001n01" type="normal"/> <node id="yyy" uname="c001n02" type="normal"/> </nodes> <resources> <primitive id="myAddr" class="ocf" provider="heartbeat" type="IPaddr"> <operations> <op id="myAddr-monitor" name="monitor" interval="300s"/> </operations> <instance_attributes id="myAddr-params"> <nvpair id="myAddr-ip" name="ip" value="192.0.2.10"/> </instance_attributes> </primitive> </resources> <constraints> <rsc_location id="myAddr-prefer" rsc="myAddr" node="c001n01" score="INFINITY"/> </constraints> <rsc_defaults> <meta_attributes id="rsc_defaults-options"> <nvpair id="rsc-default-1" name="resource-stickiness" value="100"/> <nvpair id="rsc-default-2" name="migration-threshold" value="10"/> </meta_attributes> </rsc_defaults> <op_defaults> <meta_attributes id="op_defaults-options"> <nvpair id="op-default-1" name="timeout" value="30s"/> </meta_attributes> </op_defaults> </configuration> <status/> </cib>
c001n01 until either the resource fails 10 times or the host shuts down.
Example A.3. An advanced configuration with groups, clones and STONITH
<cib crm_feature_set="3.0.7" validate-with="pacemaker-1.2" admin_epoch="1" epoch="0" num_updates="0"> <configuration> <crm_config> <cluster_property_set id="cib-bootstrap-options"> <nvpair id="option-1" name="symmetric-cluster" value="true"/> <nvpair id="option-2" name="no-quorum-policy" value="stop"/> <nvpair id="option-3" name="stonith-enabled" value="true"/> </cluster_property_set> </crm_config> <nodes> <node id="xxx" uname="c001n01" type="normal"/> <node id="yyy" uname="c001n02" type="normal"/> <node id="zzz" uname="c001n03" type="normal"/> </nodes> <resources> <primitive id="myAddr" class="ocf" provider="heartbeat" type="IPaddr"> <operations> <op id="myAddr-monitor" name="monitor" interval="300s"/> </operations> <instance_attributes id="myAddr-attrs"> <nvpair id="myAddr-attr-1" name="ip" value="192.0.2.10"/> </instance_attributes> </primitive> <group id="myGroup"> <primitive id="database" class="lsb" type="oracle"> <operations> <op id="database-monitor" name="monitor" interval="300s"/> </operations> </primitive> <primitive id="webserver" class="lsb" type="apache"> <operations> <op id="webserver-monitor" name="monitor" interval="300s"/> </operations> </primitive> </group> <clone id="STONITH"> <meta_attributes id="stonith-options"> <nvpair id="stonith-option-1" name="globally-unique" value="false"/> </meta_attributes> <primitive id="stonithclone" class="stonith" type="external/ssh"> <operations> <op id="stonith-op-mon" name="monitor" interval="5s"/> </operations> <instance_attributes id="stonith-attrs"> <nvpair id="stonith-attr-1" name="hostlist" value="c001n01,c001n02"/> </instance_attributes> </primitive> </clone> </resources> <constraints> <rsc_location id="myAddr-prefer" rsc="myAddr" node="c001n01" score="INFINITY"/> <rsc_colocation id="group-with-ip" rsc="myGroup" with-rsc="myAddr" score="INFINITY"/> </constraints> <op_defaults> <meta_attributes id="op_defaults-options"> <nvpair id="op-default-1" name="timeout" value="30s"/> </meta_attributes> </op_defaults> <rsc_defaults> <meta_attributes id="rsc_defaults-options"> <nvpair id="rsc-default-1" name="resource-stickiness" value="100"/> <nvpair id="rsc-default-2" name="migration-threshold" value="10"/> </meta_attributes> </rsc_defaults> </configuration> <status/> </cib>
| Revision History | |||||
|---|---|---|---|---|---|
| Revision 1-0 | 19 Oct 2009 | ||||
| |||||
| Revision 2-0 | 26 Oct 2009 | ||||
| |||||
| Revision 3-0 | Tue Nov 12 2009 | ||||
| |||||
| Revision 3-1 | Tue Nov 12 2009 | , , , , | |||
| |||||
| Revision 3-2 | Fri Nov 4 2011 | ||||
| |||||
| Revision 4-0 | Mon Oct 8 2012 | ||||
| |||||
| Revision 5-0 | Mon Feb 23 2015 | ||||
| |||||
| Revision 6-0 | Tue Dec 8 2015 | ||||
| |||||
| Revision 7-0 | Tue May 3 2016 | ||||
| |||||
| Revision 7-1 | Fri Oct 28 2016 | ||||
| |||||
| Revision 8-0 | Tue Oct 25 2016 | ||||
| |||||
| Revision 9-0 | Tue Jul 11 2017 | ||||
| |||||
| Revision 10-0 | Fri Oct 6 2017 | ||||
| |||||
| Revision 11-0 | Fri Jan 12 2018 | ||||
| |||||
| Revision 12-0 | Mon Jan 28 2019 | , , | |||
| |||||
| Revision 12-1 | Mon May 13 2019 | , | |||
| |||||
| Revision 12-2 | Wed Jun 19 2019 | ||||
| |||||
| Revision 13-0 | Tue Oct 15 2019 | ||||
| |||||
| Revision 13-1 | Wed Apr 1 2020 | ||||
| |||||