4. Cluster Nodes¶
4.1. Defining a Cluster Node¶
Each cluster node will have an entry in the nodes
section containing at
least an ID and a name. A cluster node’s ID is defined by the cluster layer
(Corosync).
Example Corosync cluster node entry
<node id="101" uname="pcmk-1"/>
In normal circumstances, the admin should let the cluster populate this information automatically from the cluster layer.
4.1.1. Where Pacemaker Gets the Node Name¶
The name that Pacemaker uses for a node in the configuration does not have to be the same as its local hostname. Pacemaker uses the following for a Corosync node’s name, in order of most preferred first:
- The value of
name
in thenodelist
section ofcorosync.conf
- The value of
ring0_addr
in thenodelist
section ofcorosync.conf
- The local hostname (value of
uname -n
)
If the cluster is running, the crm_node -n
command will display the local
node’s name as used by the cluster.
If a Corosync nodelist
is used, crm_node --name-for-id
with a Corosync
node ID will display the name used by the node with the given Corosync
nodeid
, for example:
crm_node --name-for-id 2
4.2. Node Attributes¶
Pacemaker allows node-specific values to be specified using node attributes. A node attribute has a name, and may have a distinct value for each node.
Node attributes come in two types, permanent and transient. Permanent node
attributes are kept within the node
entry, and keep their values even if
the cluster restarts on a node. Transient node attributes are kept in the CIB’s
status
section, and go away when the cluster stops on the node.
While certain node attributes have specific meanings to the cluster, they are mainly intended to allow administrators and resource agents to track any information desired.
For example, an administrator might choose to define node attributes for how much RAM and disk space each node has, which OS each uses, or which server room rack each node is in.
Users can configure Rules that use node attributes to affect where resources are placed.
4.2.1. Setting and querying node attributes¶
Node attributes can be set and queried using the crm_attribute
and
attrd_updater
commands, so that the user does not have to deal with XML
configuration directly.
Here is an example command to set a permanent node attribute, and the XML configuration that would be generated:
Result of using crm_attribute to specify which kernel pcmk-1 is running
# crm_attribute --type nodes --node pcmk-1 --name kernel --update $(uname -r)
<node id="1" uname="pcmk-1">
<instance_attributes id="nodes-1-attributes">
<nvpair id="nodes-1-kernel" name="kernel" value="3.10.0-862.14.4.el7.x86_64"/>
</instance_attributes>
</node>
To read back the value that was just set:
# crm_attribute --type nodes --node pcmk-1 --name kernel --query
scope=nodes name=kernel value=3.10.0-862.14.4.el7.x86_64
The --type nodes
indicates that this is a permanent node attribute;
--type status
would indicate a transient node attribute.
Warning
Attribute values with newline or tab characters are currently displayed with
newlines as "\n"
and tabs as "\t"
, when crm_attribute
or
attrd_updater
query commands use --output-as=text
or leave
--output-as
unspecified:
# crm_attribute -N node1 -n test_attr -v "$(echo -e "a\nb\tc")" -t status
# crm_attribute -N node1 -n test_attr --query -t status
scope=status name=test_attr value=a\nb\tc
This format is deprecated. In a future release, the values will be displayed with literal whitespace characters:
# crm_attribute -N node1 -n test_attr --query -t status
scope=status name=test_attr value=a
b c
Users should either avoid attribute values with newlines and tabs, or ensure that they can handle both formats.
However, it’s best to use --output-as=xml
when parsing attribute values
from output. Newlines, tabs, and special characters are replaced with XML
character references that a conforming XML processor can recognize and
convert to literals (since 2.1.8):
# crm_attribute -N node1 -n test_attr --query -t status --output-as=xml
<pacemaker-result api-version="2.35" request="crm_attribute -N laptop -n test_attr --query -t status --output-as=xml">
<attribute name="test_attr" value="a b	c" scope="status"/>
<status code="0" message="OK"/>
</pacemaker-result>
4.2.2. Special node attributes¶
Certain node attributes have special meaning to the cluster.
Node attribute names beginning with #
are considered reserved for these
special attributes. Some special attributes do not start with #
, for
historical reasons.
Certain special attributes are set automatically by the cluster, should never be modified directly, and can be used only within Rules; these are listed under built-in node attributes.
For true/false values, the cluster considers a value of “1”, “y”, “yes”, “on”, or “true” (case-insensitively) to be true, “0”, “n”, “no”, “off”, “false”, or unset to be false, and anything else to be an error.
Name | Description |
---|---|
fail-count-* | Attributes whose names start with
|
last-failure-* | Attributes whose names start with
|
maintenance | If true, the cluster will not start or stop any
resources on this node. Any resources active on the
node become unmanaged, and any recurring operations
for those resources (except those specifying
|
probe_complete | This is managed by the cluster to detect when nodes need to be reprobed, and should never be used directly. |
resource-discovery-enabled | If the node is a remote node, fencing is enabled,
and this attribute is explicitly set to false
(unset means true in this case), resource discovery
(probes) will not be done on this node. This is
highly discouraged; the |
shutdown | This is managed by the cluster to orchestrate the shutdown of a node, and should never be used directly. |
site-name | If set, this will be used as the value of the
|
standby | If true, the node is in standby mode. This is
typically set and queried via the |
terminate | If the value is true or begins with any nonzero number, the node will be fenced. This is typically set by tools rather than directly. |
#digests-* | Attributes whose names start with |
#node-unfenced | When the node was last unfenced (as seconds since the epoch). This is managed by the cluster and should never be used directly. |
4.3. Tracking Node Health¶
A node may be functioning adequately as far as cluster membership is concerned, and yet be “unhealthy” in some respect that makes it an undesirable location for resources. For example, a disk drive may be reporting SMART errors, or the CPU may be highly loaded.
Pacemaker offers a way to automatically move resources off unhealthy nodes.
4.3.1. Node Health Attributes¶
Pacemaker will treat any node attribute whose name starts with #health
as
an indicator of node health. Node health attributes may have one of the
following values:
Value | Intended significance |
---|---|
red |
This indicator is unhealthy |
yellow |
This indicator is becoming unhealthy |
green |
This indicator is healthy |
integer | A numeric score to apply to all resources on this node (0 or positive is healthy, negative is unhealthy) |
4.3.2. Node Health Strategy¶
Pacemaker assigns a node health score to each node, as the sum of the values of all its node health attributes. This score will be used as a location constraint applied to this node for all resources.
The node-health-strategy
cluster option controls how Pacemaker responds to
changes in node health attributes, and how it translates red
, yellow
,
and green
to scores.
Allowed values are:
Value | Effect |
---|---|
none | Do not track node health attributes at all. |
migrate-on-red | Assign the value of |
only-green | Assign the value of |
progressive | Assign the value of the |
custom | Track node health attributes using the same values as
|
4.3.3. Exempting a Resource from Health Restrictions¶
If you want a resource to be able to run on a node even if its health score
would otherwise prevent it, set the resource’s allow-unhealthy-nodes
meta-attribute to true
(available since 2.1.3).
This is particularly useful for node health agents, to allow them to detect when the node becomes healthy again. If you configure a health agent without this setting, then the health agent will be banned from an unhealthy node, and you will have to investigate and clear the health attribute manually once it is healthy to allow resources on the node again.
If you want the meta-attribute to apply to a clone, it must be set on the clone itself, not on the resource being cloned.
4.3.4. Configuring Node Health Agents¶
Since Pacemaker calculates node health based on node attributes, any method that sets node attributes may be used to measure node health. The most common are resource agents and custom daemons.
Pacemaker provides examples that can be used directly or as a basis for custom
code. The ocf:pacemaker:HealthCPU
, ocf:pacemaker:HealthIOWait
, and
ocf:pacemaker:HealthSMART
resource agents set node health attributes based
on CPU and disk status.
To take advantage of this feature, add the resource to your cluster (generally as a cloned resource with a recurring monitor action, to continually check the health of all nodes). For example:
Example HealthIOWait resource configuration
<clone id="resHealthIOWait-clone">
<primitive class="ocf" id="HealthIOWait" provider="pacemaker" type="HealthIOWait">
<instance_attributes id="resHealthIOWait-instance_attributes">
<nvpair id="resHealthIOWait-instance_attributes-red_limit" name="red_limit" value="30"/>
<nvpair id="resHealthIOWait-instance_attributes-yellow_limit" name="yellow_limit" value="10"/>
</instance_attributes>
<operations>
<op id="resHealthIOWait-monitor-interval-5" interval="5" name="monitor" timeout="5"/>
<op id="resHealthIOWait-start-interval-0s" interval="0s" name="start" timeout="10s"/>
<op id="resHealthIOWait-stop-interval-0s" interval="0s" name="stop" timeout="10s"/>
</operations>
</primitive>
</clone>
The resource agents use attrd_updater
to set proper status for each node
running this resource, as a node attribute whose name starts with #health
(for HealthIOWait
, the node attribute is named #health-iowait
).
When a node is no longer faulty, you can force the cluster to make it available to take resources without waiting for the next monitor, by setting the node health attribute to green. For example:
Force node1 to be marked as healthy
# attrd_updater --name "#health-iowait" --update "green" --node "node1"