Edition 2
Abstract
Table of Contents
List of Figures
List of Tables
List of Examples
Table of Contents
Mono-spaced Bold
To see the contents of the filemy_next_bestselling_novel
in your current working directory, enter thecat my_next_bestselling_novel
command at the shell prompt and press Enter to execute the command.
Press Enter to execute the command.Press Ctrl+Alt+F2 to switch to a virtual terminal.
mono-spaced bold
. For example:
File-related classes includefilesystem
for file systems,file
for files, anddir
for directories. Each class has its own associated set of permissions.
Choose Mouse Preferences. In the Buttons tab, select the Left-handed mouse check box and click to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand).→ → from the main menu bar to launchTo insert a special character into a gedit file, choose → → from the main menu bar. Next, choose → from the Character Map menu bar, type the name of the character in the Search field and click . The character you sought will be highlighted in the Character Table. Double-click this highlighted character to place it in the Text to copy field and then click the button. Now switch back to your document and choose → from the gedit menu bar.
Mono-spaced Bold Italic
or Proportional Bold Italic
To connect to a remote machine using ssh, typessh username@domain.name
at a shell prompt. If the remote machine isexample.com
and your username on that machine is john, typessh john@example.com
.Themount -o remount file-system
command remounts the named file system. For example, to remount the/home
file system, the command ismount -o remount /home
.To see the version of a currently installed package, use therpm -q package
command. It will return a result as follows:package-version-release
.
Publican is a DocBook publishing system.
mono-spaced roman
and presented thus:
books Desktop documentation drafts mss photos stuff svn books_tests Desktop1 downloads images notes scripts svgs
mono-spaced roman
but add syntax highlighting as follows:
package org.jboss.book.jca.ex1;
import javax.naming.InitialContext;
public class ExClient
{
public static void main(String args[])
throws Exception
{
InitialContext iniCtx = new InitialContext();
Object ref = iniCtx.lookup("EchoBean");
EchoHome home = (EchoHome) ref;
Echo echo = home.create();
System.out.println("Created Echo");
System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));
}
}
Note
Important
Warning
Table of Contents
Fencing
Old daemon names
Old name | New name |
---|---|
attrd
|
pacemaker-attrd
|
cib
|
pacemaker-based
|
crmd
|
pacemaker-controld
|
lrmd
|
pacemaker-execd
|
stonithd
|
pacemaker-fenced
|
pacemaker_remoted
|
pacemaker-remoted
|
Table of Contents
/etc/corosync/corosync.conf
.
Example 2.1. Corosync configuration file for two nodes myhost1 and myhost2
totem { version: 2 secauth: off cluster_name: mycluster transport: udpu } nodelist { node { ring0_addr: myhost1 nodeid: 1 } node { ring0_addr: myhost2 nodeid: 2 } } quorum { provider: corosync_votequorum two_node: 1 } logging { to_syslog: yes }
Example 2.2. Corosync configuration file for three nodes myhost1, myhost2 and myhost3
totem { version: 2 secauth: off cluster_name: mycluster transport: udpu } nodelist { node { ring0_addr: myhost1 nodeid: 1 } node { ring0_addr: myhost2 nodeid: 2 } node { ring0_addr: myhost3 nodeid: 3 } } quorum { provider: corosync_votequorum } logging { to_syslog: yes }
totem
section defines what protocol version and options (including encryption) to use,
[4] and gives the cluster a unique name (mycluster
in these examples).
node
section lists the nodes in this cluster.
quorum
section defines how the cluster uses quorum. The important thing is that two-node clusters must be handled specially, so two_node: 1
must be defined for two-node clusters (and only for two-node clusters).
logging
section should be self-explanatory.
Table of Contents
/etc/corosync/corosync.conf
and /etc/corosync/authkey
(if it exists) from an existing node. You may need to modify the mcastaddr option to match the new node’s IP address.
pcs cluster stop
if you are using pcs for cluster management.
# crm_node -R pcmk-1
Table of Contents
Note
# crm_attribute --name stonith-enabled --update 1
# crm_standby --query --node node1
# crm_resource -r my-test-rsc --set-parameter migration-threshold --parameter-value 3 --meta
admin_epoch
, one should use:
# cibadmin --modify --xml-text '<cib admin_epoch="42"/>'
Example 4.1. XML attributes set for a cib element
<cib crm_feature_set="3.0.7" validate-with="pacemaker-1.2" admin_epoch="42" epoch="116" num_updates="1" cib-last-written="Mon Jan 12 15:46:39 2015" update-origin="rhel7-1" update-client="crm_attribute" have-quorum="1" dc-uuid="1">
crm_attribute
tool. To get the current value of cluster-delay
, you can run:
# crm_attribute --query --name cluster-delay
# crm_attribute -G -n cluster-delay
# crm_attribute -G -n cluster-delay scope=crm_config name=cluster-delay value=60s
# crm_attribute -G -n clusta-deway scope=crm_config name=clusta-deway value=(null) Error performing operation: No such device or address
# crm_attribute --name cluster-delay --update 30s
# crm_attribute --name cluster-delay --delete Deleted crm_config option: id=cib-bootstrap-options-cluster-delay name=cluster-delay
Example 4.2. Deleting an option that is listed twice
# crm_attribute --name batch-limit --delete Multiple attributes match name=batch-limit in crm_config: Value: 50 (set=cib-bootstrap-options, id=cib-bootstrap-options-batch-limit) Value: 100 (set=custom, id=custom-batch-limit) Please choose from one of the matches above and supply the 'id' with --id
Table 4.1. Environment Variables Used to Connect to Remote Instances of the CIB
Environment Variable | Default | Description |
---|---|---|
CIB_user
|
$USER
| |
CIB_passwd
|
| |
CIB_server
|
localhost
| |
CIB_port
|
| |
CIB_encrypted
|
TRUE
|
# export CIB_port=1234; export CIB_server=c001n01; export CIB_user=someuser; # cibadmin -Q
remote-tls-port
(encrypted) or remote-clear-port
(unencrypted) CIB properties (i.e., those kept in the cib
tag, like num_updates
and epoch
).
Table 4.2. Extra top-level CIB properties for remote access
Important
Table of Contents
crm_mon
and stonith_admin
. This is an ongoing project, and more tools will be converted over time. This system lets you control the formatting of output with --output-as=
and the destination of output with --output-to=
.
# crm_mon --help-output Usage: crm_mon [OPTION?] Provides a summary of cluster's current state. Outputs varying levels of detail in a number of different formats. Output Options: --output-as=FORMAT Specify output format as one of: console (default), html, text, xml --output-to=DEST Specify file name for output (or "-" for stdout) --html-cgi Add text needed to use output in a CGI program --html-stylesheet=URI Link to an external CSS stylesheet --html-title=TITLE Page title --text-fancy Use more highly formatted output
crm_mon
utility displays the current state of an active cluster. It can show the cluster status organized by node or by resource, and can be used in either single-shot or dynamically updating mode. It can also display operations performed and information about failures.
crm_mon --help
for a full description of its many options.
Example 5.1. Sample output from crm_mon -1
Cluster Summary: * Stack: corosync * Current DC: node2 (version 2.0.0-1) - partition with quorum * Last updated: Mon Jan 29 12:18:42 2018 * Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 * 5 nodes configured * 2 resources configured Node List: * Online: [ node1 node2 node3 node4 node5 ] * Active resources: * Fencing (stonith:fence_xvm): Started node1 * IP (ocf:heartbeat:IPaddr2): Started node2
Example 5.2. Sample output from crm_mon -n -1
Cluster Summary: * Stack: corosync * Current DC: node2 (version 2.0.0-1) - partition with quorum * Last updated: Mon Jan 29 12:21:48 2018 * Last change: Mon Jan 29 12:18:40 2018 by root via crm_attribute on node3 * 5 nodes configured * 2 resources configured * Node List: * Node node1: online * Fencing (stonith:fence_xvm): Started * Node node2: online * IP (ocf:heartbeat:IPaddr2): Started * Node node3: online * Node node4: online * Node node5: online
crm_mon
's HTML output have a CSS class associated with them. Not everything does, but some of the most interesting portions do. In the following example, the status of each node has an "online" class and the details of each resource have an "rsc-ok" class.
<h2>Node List</h2> <ul> <li> <span>Node: cluster01</span><span class="online"> online</span> </li> <li><ul><li><span class="rsc-ok">ping (ocf::pacemaker:ping): Started</span></li></ul></li> <li> <span>Node: cluster02</span><span class="online"> online</span> </li> <li><ul><li><span class="rsc-ok">ping (ocf::pacemaker:ping): Started</span></li></ul></li> </ul>
<style> .online { color: green } .rsc-ok { color: green } </style>
--html-stylesheet=<URL>
to crm_mon
. The link is added after the default stylesheet, so your changes take precedence. You don’t need to duplicate the entire default. Only include what you want to change.
cibadmin
command. With cibadmin
, you can query, add, remove, update or replace any part of the configuration. All changes take effect immediately, so there is no need to perform a reload-like operation.
cibadmin
is to use it to save the current configuration to a temporary file, edit that file with your favorite text or XML editor, and then upload the revised configuration.
Example 5.3. Safely using an editor to modify the cluster configuration
# cibadmin --query > tmp.xml # vi tmp.xml # cibadmin --replace --xml-file tmp.xml
pacemaker.rng
, which may be deployed in a location such as /usr/share/pacemaker
depending on your operating system distribution and how you installed the software.
Example 5.4. Safely using an editor to modify only the resources section
# cibadmin --query --scope resources > tmp.xml # vi tmp.xml # cibadmin --replace --scope resources --xml-file tmp.xml
Example 5.5. Searching for STONITH-related configuration items
# cibadmin --query | grep stonith <nvpair id="cib-bootstrap-options-stonith-action" name="stonith-action" value="reboot"/> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="1"/> <primitive id="child_DoFencing" class="stonith" type="external/vmware"> <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith"> <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith"> <lrm_resource id="child_DoFencing:1" type="external/vmware" class="stonith"> <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith"> <lrm_resource id="child_DoFencing:2" type="external/vmware" class="stonith"> <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith"> <lrm_resource id="child_DoFencing:3" type="external/vmware" class="stonith">
primitive
tag with id child_DoFencing
, you would run:
# cibadmin --delete --xml-text '<primitive id="child_DoFencing"/>'
Important
cib.xml
file directly. Pacemaker will detect such changes and refuse to use the configuration.
crm_shadow
creates a "shadow" copy of the configuration and arranges for all the command-line tools to use it.
crm_shadow --create
with a name of your choice, and follow the simple on-screen instructions. Shadow copies are identified with a name to make it possible to have more than one.
Warning
Example 5.6. Creating and displaying the active sandbox
# crm_shadow --create test Setting up shadow instance Type Ctrl-D to exit the crm_shadow shell shadow[test]: shadow[test] # crm_shadow --which test
--commit
option, or discard them using the --delete
option. Again, be sure to follow the on-screen instructions carefully!
crm_shadow
options and commands, invoke it with the --help
option.
Example 5.7. Use sandbox to make multiple changes all at once, discard them, and verify real configuration is untouched
shadow[test] # crm_failcount -r rsc_c001n01 -G scope=status name=fail-count-rsc_c001n01 value=0 shadow[test] # crm_standby --node c001n02 -v on shadow[test] # crm_standby --node c001n02 -G scope=nodes name=standby value=on shadow[test] # cibadmin --erase --force shadow[test] # cibadmin --query <cib crm_feature_set="3.0.14" validate-with="pacemaker-3.0" epoch="112" num_updates="2" admin_epoch="0" cib-last-written="Mon Jan 8 23:26:47 2018" update-origin="rhel7-1" update-client="crm_node" update-user="root" have-quorum="1" dc-uuid="1"> <configuration> <crm_config/> <nodes/> <resources/> <constraints/> </configuration> <status/> </cib> shadow[test] # crm_shadow --delete test --force Now type Ctrl-D to exit the crm_shadow shell shadow[test] # exit # crm_shadow --which No active shadow configuration defined # cibadmin -Q <cib crm_feature_set="3.0.14" validate-with="pacemaker-3.0" epoch="110" num_updates="2" admin_epoch="0" cib-last-written="Mon Jan 8 23:26:47 2018" update-origin="rhel7-1" update-client="crm_node" update-user="root" have-quorum="1"> <configuration> <crm_config> <cluster_property_set id="cib-bootstrap-options"> <nvpair id="cib-bootstrap-1" name="stonith-enabled" value="1"/> <nvpair id="cib-bootstrap-2" name="pe-input-series-max" value="30000"/>
crm_simulate
shows the results of the same logic the cluster itself uses to respond to a particular cluster configuration and status.
crm_simulate
can handle these compressed files directly, though if you want to edit the file, you’ll need to uncompress it first.
crm_shadow
to create a sandbox version of the CIB; the --live-check
option will use the shadow CIB if one is in effect.
Example 5.9. Simulate cluster response to current live CIB or shadow CIB
crm_simulate --simulate --live-check
--show-scores
option, you will also see all the scores that went into the decision-making. The node with the highest cumulative score for a resource will run it. You can look for -INFINITY
scores in particular to see where complete bans came into effect.
-VVVV
to get more detailed messages about what’s happening under the hood. You can add up to two more V’s even, but that’s usually useful only if you’re a masochist or tracing through the source code.
--dot-file
option. This relies on the separate Graphviz
[6] project.
Example 5.10. Generate a visual graph of cluster actions from a saved CIB
crm_simulate --simulate --xml-file $FILENAME --dot-file $FILENAME.dot dot $FILENAME.dot -Tsvg > $FILENAME.svg
$FILENAME.dot
will contain a GraphViz representation of the cluster’s response to your changes, including all actions with their ordering dependencies.
$FILENAME.svg
will be the same information in a standard graphical format that you can view in your browser or other app of choice. You could, of course, use other dot
options to generate other formats.
cibadmin
with either a shadow CIB or the CIB_file
environment variable set to the filename, or use higher-level tool support (see the man pages of the specific tool you’re using for how to perform actions on a saved CIB file rather than the live CIB).
crm_simulate
man page for more details.
crm_shadow --commit
, you can use crm_simulate
to see how the cluster will react to the changes.
crm_attribute
and attrd_updater
are confusingly similar tools with subtle differences.
attrd_updater
can query and update node attributes. crm_attribute
can query and update not only node attributes, but also cluster options, resource defaults, and operation defaults.
Table 5.1. Types of Node Attributes
Type | Recorded in CIB? | Recorded in attribute manager memory? | Survive full cluster restart? | Manageable by crm_attribute? | Manageable by attrd_updater? |
---|---|---|---|---|---|
permanent
|
yes
|
no
|
yes
|
yes
|
no
|
transient
|
yes
|
yes
|
no
|
yes
|
yes
|
private
|
no
|
yes
|
no
|
no
|
yes
|
crm_attribute
can manage permanent and transient node attributes, while attrd_updater
can manage transient and private node attributes.
attrd_updater
always contacts the Pacemaker attribute manager directly, while crm_attribute
will contact the attribute manager only for transient node attributes, and will instead modify the CIB directly for permanent node attributes (and for transient node attributes when unable to contact the attribute manager).
attrd_updater
can change an attribute’s "dampening" (whether changes are immediately flushed to the CIB or after a specified amount of time, to minimize disk writes for frequent changes), set private node attributes (which are never written to the CIB), and set attributes for nodes that don’t yet exist.
crm_attribute
can set permanent node attributes (which are only in the CIB and not managed by the attribute manager), and can be used with saved CIB files and shadow CIBs.
crm_failcount
: query or delete resource fail counts
crm_node
: manage cluster nodes
crm_report
: generate a detailed cluster report for bug submissions
crm_resource
: manage cluster resources
crm_standby
: manage standby status of nodes
crm_verify
: validate a CIB
stonith_admin
: manage fencing devices
/etc/sysconfig/pacemaker
or /etc/default/pacemaker
).
dateformat -%Y%m%d-%H
to the rotation configuration to avoid this.
notice: Calculated transition 19, saving inputs in /var/lib/pacemaker/pengine/pe-input-1463.bz2
crm_simulate
command, described in Section 5.5, “Simulate Cluster Activity with crm_simulate”, can be used to replay the file.
Table of Contents
Table 7.1. Upgrade Methods
Method | Available between all versions | Can be used with Pacemaker Remote nodes | Service outage during upgrade | Service recovery during upgrade | Exercises failover logic | Allows change of messaging layer [a] |
---|---|---|---|---|---|---|
yes
|
yes
|
always
|
N/A
|
no
|
yes
| |
no
|
yes
|
yes
|
yes
|
no
| ||
yes
|
no
|
only due to failure
|
no
|
no
|
yes
| |
[a]
Currently, Corosync version 2 and greater is the only supported cluster stack, but other stacks have been supported by past versions, and may be supported by future versions.
[b]
Any active resources will be moved off the node being upgraded, so there will be at least a brief outage unless all resources can be migrated "live".
|
crm_verify
tool.
crm_verify
tool.
Note
Table 7.2. Version Compatibility Table
Version being Installed | Oldest Compatible Version |
---|---|
Pacemaker 2.y.z
| |
Pacemaker 1.y.z
|
Pacemaker 1.0.0
|
Pacemaker 0.7.z
|
Pacemaker 0.6.z
|
[a]
Rolling upgrades from Pacemaker 1.1.z to 2.y.z are possible only if the cluster uses corosync version 2 or greater as its messaging layer, and the Cluster Information Base (CIB) uses schema 1.0 or higher in its validate-with property.
|
# crm_attribute --name maintenance-mode --update true
crm_verify
tool.
# crm_attribute --name maintenance-mode --delete
Note
cibadmin --upgrade
.
# crm_shadow --create shadow
# crm_verify --live-check
# cibadmin --upgrade
# crm_shadow --diff
# crm_shadow --edit
# crm_simulate --live-check --save-dotfile shadow.dot -S # dot -Tsvg shadow.dot -o shadow.svg
crm_simulate
and dot
.
# crm_shadow --commit shadow --force
Note
upgrade*.xsl
conversion scripts provided with the source code. These will often be installed in a location such as /usr/share/pacemaker
, or may be obtained from the source repository.
# xsltproc /path/to/upgrade06.xsl config06.xml > config10.xml
pacemaker.rng
script (from the same location as the xsl files).
# xmllint --relaxng /path/to/pacemaker.rng config10.xml
master
tag has been deprecated in favor of using a clone
tag with the new promotable
meta-attribute set to true
. "Master/slave" clone resources are now referred to as "promotable" clone resources, though it will take longer for the full terminology change to be completed.
Rules, instance_attributes, meta_attributes
and sets of operations can be defined once and referenced in multiple places.
cibadmin
help text.
master_slave
was renamed to master
attributes
container tag was removed
pre-req
has been renamed requires
interval
, start
/stop
must have it set to zero
stonith-enabled
option now defaults to true.
stonith-enabled
is true (or unset) and no STONITH resources have been defined
resource-failure-stickiness
has been replaced by migration-threshold
.
crm_config
.
Table of Contents
/usr/lib/ocf/resource.d/
so that they are not confused with (or overwritten by) the agents shipped by existing providers.
/usr/lib/ocf/resource.d/bigCorp/bigApp
and define a resource:
<primitive id="custom-app" class="ocf" provider="bigCorp" type="bigApp"/>
Table 8.1. Required Actions for OCF Agents
Action | Description | Instructions |
---|---|---|
start
|
Start the resource
| |
stop
|
Stop the resource
| |
monitor
|
Check the resource’s state
|
NOTE: The monitor script should test the state of the resource on the local machine only.
|
meta-data
|
Describe the resource
|
NOTE: This is not performed as root.
|
validate-all
|
Verify the supplied parameters
|
Table 8.2. Optional Actions for OCF Resource Agents
Action | Description | Instructions |
---|---|---|
promote
|
Promote the local instance of a promotable clone resource to the master (primary) state.
| |
demote
|
Demote the local instance of a promotable clone resource to the slave (secondary) state.
| |
notify
|
Used by the cluster to send the agent pre- and post-notification events telling the resource what has happened and will happen.
|
recover
, is not currently used by the cluster. It is intended to be a variant of the start
action that tries to recover a resource locally.
Table 8.3. Types of recovery performed by the cluster
Type | Description | Action Taken by the Cluster |
---|---|---|
soft
|
A transient error occurred
| |
hard
|
A non-transient error that may be specific to the current node occurred
| |
fatal
|
A non-transient error that will be common to all cluster nodes (e.g. a bad configuration was specified)
|
OCF_SUCCESS
) can be considered to have failed, if 0 was not the expected return value.
Table 8.4. OCF Return Codes and their Recovery Types
RC | OCF Alias | Description | RT |
---|---|---|---|
0
|
OCF_SUCCESS
|
soft
| |
1
|
OCF_ERR_GENERIC
|
soft
| |
2
|
OCF_ERR_ARGS
|
hard
| |
3
|
OCF_ERR_UNIMPLEMENTED
|
hard
| |
4
|
OCF_ERR_PERM
|
hard
| |
5
|
OCF_ERR_INSTALLED
|
hard
| |
6
|
OCF_ERR_CONFIGURED
|
fatal
| |
7
|
OCF_NOT_RUNNING
|
N/A
| |
8
|
OCF_RUNNING_MASTER
|
soft
| |
9
|
OCF_FAILED_MASTER
|
soft
| |
other
|
N/A
|
soft
|
multiple-active
property.
OCF_ERR_UNIMPLEMENTED
do not cause any type of recovery.
some_service
is configured correctly and currently inactive, the following sequence will help you determine if it is LSB-compatible:
# /etc/init.d/some_service start ; echo "result: $?"
# /etc/init.d/some_service status ; echo "result: $?"
# /etc/init.d/some_service start ; echo "result: $?"
# /etc/init.d/some_service stop ; echo "result: $?"
# /etc/init.d/some_service status ; echo "result: $?"
# /etc/init.d/some_service stop ; echo "result: $?"
Revision History | |||
---|---|---|---|
Revision 1-0 | Tue Jan 23 2018 | , | |
| |||
Revision 1-1 | Mon Jan 28 2019 | , | |
| |||
Revision 1-2 | Mon May 13 2019 | ||
| |||
Revision 1-3 | Tue Oct 15 2019 | ||
| |||
Revision 2-0 | Tue Nov 5 2019 | ||
| |||
Revision 2-1 | Sun Dec 22 2019 | ||
|