Product SiteDocumentation Site

Pacemaker 2.0

Configuration Explained

An A-Z guide to Pacemaker's Configuration Options

Edition 12

Andrew Beekhof

Primary author 
Red Hat

Ken Gaillot

Red Hat

Philipp Marek

Style and formatting updates. Indexing. 

Tanja Roth

Utilization chapter Resource Templates chapter Multi-Site Clusters chapter 

Lars Marowsky-Bree

Multi-Site Clusters chapter 

Yan Gao

Utilization chapter Resource Templates chapter Multi-Site Clusters chapter 

Thomas Schraitle

Utilization chapter Resource Templates chapter Multi-Site Clusters chapter 

Dejan Muhamedagic

Resource Templates chapter 

Legal Notice

Copyright © 2009-2019 Andrew Beekhof.
The text of and illustrations in this document are licensed under version 4.0 or later of the Creative Commons Attribution-ShareAlike International Public License ("CC-BY-SA")[1].
In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
In addition to the requirements of this license, the following activities are looked upon favorably:
  1. If you are distributing Open Publication works on hardcopy or CD-ROM, you provide email notification to the authors of your intent to redistribute at least thirty days before your manuscript or media freeze, to give the authors time to provide updated documents. This notification should describe modifications, if any, made to the document.
  2. All substantive modifications (including deletions) be either clearly marked up in the document or else described in an attachment to the document.
  3. Finally, while it is not mandatory under this license, it is considered good form to offer a free copy of any hardcopy or CD-ROM expression of the author(s) work.


The purpose of this document is to definitively explain the concepts used to configure Pacemaker. To achieve this, it will focus exclusively on the XML syntax used to configure Pacemaker's Cluster Information Base (CIB).

Table of Contents

1. Document Conventions
1.1. Typographic Conventions
1.2. Pull-quote Conventions
1.3. Notes and Warnings
2. We Need Feedback!
1. Read-Me-First
1.1. The Scope of this Document
1.2. What Is Pacemaker?
1.3. Cluster Architecture
1.4. Pacemaker Architecture
1.5. Node Redundancy Designs
2. Cluster-Wide Configuration
2.1. Configuration Layout
2.2. CIB Properties
2.3. Cluster Options
3. Cluster Nodes
3.1. Defining a Cluster Node
3.2. Where Pacemaker Gets the Node Name
3.3. Node Attributes
3.3.1. Setting and querying node attributes
3.3.2. Special node attributes
4. Cluster Resources
4.1. What is a Cluster Resource?
4.2. Resource Classes
4.2.1. Open Cluster Framework
4.2.2. Linux Standard Base
4.2.3. Systemd
4.2.4. Upstart
4.2.5. System Services
4.2.6. STONITH
4.2.7. Nagios Plugins
4.3. Resource Properties
4.4. Resource Options
4.4.1. Resource Meta-Attributes
4.4.2. Setting Global Defaults for Resource Meta-Attributes
4.4.3. Resource Instance Attributes
4.5. Resource Operations
4.5.1. Monitoring Resources for Failure
4.5.2. Monitoring Resources When Administration is Disabled
4.5.3. Setting Global Defaults for Operations
4.5.4. When Implicit Operations Take a Long Time
4.5.5. Multiple Monitor Operations
4.5.6. Disabling a Monitor Operation
5. Resource Constraints
5.1. Scores
5.1.1. Infinity Math
5.2. Deciding Which Nodes a Resource Can Run On
5.2.1. Location Properties
5.2.2. Asymmetrical "Opt-In" Clusters
5.2.3. Symmetrical "Opt-Out" Clusters
5.2.4. What if Two Nodes Have the Same Score
5.3. Specifying the Order in which Resources Should Start/Stop
5.3.1. Ordering Properties
5.3.2. Optional and mandatory ordering
5.4. Placing Resources Relative to other Resources
5.4.1. Colocation Properties
5.4.2. Mandatory Placement
5.4.3. Advisory Placement
5.4.4. Colocation by Node Attribute
5.5. Resource Sets
5.6. Ordering Sets of Resources
5.6.1. Ordered Set
5.6.2. Ordering Multiple Sets
5.6.3. Resource Set OR Logic
5.7. Colocating Sets of Resources
6. Alerts
6.1. Alert Agents
6.2. Alert Recipients
6.3. Alert Meta-Attributes
6.4. Alert Instance Attributes
6.5. Alert Filters
6.6. Using the Sample Alert Agents
6.7. Writing an Alert Agent
7. Rules
7.1. Rule Properties
7.2. Node Attribute Expressions
7.3. Time- and Date-Based Expressions
7.3.1. Date Specifications
7.3.2. Durations
7.3.3. Sample Time-Based Expressions
7.4. Using Rules to Determine Resource Location
7.4.1. Location Rules Based on Other Node Properties
7.4.2. Using score-attribute Instead of score
7.5. Using Rules to Control Resource Options
7.6. Using Rules to Control Cluster Options
7.7. Ensuring Time-Based Rules Take Effect
8. Advanced Configuration
8.1. Specifying When Recurring Actions are Performed
8.2. Handling Resource Failure
8.2.1. Failure Counts
8.2.2. Failure Response
8.3. Moving Resources
8.3.1. Moving Resources Manually
8.3.2. Moving Resources Due to Connectivity Changes
8.3.3. Migrating Resources
8.4. Tracking Node Health
8.4.1. Node Health Attributes
8.4.2. Node Health Strategy
8.4.3. Measuring Node Health
8.5. Reloading Services After a Definition Change
9. Advanced Resource Types
9.1. Groups - A Syntactic Shortcut
9.1.1. Group Properties
9.1.2. Group Options
9.1.3. Group Instance Attributes
9.1.4. Group Contents
9.1.5. Group Constraints
9.1.6. Group Stickiness
9.2. Clones - Resources That Can Have Multiple Active Instances
9.2.1. Anonymous versus Unique Clones
9.2.2. Promotable clones
9.2.3. Clone Properties
9.2.4. Clone Options
9.2.5. Clone Contents
9.2.6. Clone Instance Attributes
9.2.7. Clone Constraints
9.2.8. Clone Stickiness
9.2.9. Clone Resource Agent Requirements
9.2.10. Monitoring Promotable Clone Resources
9.2.11. Determining Which Instance is Promoted
9.3. Bundles - Isolated Environments
9.3.1. Bundle Properties
9.3.2. Docker Properties
9.3.3. rkt Properties
9.3.4. Bundle Network Properties
9.3.5. Bundle Storage Properties
9.3.6. Bundle Primitive
9.3.7. Bundle Node Attributes
9.3.8. Bundle Meta-Attributes
9.3.9. Limitations of Bundles
10. Reusing Parts of the Configuration
10.1. Reusing Resource Definitions
10.1.1. Configuring Resources with Templates
10.1.2. Using Templates in Constraints
10.1.3. Using Templates in Resource Sets
10.2. Reusing Rules, Options and Sets of Operations
10.3. Tagging Configuration Elements
10.3.1. Configuring Tags
10.3.2. Using Tags in Constraints and Resource Sets
11. Utilization and Placement Strategy
11.1. Utilization attributes
11.2. Placement Strategy
11.3. Allocation Details
11.3.1. Which node is preferred to get consumed first when allocating resources?
11.3.2. Which node has more free capacity?
11.3.3. Which resource is preferred to be assigned first?
11.4. Limitations and Workarounds
12.1. What Is STONITH?
12.2. What STONITH Device Should You Use?
12.3. Special Treatment of STONITH Resources
12.4. Unfencing
12.5. Configuring STONITH
12.5.1. Example STONITH Configuration
12.6. Advanced STONITH Configurations
12.6.1. Example Dual-Layer, Dual-Device Fencing Topologies
12.7. Remapping Reboots
13. Status — Here be dragons
13.1. Node Status
13.2. Transient Node Attributes
13.3. Operation History
13.3.1. Simple Operation History Example
13.3.2. Complex Operation History Example
14. Multi-Site Clusters and Tickets
14.1. Challenges for Multi-Site Clusters
14.2. Conceptual Overview
14.2.1. Ticket
14.2.2. Dead Man Dependency
14.2.3. Cluster Ticket Registry
14.2.4. Configuration Replication
14.3. Configuring Ticket Dependencies
14.4. Managing Multi-Site Clusters
14.4.1. Granting and Revoking Tickets Manually
14.4.2. Granting and Revoking Tickets via a Cluster Ticket Registry
14.4.3. General Management of Tickets
14.5. For more information
A. Sample Configurations
A.1. Empty
A.2. Simple
A.3. Advanced Configuration
B. Revision History

List of Figures

1.1. Example Cluster Stack
1.2. Internal Components
1.3. Active/Passive Redundancy
1.4. Shared Failover
1.5. N to N Redundancy
5.1. Visual representation of the four resources' start order for the above constraints
5.2. Visual representation of the start order for two ordered sets of unordered resources
5.3. Visual representation of the start order for the three sets defined above
5.4. Visual representation of the above example (resources are placed from left to right)

List of Tables

2.1. CIB Properties
2.2. Cluster Options
3.1. Node attributes with special significance
4.1. Properties of a Primitive Resource
4.2. Meta-attributes of a Primitive Resource
4.3. Properties of an Operation
5.1. Properties of a rsc_location Constraint
5.2. Properties of a rsc_order Constraint
5.3. Properties of a rsc_colocation Constraint
5.4. Properties of a resource_set
6.1. Meta-Attributes of an Alert
6.2. Environment variables passed to alert agents
7.1. Properties of a Rule
7.2. Properties of an Expression
7.3. Built-in node attributes
7.4. Properties of a Date Expression
7.5. Properties of a Date Specification
8.1. Common Options for a ping Resource
8.2. Allowed Values for Node Health Attributes
8.3. Node Health Strategies
9.1. Properties of a Group Resource
9.2. Properties of a Clone Resource
9.3. Clone-specific configuration options
9.4. Additional colocation constraint options for promotable clone resources
9.5. Additional colocation set options relevant to promotable clone resources
9.6. Additional ordered set options relevant to promotable clone resources
9.7. Role implications of OCF return codes
9.8. Environment variables supplied with Clone notify actions
9.9. Extra environment variables supplied for promotable clones
9.10. Properties of a Bundle
9.11. Properties of a Bundle’s Docker Element
9.12. Properties of a Bundle’s rkt Element
9.13. Properties of a Bundle’s Network Element
9.14. Properties of a Bundle’s Port-Mapping Element
9.15. Properties of a Bundle’s Storage-Mapping Element
12.1. Additional Properties of Fencing Resources
12.2. Properties of Fencing Levels
13.1. Authoritative Sources for State Information
13.2. Node Status Fields
13.3. Contents of an lrm_rsc_op job

List of Examples

2.1. An empty configuration
3.1. Example Corosync cluster node entry
3.2. Result of using crm_attribute to specify which kernel pcmk-1 is running
4.1. A system resource definition
4.2. An OCF resource definition
4.3. An LSB resource with cluster options
4.4. An example OCF resource with instance attributes
4.5. Displaying the metadata for the Dummy resource agent template
4.6. An OCF resource with a non-default start timeout
4.7. An OCF resource with a recurring health check
4.8. An OCF resource with custom timeouts for its implicit actions
4.9. An OCF resource with two recurring health checks, performing different levels of checks specified via OCF_CHECK_LEVEL.
4.10. Example of an OCF resource with a disabled health check
5.1. Opt-in location constraints for two resources
5.2. Opt-out location constraints for two resources
5.3. Constraints where a resource prefers two nodes equally
5.4. Optional and mandatory ordering constraints
5.5. Mandatory colocation constraint for two resources
5.6. Mandatory anti-colocation constraint for two resources
5.7. Advisory colocation constraint for two resources
5.8. A set of 3 resources
5.9. A chain of ordered resources
5.10. A chain of ordered resources expressed as a set
5.11. Ordered sets of unordered resources
5.12. Advanced use of set ordering - Three ordered sets, two of which are internally unordered
5.13. Resource Set "OR" logic: Three ordered sets, where the first set is internally unordered with "OR" logic
5.14. Colocation chain as individual constraints, where A is placed first, then B, then C, then D
5.15. Equivalent colocation chain expressed using resource_set
5.16. Using colocated sets to specify a shared dependency
5.17. Colocation in which the members of the middle set have no interdependencies, and the last set listed applies only to instances in the master role
6.1. Simple alert configuration
6.2. Alert configuration with recipient
6.3. Alert configuration with meta-attributes
6.4. Alert configuration with instance attributes
6.5. Alert configuration to receive only node events and fencing events
6.6. Alert configuration to be called when certain node attributes change
6.7. Sending cluster events as SNMP traps
6.8. Sending cluster events as e-mails
7.1. True if now is any time in the year 2005
7.2. Equivalent expression
7.3. 9am-5pm Monday-Friday
7.4. 9am-6pm Monday through Friday or anytime Saturday
7.5. 9am-5pm or 9pm-12am Monday through Friday
7.6. Mondays in March 2005
7.7. A full moon on Friday the 13th
7.8. Prevent myApacheRsc from running on c001n03
7.9. Prevent myApacheRsc from running on c001n03 - expanded version
7.10. A sample nodes section for use with score-attribute
7.11. Defining different resource options based on the node name
7.12. Change resource-stickiness during working hours
8.1. Specifying a Base for Recurring Action Intervals
8.2. An example ping cluster resource that checks node connectivity once every minute
8.3. Don’t run a resource on unconnected nodes
8.4. Run only on nodes connected to three or more ping targets.
8.5. Prefer the node with the most connected ping nodes
8.6. How the cluster translates the above location constraint
8.7. A more complex example of choosing a location based on connectivity
8.8. The DRBD agent’s logic for supporting reload
8.9. The DRBD Agent Advertising Support for the reload Operation
8.10. Parameter that can be changed using reload
9.1. A group of two primitive resources
9.2. How the cluster sees a group resource
9.3. Some constraints involving groups
9.4. A clone that runs a web server on all nodes
9.5. Some constraints involving clones
9.6. Constraints involving promotable clone resources
9.7. Colocate C and D with A’s and B’s master instances
9.8. Start C and D after first promoting A and B
9.9. Notification variables
9.10. Monitoring both states of a promotable clone resource
9.11. Explicitly preferring node1 to be promoted to master
9.12. A bundle for a containerized web server
10.1. Resource template for a migratable Xen virtual machine
10.2. Xen primitive resource using a resource template
10.3. Equivalent Xen primitive resource not using a resource template
10.4. Xen resource overriding template values
10.5. Referencing rules from other constraints
10.6. Referencing attributes, options, and operations from other resources
10.7. Tag referencing three resources
10.8. Constraint using a tag
10.9. Equivalent constraints without tags
11.1. Specifying CPU and RAM capacities of two nodes
11.2. Specifying CPU and RAM consumed by several resources
12.1. Obtaining a list of STONITH Parameters
12.2. An IPMI-based STONITH Resource
12.3. Fencing topology with different devices for different nodes
13.1. A bare-bones status entry for a healthy node cl-virt-1
13.2. A set of transient node attributes for node cl-virt-1
13.3. A record of the apcstonith resource
13.4. A monitor operation (determines current state of the apcstonith resource)
13.5. Resource history of a pingd clone with multiple jobs
14.1. Constraint that fences node if ticketA is revoked
14.2. Constraint that demotes rsc1 if ticketA is revoked
14.3. Ticket constraint for multiple resources
A.1. An Empty Configuration
A.2. A simple configuration with two nodes, some cluster options and a resource
A.3. An advanced configuration with groups, clones and STONITH