Product SiteDocumentation Site

Pacemaker Development

Working with the Pacemaker Code Base

Edition 2

Andrew Beekhof

Red Hat

Ken Gaillot

Red Hat

Jan Pokorný

Red Hat

Legal Notice

Copyright © 2016-2018 Andrew Beekhof.
The text of and illustrations in this document are licensed under version 4.0 or later of the Creative Commons Attribution-ShareAlike International Public License ("CC-BY-SA")[1].
In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
In addition to the requirements of this license, the following activities are looked upon favorably:
  1. If you are distributing Open Publication works on hardcopy or CD-ROM, you provide email notification to the authors of your intent to redistribute at least thirty days before your manuscript or media freeze, to give the authors time to provide updated documents. This notification should describe modifications, if any, made to the document.
  2. All substantive modifications (including deletions) be either clearly marked up in the document or else described in an attachment to the document.
  3. Finally, while it is not mandatory under this license, it is considered good form to offer a free copy of any hardcopy or CD-ROM expression of the author(s) work.


This document has guidelines and tips for developers interested in editing Pacemaker source code and submitting changes for inclusion in the project.

Table of Contents

1. Frequently Asked Questions
2. C Coding Guidelines
2.1. Style Guidelines
2.1.1. C Boilerplate
2.1.2. Line Formatting
2.1.3. Pointers
2.1.4. Function Definitions
2.1.5. Control Statements (if, else, while, for, switch)
2.1.6. Operators
2.2. Best Practices
2.2.1. New Struct and Enum Members
2.2.2. Documentation
2.2.3. Symbol Naming
2.2.4. Memory Allocation
2.2.5. Logging
2.2.6. Regular Expressions
2.2.7. vim Settings
3. Python Coding Guidelines
3.1. Python Boilerplate
3.2. Python Compatibility
3.2.1. Python Future Imports
3.2.2. Other Python Compatibility Requirements
3.2.3. Python Usages to Avoid
3.3. Formatting Python Code
4. Evolution of the project
4.1. Foreword
4.2. Common ancestor: heartbeat project
4.2.1. Influence of heartbeat on Pacemaker
4.3. Notable Restructuring Steps in the Codebase
A. Revision History

Chapter 1. Frequently Asked Questions

Who is this document intended for?
Anyone who wishes to read and/or edit the Pacemaker source code. Casual contributors should feel free to read just this FAQ, and consult other sections as needed.
Where is the source code for Pacemaker?
The source code for Pacemaker is kept on GitHub, as are all software projects under the ClusterLabs umbrella. Pacemaker uses Git for source code management. If you are a Git newbie, the gittutorial(7) man page is an excellent starting point. If you’re familiar with using Git from the command line, you can create a local copy of the Pacemaker source code with: git clone
What are the different Git branches and repositories used for?
  • The master branch is the primary branch used for development.
  • The 2.0 branch contains the latest official release, and normally does not receive any changes. During the release cycle, it will contain release candidates for the next official release, and will receive only bug fixes.
  • The 1.1 branch is similar to both the master and 2.0 branches, but for the 1.1 release series. The 1.1 branch receives only backports of certain bug fixes and backward-compatible features from the master branch. During the release cycle, it will contain release candidates for the next official 1.1 release.
  • The 1.0 repository is a frozen snapshot of the 1.0 release series, and is no longer developed.
  • Messages will be posted to the mailing list during the release cycle, with instructions about which branches to use when submitting requests.
How do I build from the source code?
See in the main checkout directory.
What coding style should I follow?
You’ll be mostly fine if you simply follow the example of existing code. When unsure, see the relevant section of this document for language-specific recommendations. Pacemaker has grown and evolved organically over many years, so you will see much code that doesn’t conform to the current guidelines. We discourage making changes solely to bring code into conformance, as any change requires developer time for review and opens the possibility of adding bugs. However, new code should follow the guidelines, and it is fine to bring lines of older code into conformance when modifying that code for other reasons.
How should I format my Git commit messages?
See existing examples in the git log. The first line should look like change-type: affected-code: explanation where change-type should be Fix or Bug for most bug fixes, Feature for new features, Log for changes to log messages or handling, Doc for changes to documentation or comments, or Test for changes in CTS and regression tests. You will sometimes see Low, Med (or Mid) and High used instead for bug fixes, to indicate the severity. The important thing is that only commits with Feature, Fix, Bug, or High will automatically be included in the change log for the next release. The affected-code is the name of the component(s) being changed, for example, controller or libcrmcommon (it’s more free-form, so don’t sweat getting it exact). The explanation briefly describes the change. The git project recommends the entire summary line stay under 50 characters, but more is fine if needed for clarity. Except for the most simple and obvious of changes, the summary should be followed by a blank line and then a longer explanation of why the change was made.
How can I test my changes?
Most importantly, Pacemaker has regression tests for most major components; these will automatically be run for any pull requests submitted through GitHub. Additionally, Pacemaker’s Cluster Test Suite (CTS) can be used to set up a test cluster and run a wide variety of complex tests. This document will have more detail on testing in the future.
What is Pacemaker’s license?
Except where noted otherwise in the file itself, the source code for all Pacemaker programs is licensed under version 2 or later of the GNU General Public License (GPLv2+), its headers and libraries under version 2.1 or later of the less restrictive GNU Lesser General Public License (LGPLv2.1+), its documentation under version 4.0 or later of the Creative Commons Attribution-ShareAlike International Public License (CC-BY-SA-4.0), and its init scripts under the Revised BSD license. If you find any deviations from this policy, or wish to inquire about alternate licensing arrangements, please e-mail the mailing list. Licensing issues are also discussed on the ClusterLabs wiki.
How can I contribute my changes to the project?
Contributions of bug fixes or new features are very much appreciated! Patches can be submitted as pull requests via GitHub (the preferred method, due to its excellent features), or e-mailed to the mailing list as an attachment in a format Git can import.
What if I still have questions?
Ask on the mailing list for development-related questions, or on the mailing list for general questions about using Pacemaker. Developers often also hang out on freenode’s #clusterlabs IRC channel.

Chapter 2. C Coding Guidelines

2.1. Style Guidelines

Pacemaker is a large, distributed project accepting contributions from developers with a wide range of skill levels and organizational affiliations, and maintained by multiple people over long periods of time. The guidelines in this section are not technically better than alternative approaches, but make project management easier.
Many of these simply ensure stylistic consistency, which makes reading, writing, and reviewing code easier.

2.1.1. C Boilerplate

Every C file should start with a short copyright notice listing the original author, like:
 * Copyright <YYYY[-YYYY]> Andrew Beekhof <>
 * This source code is licensed under <LICENSE> WITHOUT ANY WARRANTY.
The first <YYYY> is the year the code was originally created. [2] If the code is modified in later years, add -YYYY with the most recent year of modification.
<LICENSE> should follow the policy set forth in the COPYING file, generally one of "GNU General Public License version 2 or later (GPLv2+)" or "GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)".
Header files should additionally protect against multiple inclusion by defining a unique symbol.
#  define MY_HEADER_NAME__H

// header code here

#endif // MY_HEADER_NAME__H
Public API header files should additionally declare "C" compatibility for inclusion by C++, and give a Doxygen file description. For example:
#ifdef __cplusplus
extern "C" {

 * \file
 * \brief My brief description here
 * \ingroup core

// header code here

#ifdef __cplusplus

2.1.2. Line Formatting

  • Indentation must be 4 spaces, no tabs.
  • Do not leave trailing whitespace.
  • Lines should be no longer than 80 characters unless limiting line length significantly impacts readability.

2.1.3. Pointers

  • The * goes by the variable name, not the type:
char *foo;
  • Use a space before the * and after the closing parenthesis in a cast:
char *foo = (char *) bar;

2.1.4. Function Definitions

  • In the function definition, put the return type on its own line, and place the opening brace by itself on a line.
  • For functions with enough arguments that they must break to the next line, align arguments with the first argument.
  • When a function argument is a function itself, use the pointer form.
static int
function_name(int bar, const char *a, const char *b,
              const char *c, void (*d)())
  • If a function name gets really long, start the arguments on their own line with 8 spaces of indentation:
static int
        int bar, const char *a, const char *b,
        const char *c, const char *d)

2.1.5. Control Statements (if, else, while, for, switch)

  • The keyword is followed by one space, then left parenthesis without space, condition, right parenthesis, space, opening bracket on the same line. else and else if are on the same line with the ending brace and opening brace, separated by a space.
  • Always use braces around control statement blocks, even if they only contain one line. This makes code review diffs smaller if a line gets added in the future, and avoids any chance of bad indenting making a line incorrectly appear to be part of the block.
  • Do not put assignments in if or while conditionals. This ensures that the developer’s intent is always clear, making code reviews easier and reducing the chance of using assignment where comparison is intended.
a = f();
if (a < 0) {
} else if (some_other_condition) {
} else {
  • In a switch statement, case is indented one level, and the body of each case is indented by another level. The opening brace is on the same line as switch.
switch (expression) {
    case 0:
    case 1:

2.1.6. Operators

  • Operators have spaces from both sides.
  • Do not rely on operator precedence; use parentheses when mixing operators with different priority.
  • No space is used after opening parenthesis and before closing parenthesis.
x = a + b - (c * d);

2.2. Best Practices

The guidelines in this section offer technical advantages.

2.2.1. New Struct and Enum Members

In the public APIs, always add new struct members to the end of the struct. This allows us to maintain backward API/ABI compatibility (as long as the application being linked allocates structs via API functions).
This generally applies to enum values as well, as the compiler will define enum values to 0, 1, etc., in the order given, so inserting a value in the middle will change the numerical values of all later values, making them backward-incompatible. However, if enum numerical values are explicitly specified rather than left to the compiler, new values can be added anywhere.

2.2.2. Documentation

All public API header files, functions, structs, enums, etc., should be documented with Doxygen comment blocks, as Pacemaker’s online API documentation is automatically generated via Doxygen. It is helpful to document private symbols in the same way, with an \internal tag in the Doxygen comment.

2.2.3. Symbol Naming

  • All file and function names should be unique across the entire project, to allow for individual tracing via PCMK_trace_files and PCMK_trace_functions, as well as making detail logs easier to follow.
  • Any exposed symbols in libraries (non-static function names, type names, etc.) must begin with a prefix appropriate to the library, for example, crm_, pe_, st_, lrm_. This reduces the chance of naming collisions with software linked against the library.
  • Time intervals are sometimes represented in Pacemaker code as user-defined text specifications (e.g. "10s"), other times as an integer number of seconds or milliseconds, and still other times as a string representation of an integer number. Variables for these should be named with an indication of which is being used (e.g. interval_spec, interval_ms, or interval_ms_s instead of interval).

2.2.4. Memory Allocation

Always use calloc() rather than malloc(). It has no additional cost on modern operating systems, and reduces the severity of uninitialized memory usage bugs.

2.2.5. Logging

  • When format strings are used for derived data types whose implementation may vary across platforms (pid_t, time_t, etc.), the safest approach is to use %lld in the format string, and cast the value to (long long).

2.2.6. Regular Expressions

  • Use REG_NOSUB with regcomp() whenever possible, for efficiency.
  • Be sure to use regfree() appropriately.

2.2.7. vim Settings

Developers who use vim to edit source code can add the following settings to their ~/.vimrc file to follow Pacemaker C coding guidelines:
" follow Pacemaker coding guidelines when editing C source code files
filetype plugin indent on
au FileType c   setlocal expandtab tabstop=4 softtabstop=4 shiftwidth=4 textwidth=80
autocmd BufNewFile,BufRead *.h set filetype=c
let c_space_errors = 1

[2] See the U.S. Copyright Office’s "Compendium of U.S. Copyright Office Practices", particularly "Chapter 2200: Notice of Copyright", sections 2205.1(A) and 2205.1(F), or "Updating Copyright Notices" for a more readable summary.

Chapter 3. Python Coding Guidelines

3.1. Python Boilerplate

If a Python file is meant to be executed (as opposed to imported), it should have a .in extension, and its first line should be:
which will be replaced with the appropriate python executable when Pacemaker is built. To make that happen, add an AC_CONFIG_FILES() line to, and add the file name without .in to .gitignore (see existing examples).
After the above line if any, every Python file should start like this:

# Pacemaker targets compatibility with Python 2.7 and 3.2+
from __future__ import print_function, unicode_literals, absolute_import, division

__copyright__ = "Copyright <YYYY[-YYYY]> Andrew Beekhof <>"
If the file is meant to be directly executed, the first line (<SHEBANG>) should be #!/usr/bin/python. If it is meant to be imported, omit this line.
<BRIEF-DESCRIPTION> is obviously a brief description of the file’s purpose. The string may contain any other information typically used in a Python file docstring.
The import statement is discussed further in Section 3.2.1, “Python Future Imports”.
<YYYY> is the year the code was originally created. [3] If the code is modified in later years, add -YYYY with the most recent year of modification.
<LICENSE> should follow the policy set forth in the COPYING file, generally one of "GNU General Public License version 2 or later (GPLv2+)" or "GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)".

3.2. Python Compatibility

Pacemaker targets compatibility with Python 2.7, and Python 3.2 and later. These versions have added features to be more compatible with each other, allowing us to support both the 2 and 3 series with the same code. It is a good idea to test any changes with both Python 2 and 3.

3.2.1. Python Future Imports

The future imports used in Section 3.1, “Python Boilerplate” mean:
  • All print statements must use parentheses, and printing without a newline is accomplished with the end=' ' parameter rather than a trailing comma.
  • All string literals will be treated as Unicode (the u prefix is unnecessary, and must not be used, because it is not available in Python 3.2).
  • Local modules must be imported using from . import (rather than just import). To import one item from a local module, use from .modulename import (rather than from modulename import).
  • Division using / will always return a floating-point result (use // if you want the integer floor instead).

3.2.2. Other Python Compatibility Requirements

  • When specifying an exception variable, always use as instead of a comma (e.g. except Exception as e or except (TypeError, IOError) as e). Use e.args to access the error arguments (instead of iterating over or subscripting e).
  • Use in (not has_key()) to determine if a dictionary has a particular key.
  • Always use the I/O functions from the io module rather than the native I/O functions (e.g. rather than open()).
  • When opening a file, always use the t (text) or b (binary) mode flag.
  • When creating classes, always specify a parent class to ensure that it is a "new-style" class (e.g. class Foo(object): rather than class Foo:)
  • Be aware of the bytes type added in Python 3. Many places where strings are used in Python 2 use bytes or bytearrays in Python 3 (for example, the pipes used with subprocess.Popen()). Code should handle both possibilities.
  • Be aware that the items(), keys(), and values() methods of dictionaries return lists in Python 2 and views in Python 3. In many case, no special handling is required, but if the code needs to use list methods on the result, cast the result to list first.
  • Do not raise or catch strings as exceptions (e.g. raise "Bad thing").
  • Do not use the cmp parameter of sorting functions (use key instead, if needed) or the __cmp__() method of classes (implement rich comparison methods such as __lt__() instead, if needed).
  • Do not use the buffer type.
  • Do not use features not available in all targeted Python versions. Common examples include:
    • The html, ipaddress, and UserDict modules
    • The function
    • The subprocess.DEVNULL constant
    • subprocess module-specific exceptions

3.2.3. Python Usages to Avoid

Avoid the following if possible, otherwise research the compatibility issues involved (hacky workarounds are often available):
  • long integers
  • octal integer literals
  • mixed binary and string data in one data file or variable
  • metaclasses
  • locale.strcoll and locale.strxfrm
  • the configparser and ConfigParser modules
  • importing compatibility modules such as six (so we don’t have to add them to Pacemaker’s dependencies)

3.3. Formatting Python Code

  • Indentation must be 4 spaces, no tabs.
  • Do not leave trailing whitespace.
  • Lines should be no longer than 80 characters unless limiting line length significantly impacts readability. For Python, this limitation is flexible since breaking a line often impacts readability, but definitely keep it under 120 characters.
  • Where not conflicting with this style guide, it is recommended (but not required) to follow PEP 8.
  • It is recommended (but not required) to format Python code such that pylint --disable=line-too-long,too-many-lines,too-many-instance-attributes,too-many-arguments,too-many-statements produces minimal complaints (even better if you don’t need to disable all those checks).

[3] See the U.S. Copyright Office’s "Compendium of U.S. Copyright Office Practices", particularly "Chapter 2200: Notice of Copyright", sections 2205.1(A) and 2205.1(F), or "Updating Copyright Notices" for a more readable summary.

Chapter 4. Evolution of the project

4.1. Foreword

This section is currently not meant as a definite summary of how Pacemaker got into where it stands now, but rather to provide few valuable pointers an entusiasts (presumably software archeologist type of person) may find useful. Moreover, well-intentioned contributors to Pacemaker project may want to review them occasionally since, as the famous quote has it, "those who do not learn history are doomed to repeat it".
For anything more talkative with less emphasis on actual code, other places will serve better for the time being (and if not, should not be too hard to volunteer extensions to those writeups):

4.2. Common ancestor: heartbeat project

Pacemaker can be considered as a spin-off from ‘heartbeat’, original comprehensive HA suite started by Alan Robertson, and some portions of code are shared, at least on the conceptual level if not verbatim, till today, even if the effective percentage continually declines. Note that till Pacemaker 2.0, it also used to stand for one (and initially the only) of supported messaging back-ends (removal of this support made for one such notable drop of reused code), see also pre-2.0 commit 55ab749bf.
The codebase for heartbeat used to be hosted at, but since that does not appear working anymore, an archive checkout from 2016 is shared at as a dedicated read-only repository, and anchored there, the most notable commits are:

4.2.1. Influence of heartbeat on Pacemaker

On a closer look, we can identify these things in common:
  • extensive use of data types and functions of GLib
  • Cluster Testing System (CTS) is inherited from initial implementation by Alan Robertson

Revision History

Revision History
Revision 1-0Tue Jul 26 2016Ken Gaillot
Convert coding guidelines and developer FAQ to Publican document
Revision 1-1Mon Aug 29 2016Ken Gaillot
Add Python coding guidelines, and more about licensing
Revision 2-0Fri Jan 12 2018Ken Gaillot
Drop support for Python 2.6
Revision 2-1Tue Sep 18 2018Jan Pokorný
Start documenting notable evolutionary points
Revision 2-2Fri Dec 7 2018Ken Gaillot
Update FAQ and C guidelines