PostgreSQL Community Components on PGSQL.CC

pgBadger 13.x Release Notes

Mon, 01 Jan 0001 00:00:00 +0000

These entries preserve the complete upstream change record for pgBadger 13.x, newest first.

v13.2 · 2025-12-29

This is a maintenance release of pgBadger that fixes issues and applied patches reported by users since last release.

Fix normalization that was not handling properly balanced single-quoted strings along with escaped quotes inside. Thanks to Bertrand Bourgier for the report.
Fix placeholder requirements in the doc.
Fix case where no error sample log entries was reported. Thanks to john doe for the report.
Fix possible precedence problem between ! and %s. Thanks to Luca Santarelli and Philipp Trulson for the report.
Update pgFormatter code to v5.9
Add github CI action for testing on commit push.
Fix parsing of %r placeholder in log_line_prefix. Thanks to nike7o0 for the report.
Fix uninitialized value warning. Thanks to Ales Zeleny for the report.
Enhance docs of ssh-options for postgres log parsing with examples. Thanks to Ulrich Konrad for the patch.
Add command –ssh-sudo to run commands over ssh as sudo. Thanks to Andrew Jackson for the patch.
Fix possible precedence problem between ! and string eq. Thanks to Adrien Nayrat for the report.
Fix parsing of pgbouncer stats. Thanks to mrgtt for the report.

v13.1 · 2025-03-16

This is a maintenance release of pgBadger that fixes issues reported by users since last release and adds some new features:

Annotate Command (annotate)

Mon, 01 Jan 0001 00:00:00 +0000

Annotations included with the backup command can be added, modified, or removed afterwards using the annotate command.

Command Options

Backup Annotation Option (`--annotation`)

Annotate backup with user-defined key/value pairs.

Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.

Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output.

Dynamic Configuration Settings

Mon, 01 Jan 0001 00:00:00 +0000

Dynamic configuration is stored in the DCS (Distributed Configuration Store) and applied on all cluster nodes.

In order to change the dynamic configuration you can use either patronictl_edit_config tool or Patroni REST API.

loop_wait: the number of seconds the loop will sleep. Default value: 10, minimum possible value: 1
ttl: the TTL to acquire the leader lock (in seconds). Think of it as the length of time before initiation of the automatic failover process. Default value: 30, minimum possible value: 20
retry_timeout: timeout for DCS and PostgreSQL operation retries (in seconds). DCS or network issues shorter than this will not cause Patroni to demote the leader. Default value: 10, minimum possible value: 3

Warning

when changing values of loop_wait, retry_timeout, or ttl you have to follow the rule:

Features

Mon, 01 Jan 0001 00:00:00 +0000

Several levels of brutality when rotating connections:

Session pooling

Most polite method. When a client connects, a server connection will be assigned to it for the whole duration it stays connected. When the client disconnects, the server connection will be put back into pool. This mode supports all PostgreSQL features.

Transaction pooling

A server connection is assigned to a client only during a transaction. When PgBouncer notices that the transaction is over, the server will be put back into the pool. This mode breaks a few session-based features of PostgreSQL. You can use it only when the application cooperates by not using features that break. See the table below for incompatible features.

Statement pooling

Most aggressive method. This is transaction pooling with a twist: Multi-statement transactions are disallowed. This is meant to enforce “autocommit” mode on the client, mostly targeted at PL/Proxy.
Low memory requirements (2 kB per connection by default). This is because PgBouncer does not need to see full packets at once.

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

Patroni is a template for high availability (HA) PostgreSQL solutions using Python. Patroni originated as a fork of Governor, the project from Compose. It includes plenty of new features.

For additional background info, see:

PostgreSQL HA with Kubernetes and Patroni, talk by Josh Berkus at KubeCon 2016 (video)
Feb. 2016 Zalando Tech blog post

Development Status

Patroni is in active development and accepts contributions. See our Contributing section below for more details.

News

Mon, 01 Jan 0001 00:00:00 +0000

New Distribution Tarball

July 20, 2026

Starting with pgBackRest 2.59.0, every release includes a distribution tarball that makes building from source simpler. Unlike a checkout of the git repository, the tarball ships the generated source and the rendered documentation pre-built, so pgBackRest builds and installs without the code generation or documentation tooling that a repository checkout requires.

The tarball contains the pgBackRest source with the pre-generated code, the command reference man page, the HTML documentation, and a smoke test to verify the build. It builds with meson and ninja using only the usual pgBackRest libraries.

Starter Guide

Mon, 01 Jan 0001 00:00:00 +0000

YAML Configuration Settings

Mon, 01 Jan 0001 00:00:00 +0000

Global/Universal

thread_pool_size: size of thread pool used by Patroni to execute asynchronous tasks and communicate via REST API with other members during leader race or failsafe checks. Minimal value is 5, default value is 5.
thread_stack_size: specifies the stack size to be used for threads started by Patroni. Value must be aligned by 64kB. Minimal value is 64kB, default value (set by Patroni) is 512kB.
name: the name of the host. Must be unique for the cluster. The value __patroni_strict_sync_replica_placeholder__ is reserved for internal use by Patroni and cannot be used as a node name.
namespace: path within the configuration store where Patroni will keep information about the cluster. Default value: “/service”
scope: cluster name

pgBadger 12.x Release Notes

Mon, 01 Jan 0001 00:00:00 +0000

These entries preserve the complete upstream change record for pgBadger 12.x, newest first.

v12.4 · 2023-12-25

This is a maintenance release of pgBadger that fixes issues reported by users since last release.

Fix pgbouncer report with version 1.21. Thanks to Ales Zeleny for the patch.
Prevent parallelism perl file to be higher than the number of files. Thanks to maliangzhu for the report.
Fix regression test broken since v12.3. Thanks to ieshin for the report.
Fix cases where LOG entries where counted as ERROR log level entries. Thanks to Matti Linnanvuori for the report.

v12.3 · 2023-11-27

This is a maintenance release of pgBadger that fixes issues reported by users since last release. It also adds some new features:

Archive Get Command (archive-get)

Mon, 01 Jan 0001 00:00:00 +0000

This command is used by PostgreSQL to restore a backup, perform PITR, or as an alternative to streaming for keeping a replica up to date. WAL segments are required for PostgreSQL recovery or to maintain a replica.

When multiple repositories are configured, WAL will be fetched from the repositories in priority order (e.g. repo1, repo2, etc.). In general it is better if faster/cheaper storage has higher priority. If a repository is specified with the --repo option then only that repository will be searched.

Configuration: pgbouncer.ini

Mon, 01 Jan 0001 00:00:00 +0000

Description

The configuration file is in “ini” format. Section names are between [ and ]. Lines starting with ; or # are taken as comments and ignored. The characters ; and # are not recognized as special when they appear later in the line.

Generic settings

logfile

Specifies the log file. For daemonization (-d), either this or syslog need to be set.

Download and Installation

Mon, 01 Jan 0001 00:00:00 +0000

Choose a distribution channel

Channel	Use it for	Location
Official release	Source tarballs and release notes	GitHub Releases
RPM package	RPM-based Linux distributions	PostgreSQL Yum Repository
Debian/Ubuntu package	APT-based Linux distributions	PostgreSQL Apt Repository
Development source	Current unreleased code	darold/pgbadger

The documentation snapshot in this site is based on the 13.2 source tree. Always check the release page before downloading: the local snapshot is intentionally fixed, while upstream releases continue to change.

Installation

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requirements for Mac OS

To install requirements on a Mac, run the following:

brew install postgresql etcd haproxy libyaml python

Psycopg

Starting from psycopg2-2.8 the binary version of psycopg2 will no longer be installed by default. Installing it from the source code requires C compiler and postgres+python dev packages. Since in the python world it is not possible to specify dependency as psycopg2 OR psycopg2-binary you will have to decide how to install it.

Load Balancing Fundamentals

Mon, 01 Jan 0001 00:00:00 +0000

This document is an introduction to HAProxy for all those who don’t know it, as well as for those who want to re-discover it when they know older versions. Its primary focus is to provide users with all the elements to decide if HAProxy is the product they’re looking for or not. Advanced users may find here some parts of solutions to some ideas they had just because they were not aware of a given new feature. Some sizing information is also provided, the product’s lifecycle is explained, and comparisons with partially overlapping products are provided.

User Guide (Debian/Ubuntu)

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

This user guide is intended to be followed sequentially from beginning to end — each section depends on the last. For example, the Restore section relies on setup that is performed in the Quick Start section. Once pgBackRest is up and running then skipping around is possible but following the user guide in order is recommended the first time through.

Although the examples in this guide are targeted at Debian/Ubuntu and PostgreSQL 17, it should be fairly easy to apply the examples to any Unix distribution and PostgreSQL version. The only OS-specific commands are those to create, start, stop, and drop PostgreSQL clusters. The pgBackRest commands will be the same on any Unix system though the location of the executable may vary. While pgBackRest strives to operate consistently across versions of PostgreSQL, there are subtle differences between versions of PostgreSQL that may show up in this guide when illustrating certain examples, e.g. PostgreSQL path/file names and settings.

pgBadger 11.x Release Notes

Mon, 01 Jan 0001 00:00:00 +0000

These entries preserve the complete upstream change record for pgBadger 11.x, newest first.

v11.8 · 2022-04-08

This release of pgBadger fix some issues reported by users since past three months and especially two fixes on new log entries detection in incremental mode.

Fix detection of new log entries with timestamp when millisecond (%m) or epoch (%n) was used in log_line_prefix.
Fix detection of new log entries in local file when multiprocess was not used.

Here is the complete list of changes and acknowledgments:

What HAProxy Is and How It Works

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is written as “HAProxy” to designate the product, and as “haproxy” to designate the executable program, software package or a process. However, both are commonly used for both purposes, and are pronounced H-A-Proxy. Very early, “haproxy” used to stand for “high availability proxy” and the name was written in two separate words, though by now it means nothing else than “HAProxy”.

Archive Push Command (archive-push)

Mon, 01 Jan 0001 00:00:00 +0000

Accepts a WAL segment from PostgreSQL and archives it in each repository defined by the indexed repo-path option (see the Repository section for information on configuring repositories). The WAL segment may be pushed immediately to the archive or stored locally depending on the value of archive-async. With multiple repositories configured, archive-push will attempt to push to as many repositories as possible.

The archive-push is intended to be configured and called by PostgreSQL. See Configure Archiving for an example.

Command-Line Reference

Mon, 01 Jan 0001 00:00:00 +0000

This page preserves the command reference generated from the pinned upstream source. Option names and help text remain verbatim so they can be compared directly with pgbadger --help.

Usage and options

Usage: pgbadger [options] logfile […]

PostgreSQL log analyzer with fully detailed reports and graphs.

Arguments:

logfile can be a single log file, a list of files, or a shell command
returning a list of files. If you want to pass log content from stdin
use - as filename. Note that input from stdin will not work with csvlog.

Options:

Environment Configuration Settings

Mon, 01 Jan 0001 00:00:00 +0000

It is possible to override some of the configuration parameters defined in the Patroni configuration file using the system environment variables. This document lists all environment variables handled by Patroni. The values set via those variables always take precedence over the ones set in the Patroni configuration file.

Global/Universal

PATRONI_CONFIGURATION: it is possible to set the entire configuration for the Patroni via PATRONI_CONFIGURATION environment variable. In this case any other environment variables will not be considered!
PATRONI_THREAD_POOL_SIZE: size of thread pool used by Patroni to execute asynchronous tasks and communicate via REST API with other members during leader race or failsafe checks. Minimal value is 5, default value is 5.
PATRONI_THREAD_STACK_SIZE: specifies the stack size to be used for threads started by Patroni. Value must be aligned by 64kB. Minimal value is 64kB, default value (set by Patroni) is 512kB.
PATRONI_NAME: name of the node where the current instance of Patroni is running. Must be unique for the cluster. The value __patroni_strict_sync_replica_placeholder__ is reserved for internal use by Patroni and cannot be used as a node name.
PATRONI_NAMESPACE: path within the configuration store where Patroni will keep information about the cluster. Default value: “/service”
PATRONI_SCOPE: cluster name
PG_MALLOC_ARENA_MAX: custom value for MALLOC_ARENA_MAX environment variable for postmaster process. If not set, postmaster will inherit MALLOC_ARENA_MAX value.

Log

PATRONI_LOG_TYPE: sets the format of logs. Can be either plain or json. To use json format, you must have the jsonlogger installed. The default value is plain.
PATRONI_LOG_LEVEL: sets the general logging level. Default value is INFO (see the docs for Python logging)
PATRONI_LOG_TRACEBACK_LEVEL: sets the level where tracebacks will be visible. Default value is ERROR. Set it to DEBUG if you want to see tracebacks only if you enable PATRONI_LOG_LEVEL=DEBUG.
PATRONI_LOG_FORMAT: sets the log formatting string. If the log type is plain, the log format should be a string. Refer to the LogRecord attributes for available attributes. If the log type is json, the log format can be a list in addition to a string. Each list item should correspond to LogRecord attributes. Be cautious that only the field name is required, and the %( and ) should be omitted. If you wish to print a log field with a different key name, use a dictionary where the dictionary key is the log field, and the value is the name of the field you want to be printed in the log. Default value is %(asctime)s %(levelname)s: %(message)s
PATRONI_LOG_DATEFORMAT: sets the datetime formatting string. (see the formatTime() documentation)
PATRONI_LOG_STATIC_FIELDS: add additional fields to the log. This option is only available when the log type is set to json. Example PATRONI_LOG_STATIC_FIELDS="{app: patroni}"
PATRONI_LOG_MAX_QUEUE_SIZE: Patroni is using two-step logging. Log records are written into the in-memory queue and there is a separate thread which pulls them from the queue and writes to stderr or file. The maximum size of the internal queue is limited by default by 1000 records, which is enough to keep logs for the past 1h20m.
PATRONI_LOG_DIR: Directory to write application logs to. The directory must exist and be writable by the user executing Patroni. If you set this env variable, the application will retain 4 25MB logs by default. You can tune those retention values with PATRONI_LOG_FILE_NUM and PATRONI_LOG_FILE_SIZE (see below).
PATRONI_LOG_MODE: Permissions for log files (for example, 0644). If not specified, permissions will be set based on the current umask value.
PATRONI_LOG_FILE_NUM: The number of application logs to retain.
PATRONI_LOG_FILE_SIZE: Size of patroni.log file (in bytes) that triggers a log rolling.
PATRONI_LOG_LOGGERS: Redefine logging level per python module. Example PATRONI_LOG_LOGGERS="{patroni.postmaster: WARNING, urllib3: DEBUG}"
PATRONI_LOG_DEDUPLICATE_HEARTBEAT_LOGS: If set to true, successive heartbeat logs that are identical shall not be output. Default value is false.

Warning

The time the HA loop executes at can be very valuable information in diagnosing failovers due to resource exhaustion and similar problems. When PATRONI_LOG_DEDUPLICATE_HEARTBEAT_LOGS is set to true there will be no log generated for the HA loop execution (unless the leader changes) and hence this potentially useful information will not be available from the logs.

Usage: pgbouncer command

Mon, 01 Jan 0001 00:00:00 +0000

Synopsis

pgbouncer [-d][-R][-v][-u user] <pgbouncer.ini>
pgbouncer -V|-h

On Windows, the options are:

pgbouncer.exe [-v][-u user] <pgbouncer.ini>
pgbouncer.exe -V|-h

Additional options for setting up a Windows service:

pgbouncer.exe --regservice   <pgbouncer.ini>
pgbouncer.exe --unregservice <pgbouncer.ini>

Description

pgbouncer is a PostgreSQL connection pooler. Any target application can be connected to pgbouncer as if it were a PostgreSQL server, and pgbouncer will create a connection to the actual server, or it will reuse one of its existing connections.

User Guide (RHEL)

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Although the examples in this guide are targeted at RHEL and PostgreSQL 14, it should be fairly easy to apply the examples to any Unix distribution and PostgreSQL version. The only OS-specific commands are those to create, start, stop, and drop PostgreSQL clusters. The pgBackRest commands will be the same on any Unix system though the location of the executable may vary. While pgBackRest strives to operate consistently across versions of PostgreSQL, there are subtle differences between versions of PostgreSQL that may show up in this guide when illustrating certain examples, e.g. PostgreSQL path/file names and settings.

pgBadger 10.x Release Notes

Mon, 01 Jan 0001 00:00:00 +0000

These entries preserve the complete upstream change record for pgBadger 10.x, newest first.

v10.3 · 2019-02-14

This release of pgBadger is a maintenance release that fixes some log format autodetection issues another pgBouncer log parsing issue reported by users. There is also a new feature:

The -o | --outfile option can now be used multiple time to dump
output in several format in a single command. For example:
    pgbadger -o out.html -o out.json /log/pgsql-11.log
will create two reports in html and json format saved in the
two corresponding files.

There’s also some bugs fixes and features enhancements.

Backup Command (backup)

Mon, 01 Jan 0001 00:00:00 +0000

When multiple repositories are configured, pgBackRest will backup to the highest priority repository (e.g. repo1) unless the --repo option is specified.

pgBackRest does not have a built-in scheduler so it’s best to run it from cron or some other scheduling mechanism.

See Perform a Backup for more details and examples.

Command Options

Backup Annotation Option (`--annotation`)

Annotate backup with user-defined key/value pairs.

Basic Features

Mon, 01 Jan 0001 00:00:00 +0000

This section will enumerate a number of features that HAProxy implements, some of which are generally expected from any modern load balancer, and some of which are a direct benefit of HAProxy’s architecture. More advanced features will be detailed in the next section.

3.3.1. Basic features : Proxying

Proxying is the action of transferring data between a client and a server over two independent connections. The following basic features are supported by HAProxy regarding proxying and connection management:

PgBouncer compilation and installation

Mon, 01 Jan 0001 00:00:00 +0000

Building

PgBouncer depends on few things to get compiled:

GNU Make 3.81+
Libevent 2.0+
pkg-config
OpenSSL 1.0.1+ for TLS support
(optional) c-ares as alternative to Libevent’s evdns
(optional) LDAP libraries
(optional) PAM libraries

When dependencies are installed just run:

$ ./configure --prefix=/usr/local
$ make
$ make install

If you are building from Git, or are building for Windows, please see separate build instructions below.

DNS lookup support

PgBouncer does host name lookups at connect time instead of just once at configuration load time. This requires an asynchronous DNS implementation. The following table shows supported backends and their probing order:

Patroni REST API

Mon, 01 Jan 0001 00:00:00 +0000

Patroni has a rich REST API, which is used by Patroni itself during the leader race, by the patronictl tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring. Below you will find the list of Patroni REST API endpoints.

Health check endpoints

For all health check GET requests Patroni returns a JSON document with the status of the node, along with the HTTP status code. If you don’t want or don’t need the JSON document, you might consider using the HEAD or OPTIONS method instead of GET.

PostgreSQL Logging Configuration

Mon, 01 Jan 0001 00:00:00 +0000

pgBadger can only report information that PostgreSQL writes to the log. Start with a parseable prefix and a deliberate statement-logging policy, then add the operational events you want to analyze.

Minimum query logging

To include query text and duration, enable duration-based statement logging:

log_min_duration_statement = 0

0 logs every completed statement. On a busy server, choose a higher threshold in milliseconds to control log volume. Measure the overhead and storage growth before enabling a low threshold in production.

pgBadger 9.x Release Notes

Mon, 01 Jan 0001 00:00:00 +0000

These entries preserve the complete upstream change record for pgBadger 9.x, newest first.

v9.2 · 2017-07-27

This release of pgBadger is a maintenance release that adds some new features.

Add report of checkpoint distance and estimate.
Add support of AWS Redshift keywords to SQL code beautifier.
Add autodetection of log format in remote mode to allow remote parsing of pgbouncer log file together with PostgreSQL log file.

There’s also some bugs fixes and features enhancements.

Check Command (check)

Mon, 01 Jan 0001 00:00:00 +0000

The check command validates that pgBackRest and the archive_command setting are configured correctly for archiving and backups for the specified stanza. It will attempt to check all repositories and databases that are configured for the host on which the command is run. It detects misconfigurations, particularly in archiving, that result in incomplete backups because required WAL segments did not reach the archive. The command can be run on the PostgreSQL or repository host. The command may also be run on the standby host, however, since pg_switch_xlog()/pg_switch_wal() cannot be performed on the standby, the command will only test the repository configuration.

Configuration Reference

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

pgBackRest can be used entirely with command-line parameters but a configuration file is more practical for installations that are complex or set a lot of options. The default location for the configuration file is /etc/pgbackrest/pgbackrest.conf. If no file exists in that location then the old default of /etc/pgbackrest.conf will be checked.

The following option types are used:

String: A text string, commonly an identifier, password, etc.

Source Releases Download

Mon, 01 Jan 0001 00:00:00 +0000

PgBouncer 1.25

File	Date	Size	SHA256
pgbouncer-1.25.2.tar.gz	2026-05-08	865371 bytes	sha256
pgbouncer-1.25.1.tar.gz	2025-12-03	864801 bytes	sha256
pgbouncer-1.25.0.tar.gz	2025-11-09	863322 bytes	sha256

PgBouncer 1.24

File	Date	Size	SHA256
pgbouncer-1.24.1.tar.gz	2025-04-16	717796 bytes	sha256
pgbouncer-1.24.0.tar.gz	2025-01-10	706573 bytes	sha256

PgBouncer 1.23

File	Date	Size	SHA256
pgbouncer-1.23.1.tar.gz	2024-08-02	700025 bytes	sha256
pgbouncer-1.23.0.tar.gz	2024-07-03	694845 bytes	sha256

PgBouncer 1.22

File	Date	Size	SHA256
pgbouncer-1.22.1.tar.gz	2024-03-04	677351 bytes	sha256
pgbouncer-1.22.0.tar.gz	2024-01-31	670589 bytes	sha256

PgBouncer 1.21

File	Date	Size	SHA256
pgbouncer-1.21.0.tar.gz	2023-10-16	668211 bytes	sha256

PgBouncer 1.20

File	Date	Size	SHA256
pgbouncer-1.20.1.tar.gz	2023-08-09	638844 bytes	sha256
pgbouncer-1.20.0.tar.gz	2023-07-20	638020 bytes	sha256

PgBouncer 1.19

File	Date	Size	SHA256
pgbouncer-1.19.1.tar.gz	2023-05-31	623569 bytes	sha256
pgbouncer-1.19.0.tar.gz	2023-05-04	616947 bytes	sha256

PgBouncer 1.18

File	Date	Size	SHA256
pgbouncer-1.18.0.tar.gz	2022-12-12	600825 bytes	sha256

PgBouncer 1.17

File	Date	Size	SHA256
pgbouncer-1.17.0.tar.gz	2022-03-23	598294 bytes	sha256

PgBouncer 1.16

File	Date	Size	SHA256
pgbouncer-1.16.1.tar.gz	2021-11-11	591450 bytes	sha256
pgbouncer-1.16.0.tar.gz	2021-08-09	592136 bytes	sha256

PgBouncer 1.15

File	Date	Size	SHA256
pgbouncer-1.15.0.tar.gz	2020-11-19	588042 bytes	sha256

PgBouncer 1.14

File	Date	Size	SHA256
pgbouncer-1.14.0.tar.gz	2020-06-11	578955 bytes	sha256

PgBouncer 1.13

File	Date	Size	SHA256
pgbouncer-1.13.0.tar.gz	2020-04-27	574955 bytes	sha256

PgBouncer 1.12

File	Date	Size	SHA256
pgbouncer-1.12.0.tar.gz	2019-10-17	567465 bytes	sha256

PgBouncer 1.11

File	Date	Size	SHA256
pgbouncer-1.11.0.tar.gz	2019-08-27	571414 bytes	sha256

PgBouncer 1.10

File	Date	Size	SHA256
pgbouncer-1.10.0.tar.gz	2019-07-01	480571 bytes	sha256

PgBouncer 1.9

File	Date	Size	SHA256
pgbouncer-1.9.0.tar.gz	2018-08-13	469300 bytes	sha256

PgBouncer 1.8

File	Date	Size	SHA256
pgbouncer-1.8.1.tar.gz	2017-12-20	465930 bytes	sha256
pgbouncer-1.8.tar.gz	2017-12-19	465612 bytes	sha256

PgBouncer 1.7

File	Date	Size	SHA256
pgbouncer-1.7.2.tar.gz	2016-02-26	462374 bytes	sha256
pgbouncer-1.7.1.tar.gz	2016-02-18	461903 bytes	sha256
pgbouncer-1.7.tar.gz	2015-12-18	459080 bytes	sha256

PgBouncer 1.6

File	Date	Size	SHA256
pgbouncer-1.6.1.tar.gz	2015-09-03	431076 bytes	sha256
pgbouncer-1.6.tar.gz	2015-08-01	412700 bytes	sha256

PgBouncer 1.5

File	Date	Size	SHA256
pgbouncer-1.5.5.tar.gz	2015-04-09	336145 bytes	sha256
pgbouncer-1.5.4.tar.gz	2012-11-28	339610 bytes	sha256
pgbouncer-1.5.3.tar.gz	2012-09-12	339013 bytes	sha256
pgbouncer-1.5.2.tar.gz	2012-05-29	335338 bytes	sha256
pgbouncer-1.5.1.tar.gz	2012-04-17	334413 bytes	sha256
pgbouncer-1.5.tar.gz	2012-01-05	411488 bytes	sha256

PgBouncer 1.4

File	Date	Size	SHA256
pgbouncer-1.4.2.tgz	2011-06-16	283204 bytes	sha256
pgbouncer-1.4.1.tgz	2011-04-01	282728 bytes	sha256
pgbouncer-1.4.tgz	2011-01-11	231691 bytes	sha256

PgBouncer 1.3

File	Date	Size	SHA256
pgbouncer-1.3.4.tgz	2010-09-09	167957 bytes	sha256
pgbouncer-1.3.3.tgz	2010-05-10	167476 bytes	sha256
pgbouncer-1.3.2.tgz	2010-03-15	166756 bytes	sha256
pgbouncer-1.3.1.tgz	2009-07-06	161518 bytes	sha256
pgbouncer-1.3.tgz	2009-02-18	160154 bytes	sha256

PgBouncer 1.2

File	Date	Size	SHA256
pgbouncer-1.2.3.tgz	2008-08-08	145372 bytes	sha256
pgbouncer-1.2.2.tgz	2008-08-06	145017 bytes	sha256
pgbouncer-1.2.1.tgz	2008-08-04	144903 bytes	sha256
pgbouncer-1.2.tgz	2008-07-29	143915 bytes	sha256

PgBouncer 1.1

File	Date	Size	SHA256
pgbouncer-1.1.2.tgz	2007-12-10	122054 bytes	sha256
pgbouncer-1.1.1.tgz	2007-10-26	121042 bytes	sha256
pgbouncer-1.1.tgz	2007-10-09	120462 bytes	sha256

PgBouncer 1.0

File	Date	Size	SHA256
pgbouncer-1.0.8.tgz	2007-06-18	93636 bytes	sha256
pgbouncer-1.0.7.tgz	2007-04-19	93086 bytes	sha256
pgbouncer-1.0.6.tgz	2007-04-12	92244 bytes	sha256
pgbouncer-1.0.5.tgz	2007-04-11	91934 bytes	sha256
pgbouncer-1.0.4.tgz	2007-04-11	91889 bytes	sha256
pgbouncer-1.0.3.tgz	2007-04-11	91489 bytes	sha256
pgbouncer-1.0.2.tgz	2007-03-28	90555 bytes	sha256
pgbouncer-1.0.1.tgz	2007-03-15	89609 bytes	sha256
pgbouncer-1.0.tgz	2007-03-13	88587 bytes	sha256

Binary Packages

Various OS distributions have their native package/port of PgBouncer. So it might be good to first check if it is already available on your OS.

Parallel Processing

Mon, 01 Jan 0001 00:00:00 +0000

pgBadger has two complementary multiprocessing modes. Choose according to the shape of the input, not simply the number of CPUs.

Option	Parallel unit	Best fit	Main constraint
`-j N` / `--jobs N`	chunks of one log file	one large, seekable log	chunk boundaries can duplicate or omit a small number of queries
`-J N` / `--Jobs N`	whole log files	many independent logs	useful only when enough files are available to keep workers busy

Split one large file with `-j`

$ pgbadger -j 8 /var/log/postgresql/postgresql.log

The upstream algorithm divides each file into N byte ranges, forks one parser per range, writes temporary binary statistics, then merges those statistics into the final report.

patronictl

Mon, 01 Jan 0001 00:00:00 +0000

Patroni has a command-line interface named patronictl, which is used basically to interact with Patroni’s REST API and with the DCS. It is intended to make it easier to perform operations in the cluster, and can easily be used by humans or scripts.

Configuration

patronictl uses 3 sections of the configuration:

ctl: how to authenticate against the Patroni REST API, and how to validate the server identity. Refer to ctl settings for more details;
restapi: how to authenticate against the Patroni REST API, and how to validate the server identity. Only used if ctl configuration is not enough. patronictl is mainly interested in restapi.authentication section (in case ctl.authentication is missing) and restapi.cafile setting (in case ctl.cacert is missing). Refer to REST API settings for more details;
DCS (e.g. etcd): how to contact and authenticate against the DCS used by Patroni.

Those configuration options can come either from environment variables or from a configuration file. Look for the above sections in Environment Configuration Settings or YAML Configuration Settings to understand how you can set the options for them through environment variables or through a configuration file.

Standard Features

Mon, 01 Jan 0001 00:00:00 +0000

In this section, some features that are very commonly used in HAProxy but are not necessarily present on other load balancers are enumerated.

3.4.1. Standard features : Sampling and converting information

HAProxy supports information sampling using a wide set of “sample fetch functions”. The principle is to extract pieces of information known as samples, for immediate use. This is used for stickiness, to build conditions, to produce information in logs or to enrich HTTP headers.

Advanced Features

Mon, 01 Jan 0001 00:00:00 +0000

3.5.1. Advanced features : Management

HAProxy is designed to remain extremely stable and safe to manage in a regular production environment. It is provided as a single executable file which doesn’t require any installation process. Multiple versions can easily coexist, meaning that it’s possible (and recommended) to upgrade instances progressively by order of importance instead of migrating all of them at once. Configuration files are easily versioned. Configuration checking is done off-line so it doesn’t require to restart a service that will possibly fail. During configuration checks, a number of advanced mistakes may be detected (e.g. a rule hiding another one, or stickiness that will not work) and detailed warnings and configuration hints are proposed to fix them. Backwards configuration file compatibility goes very far away in time, with version 1.5 still fully supporting configurations for versions 1.1 written 13 years before, and 1.6 only dropping support for almost unused, obsolete keywords that can be done differently. The configuration and software upgrade mechanism is smooth and non disruptive in that it allows old and new processes to coexist on the system, each handling its own connections. System status, build options, and library compatibility are reported on startup.

Expire Command (expire)

Mon, 01 Jan 0001 00:00:00 +0000

pgBackRest does full backup rotation based on the retention type which can be a count or a time period. When a count is specified, then expiration is not concerned with when the backups were created but with how many must be retained. Differential backups are count-based but will always be expired when the full backup they depend on is expired. Incremental backups are not expired by retention independently — they are always expired with their related full or differential backup. See sections Full Backup Retention and Differential Backup Retention for details and examples.

Incremental Reports

Mon, 01 Jan 0001 00:00:00 +0000

Incremental mode stores parsed statistics in binary form, then builds one HTML report per day, a cumulative report per week, and a calendar-style index linking them together. It is intended for repeated processing of rotated logs without counting the same entries again.

Build daily and weekly reports

Run pgBadger after the daily log rotation and provide a persistent output directory:

0 4 * * * /usr/bin/pgbadger -I -q /var/log/postgresql/postgresql.log.1 -O /var/www/pg_reports/

-I enables incremental mode and -O selects the directory that holds the binary state, calendar index, and generated reports. pgBadger maintains its own incremental state in that directory, so --last-parsed is unnecessary unless you deliberately want the state file elsewhere.

Release Notes

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

pgBackRest release numbers consist of two parts, major and minor. A major release may break compatibility with the prior major release, but v2 releases are fully compatible with v1 repositories and will accept all v1 options. Minor releases can include bug fixes and features but do not change the repository format and strive to avoid changing options and naming. Documentation for the v1 release can be found here. The notes for a release may also contain “Additional Notes” but changes in this section are only to documentation or the test suite and have no direct impact on the pgBackRest codebase.

Replica imaging and bootstrap

Mon, 01 Jan 0001 00:00:00 +0000

Patroni allows customizing creation of a new replica. It also supports defining what happens when the new empty cluster is being bootstrapped. The distinction between two is well defined: Patroni creates replicas only if the initialize key is present in DCS for the cluster. If there is no initialize key - Patroni calls bootstrap exclusively on the first node that takes the initialize key lock.

Bootstrap

PostgreSQL provides initdb command to initialize a new cluster and Patroni calls it by default. In certain cases, particularly when creating a new cluster as a copy of an existing one, it is necessary to replace a built-in method with custom actions. Patroni supports executing user-defined scripts to bootstrap new clusters, supplying some required arguments to them, i.e. the name of the cluster and the path to the data directory. This is configured in the bootstrap section of the Patroni configuration. For example:

Changelog

Mon, 01 Jan 0001 00:00:00 +0000

PgBouncer 1.25.x

2026-05-08 - PgBouncer 1.25.2 - “Human touch with fresh twist in title race full of uncertainties”

Security
- Fix CVE-2026-6664: An integer overflow in network packet parsing code in PgBouncer before 1.25.2 bypasses a boundary check and can lead to a crash. An unauthenticated remote attacker can crash PgBouncer with a malformed SCRAM authentication packet.
- Fix CVE-2026-6665: The SCRAM code in PgBouncer before 1.25.2 did not check the return value of strlcat() correctly when building the contents of the SCRAM client-final-message. A malicious backend that sends a SCRAM server-final-message with a long nonce can trigger a stack overflow.
- Fix CVE-2026-6666: A possible null pointer reference in PgBouncer before 1.25.2 could lead to a crash if a server sends an error response without an SQLSTATE field.
- Fix CVE-2026-6667: PgBouncer before 1.25.2 did not perform an appropriate authorization check for the KILL_CLIENT admin command. All users with access to the administration console, which itself requires authorization, could run this command. It should only be allowed for users listed in the admin_users parameter.
Fixes

Frequently Asked Questions

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Frequently Asked Questions are intended to provide details for specific questions that may or may not be covered in the User Guide, Configuration, or Command reference. If you are unable to find details for your specific issue here, remember that the pgBackRest Issues List in GitHub is also a valuable resource.

What if I get the “could not find WAL segment” error?

The cause of this error can be a result of many different issues, some of which may be:

Help Command (help)

Mon, 01 Jan 0001 00:00:00 +0000

Three levels of help are provided. If no command is specified then general help will be displayed. If a command is specified (e.g. pgbackrest help backup) then a full description of the command will be displayed along with a list of valid options. If an option is specified in addition to a command (e.g. pgbackrest help backup type) then a full description of the option as it applies to the command will be displayed.

Output Formats

Mon, 01 Jan 0001 00:00:00 +0000

pgBadger selects output from the filename extension or from -x / --extension. Use -o / --outfile more than once to create multiple formats from the same parse.

Format	Typical extension	Best use
HTML	`.html`	interactive, human-readable report with charts
Text	`.txt`	terminal review and plain archival output
Binary	`.bin`	mergeable intermediate statistics and report rebuilds
JSON	`.json`	integration with other software; requires `JSON::XS`
Raw CSV	chosen output file	row-oriented extraction with `--dump-raw-csv`

HTML and text

The default output is out.html:

Replication modes

Mon, 01 Jan 0001 00:00:00 +0000

Patroni uses PostgreSQL streaming replication. For more information about streaming replication, see the Postgres documentation. By default Patroni configures PostgreSQL for asynchronous replication. Choosing your replication schema is dependent on your business considerations. Investigate both async and sync replication, as well as other HA solutions, to determine which solution is best for you.

Asynchronous mode durability

In asynchronous mode the cluster is allowed to lose some committed transactions to ensure availability. When the primary server fails or becomes unavailable for any other reason Patroni will automatically promote a sufficiently healthy standby to primary. Any transactions that have not been replicated to that standby remain in a “forked timeline” on the primary, and are effectively unrecoverable¹.

Sizing and Performance

Mon, 01 Jan 0001 00:00:00 +0000

Typical CPU usage figures show 15% of the processing time spent in HAProxy versus 85% in the kernel in TCP or HTTP close mode, and about 30% for HAProxy versus 70% for the kernel in HTTP keep-alive mode. This means that the operating system and its tuning have a strong impact on the global performance.

Usages vary a lot between users, some focus on bandwidth, other ones on request rate, others on connection concurrency, others on SSL performance. This section aims at providing a few elements to help with this task.

Community

Mon, 01 Jan 0001 00:00:00 +0000

Tutorials

How to Set Up PgBouncer for Postgres Plus Standard Server

Good overview of PgBouncer concepts.
What is the point of bouncing?

Discusses differences between pooling modes.

Support

Project page at GitHub
Issue tracker at GitHub
PgBouncer section at Stack Overflow
Community discussions at GitHub

Info Command (info)

Mon, 01 Jan 0001 00:00:00 +0000

The info command operates on a single stanza or all stanzas. Text output is the default and gives a human-readable summary of backups for the stanza(s) requested. This format is subject to change with any release.

For machine-readable output use --output=json. The JSON output contains far more information than the text output and is kept stable unless a bug is found.

To speed up execution, limit the output to only progress information by specifying --detail-level=progress. Note that this skips all checks except for availability of the stanza.

Project Metrics

Mon, 01 Jan 0001 00:00:00 +0000

Code Coverage

pgBackRest aims to have complete function/branch/line coverage for the core C code in /src.

Function/line coverage is complete with no exceptions.

Branch coverage excludes branches inside macros and assert() calls. Macros have their own unit tests so they do not need to be tested everywhere they appear. Asserts are not expected to have complete branch coverage since they test cases that should always be true.

Releases, Packages, and Upgrades

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is an open source project covered by the GPLv2 license, meaning that everyone is allowed to redistribute it provided that access to the sources is also provided upon request, especially if any modifications were made.

HAProxy evolves as a main development branch called “master” or “mainline”, from which new branches are derived once the code is considered stable. A lot of web sites run some development branches in production on a voluntarily basis, either to participate to the project or because they need a bleeding edge feature, and their feedback is highly valuable to fix bugs and judge the overall quality and stability of the version being developed.

Sample Reports

Mon, 01 Jan 0001 00:00:00 +0000

The example content was generated by pgBadger 11.8 in May 2022. The original examples are stored with this site, so report navigation, charts, styles, and scripts remain available without reaching the upstream server.

Example	What it demonstrates	Local copy
Complete report	PostgreSQL activity together with PgBouncer statistics	Open the complete report
Incremental report	Calendar index, weekly aggregation, and daily pages	Open the incremental index
Errors and events	A report restricted to errors and operational events	Open the error report

Complete report

The complete report is a self-contained HTML document. Use its top navigation to inspect global statistics, queries, sessions, connections, temporary files, checkpoints, autovacuum activity, locks, and PgBouncer-specific charts.

Standby cluster

Mon, 01 Jan 0001 00:00:00 +0000

Patroni also support running cascading replication to a remote datacenter (region) using a feature that is called “standby cluster”. This type of clusters has:

“standby leader”, that behaves pretty much like a regular cluster leader, except it replicates from a remote node.
cascade replicas, that are replicating from standby leader.

Standby leader holds and updates a leader lock in DCS. If the leader lock expires, cascade replicas will perform an election to choose another leader from the standbys.

Companion Products and Alternatives

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy integrates fairly well with certain products listed below, which is why they are mentioned here even if not directly related to HAProxy.

4.1. Apache HTTP server

Apache is the de-facto standard HTTP server. It’s a very complete and modular project supporting both file serving and dynamic contents. It can serve as a frontend for some application servers. It can even proxy requests and cache responses. In all of these use cases, a front load balancer is commonly needed. Apache can work in various modes, some being heavier than others. Certain modules still require the heavier pre-forked model and will prevent Apache from scaling well with a high number of connections. In this case HAProxy can provide a tremendous help by enforcing the per-server connection limits to a safe value and will significantly speed up the server and preserve its resources that will be better used by the application.

Frequently Asked Questions

Mon, 01 Jan 0001 00:00:00 +0000

How to connect to PgBouncer?

PgBouncer acts as a Postgres server, so simply point your client to the PgBouncer port.

How to load-balance queries between several servers?

PgBouncer does not have an internal multi-host configuration. It is possible via external tools:

DNS round-robin. Use several IPs behind one DNS name. PgBouncer does not look up DNS each time a new connection is launched. Instead, it caches all IPs and does round-robin internally. Note: if there are more than 8 IPs behind one name, the DNS backend must support the EDNS0 protocol. See README for details.

Repository Get Command (repo-get)

Mon, 01 Jan 0001 00:00:00 +0000

Similar to the unix cat command but works on any supported repository type. This command requires a fully qualified file name and is primarily for administration, investigation, and testing. It is not a required part of a normal pgBackRest setup.

If the repository is encrypted then repo-get will automatically decrypt the file. Files are not automatically decompressed but the output can be piped through the appropriate decompression command, e.g. gzip -d.

Watchdog support

Mon, 01 Jan 0001 00:00:00 +0000

Having multiple PostgreSQL servers running as primary can result in transactions lost due to diverging timelines. This situation is also called a split-brain problem. To avoid split-brain Patroni needs to ensure PostgreSQL will not accept any transaction commits after leader key expires in the DCS. Under normal circumstances Patroni will try to achieve this by stopping PostgreSQL when leader lock update fails for any reason. However, this may fail to happen due to various reasons:

How to Set Up a Demo etcd Cluster

Mon, 01 Jan 0001 00:00:00 +0000

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

Pause/Resume mode for the cluster

Mon, 01 Jan 0001 00:00:00 +0000

The goal

Under certain circumstances Patroni needs to temporarily step down from managing the cluster, while still retaining the cluster state in DCS. Possible use cases are uncommon activities on the cluster, such as major version upgrades or corruption recovery. During those activities nodes are often started and stopped for reasons unknown to Patroni, some nodes can be even temporarily promoted, violating the assumption of running only one primary. Therefore, Patroni needs to be able to “detach” from the running cluster, implementing an equivalent of the maintenance mode in Pacemaker.

Repository List Command (repo-ls)

Mon, 01 Jan 0001 00:00:00 +0000

Similar to the unix ls command but works on any supported repository type. This command accepts a path, absolute or relative to the repository path defined by the --repo-path option, and is primarily for administration, investigation, and testing. It is not a required part of a normal pgBackRest setup.

The default text output prints one file name per line. JSON output is available by specifying --output=json.

If more than one repository is configured, the command will default to the highest priority repository (e.g. repo1) unless the --repo option is specified.

Documentation and Community

Mon, 01 Jan 0001 00:00:00 +0000

1. Available documentation

The complete HAProxy documentation is contained in the following documents. Please ensure to consult the relevant documentation to save time and to get the most accurate response to your needs. Also please refrain from sending questions to the mailing list whose responses are present in these documents.

intro.txt (this document): it presents the basics of load balancing, HAProxy as a product, what it does, what it doesn’t do, some known traps to avoid, some OS-specific limitations, how to get it, how it evolves, how to ensure you’re running with all known fixes, how to update it, complements and alternatives.

Support and Contributing

Mon, 01 Jan 0001 00:00:00 +0000

pgBadger is maintained as an open project. Bug reports, feature proposals, documentation fixes, and patches are handled through the upstream GitHub repository.

Bugs and feature requests

Upgrade to the newest released version and confirm the behavior still occurs.
Search open issues and closed issues for an existing answer.
Reduce the problem to the smallest safe log sample and command line that still reproduces it.
Remove credentials, sensitive SQL, bind values, host names, addresses, and business data.
Open a new issue with the pgBadger version, operating system, input format, exact options, observed result, and expected result.

For crashes or parser mistakes, include only the minimum sanitized log lines needed to reproduce the boundary. A complete production log or generated report is rarely appropriate for a public issue.

Configuration Manual

Mon, 01 Jan 0001 00:00:00 +0000

DCS Failsafe Mode

Mon, 01 Jan 0001 00:00:00 +0000

The problem

Patroni is heavily relying on Distributed Configuration Store (DCS) to solve the task of leader elections and detect network partitioning. That is, the node is allowed to run Postgres as the primary only if it can update the leader lock in DCS. In case the update of the leader lock fails, Postgres is immediately demoted and started as read-only. Depending on which DCS is used, the chances of hitting the “problem” differ. For example, with Etcd which is only used for Patroni, chances are close to zero, while with K8s API (backed by Etcd) it could be observed more frequently.

License and Credits

Mon, 01 Jan 0001 00:00:00 +0000

pgBadger is free and open-source software distributed under the PostgreSQL License. It may be used, copied, modified, and distributed without a fee, subject to retaining the copyright and license notices.

PostgreSQL License

Permission to use, copy, modify, and distribute this software and its documentation for any purpose, without fee, and without a written agreement is hereby granted, provided that the above copyright notice and this paragraph and the following two paragraphs appear in all copies.

Restore Command (restore)

Mon, 01 Jan 0001 00:00:00 +0000

The restore command automatically defaults to selecting the latest backup from the first repository where backups exist (see Quick Start - Restore a Backup). The order in which the repositories are checked is dictated by the pgbackrest.conf (e.g. repo1 will be checked before repo2). To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.

1. Quick Reminder About HTTP

Mon, 01 Jan 0001 00:00:00 +0000

This document covers the configuration language as implemented in the version specified above. It does not provide any hints, examples, or advice. For such documentation, please refer to the Reference Manual or the Architecture Manual. The numbered chapters are ordered in the flat HAProxy sidebar for direct navigation.

When HAProxy is running in HTTP mode, both the request and the response are fully analyzed and indexed, thus it becomes possible to build matching criteria on almost anything found in the contents.

Server Command (server)

Mon, 01 Jan 0001 00:00:00 +0000

The pgBackRest server allows access to remote hosts without using the SSH protocol.

Command Options

TLS Server Address Option (`--tls-server-address`)

TLS server address.

IP address the server will listen on for client requests.

default: localhost
example: --tls-server-address=*

TLS Server Authorized Clients Option (`--tls-server-auth`)

TLS server authorized clients.

Clients are authorized on the server by verifying their certificate and checking their certificate CN (Common Name) against a list on the server configured with the tls-server-auth option.

Using Patroni with Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Patroni can use Kubernetes objects in order to store the state of the cluster and manage the leader key. That makes it capable of operating Postgres in Kubernetes environment without any consistency store, namely, one doesn’t need to run an extra Etcd deployment. There are two different type of Kubernetes objects Patroni can use to store the leader and the configuration keys, they are configured with the kubernetes.use_endpoints or PATRONI_KUBERNETES_USE_ENDPOINTS environment variable.

2. Configuring HAProxy

Mon, 01 Jan 0001 00:00:00 +0000

2.1. Configuration file format

HAProxy’s configuration process involves 3 major sources of parameters:

the arguments from the command-line, which always take precedence
the configuration file(s), whose format is described here
the running process’s environment, in case some environment variables are explicitly referenced

The configuration file follows a fairly simple hierarchical format which obey a few basic rules:

Citus support

Mon, 01 Jan 0001 00:00:00 +0000

Patroni makes it extremely simple to deploy Multi-Node Citus clusters.

TL;DR

There are only a few simple rules you need to follow:

Citus database extension to PostgreSQL must be available on all nodes. Absolute minimum supported Citus version is 10.0, but, to take all benefits from transparent switchovers and restarts of workers we recommend using at least Citus 11.2.
Cluster name (scope) must be the same for all Citus nodes!
Superuser credentials must be the same on coordinator and all worker nodes, and pg_hba.conf should allow superuser access between all nodes.
REST API access should be allowed from worker nodes to the coordinator. E.g., credentials should be the same and if configured, client certificates from worker nodes must be accepted by the coordinator.
Add the following section to the patroni.yaml:

citus:
  group: X  # 0 for coordinator and 1, 2, 3, etc for workers
  database: citus  # must be the same on all nodes

After that you just need to start Patroni and it will handle the rest:

Server Ping Command (server-ping)

Mon, 01 Jan 0001 00:00:00 +0000

Ping a pgBackRest TLS server to ensure it is accepting connections. This serves as an aliveness check only since no authentication is attempted.

If no host is specified on the command-line then the tls-server-host option will be used.

Command Options

TLS Server Address Option (`--tls-server-address`)

TLS server address.

IP address the server will listen on for client requests.

default: localhost
example: --tls-server-address=*

TLS Server Port Option (`--tls-server-port`)

TLS server port.

3. Global Section

Mon, 01 Jan 0001 00:00:00 +0000

Parameters in the “global” section are process-wide and often OS-specific. They are generally set once for all and do not need being changed once correct. Some of them have command-line equivalents.

The following keywords are supported in the “global” section:

Process management and security
- 51degrees-allow-unmatched
- 51degrees-cache-size
- 51degrees-data-file
- 51degrees-difference
- 51degrees-drift
- 51degrees-property-name-list
- 51degrees-property-separator
- 51degrees-use-performance-graph
- 51degrees-use-predictive-graph
- ca-base
- chroot
- cluster-secret
- cpu-affinity
- cpu-map
- cpu-policy
- cpu-set
- crt-base
- daemon
- default-path
- description
- deviceatlas-json-file
- deviceatlas-log-level
- deviceatlas-properties-cookie
- deviceatlas-separator
- dns-accept-family
- expose-deprecated-directives
- expose-experimental-directives
- external-check
- fd-hard-limit
- gid
- grace
- group
- h1-accept-payload-with-any-method
- h1-case-adjust
- h1-case-adjust-file
- h1-do-not-close-on-insecure-transfer-encoding
- h2-workaround-bogus-websocket-clients
- hard-stop-after
- harden.reject-privileged-ports.tcp
- harden.reject-privileged-ports.quic
- insecure-fork-wanted
- insecure-setuid-wanted
- issuers-chain-path
- jwt.decrypt_alg_list
- jwt.decrypt_enc_list
- key-base
- limited-quic
- localpeer
- log
- log-send-hostname
- log-tag
- lua-load
- lua-load-per-thread
- lua-prepend-path
- max-threads-per-group
- mworker-max-reloads
- nbthread
- node
- numa-cpu-mapping
- ocsp-update.disable
- ocsp-update.maxdelay
- ocsp-update.mindelay
- ocsp-update.httpproxy
- ocsp-update.mode
- pidfile
- pp2-never-send-local
- presetenv
- prealloc-fd
- resetenv
- set-dumpable
- set-var
- setenv
- ssl-default-bind-ciphers
- ssl-default-bind-ciphersuites
- ssl-default-bind-client-sigalgs
- ssl-default-bind-curves
- ssl-default-bind-options
- ssl-default-bind-sigalgs
- ssl-default-server-ciphers
- ssl-default-server-ciphersuites
- ssl-default-server-client-sigalgs
- ssl-default-server-curves
- ssl-default-server-options
- ssl-default-server-sigalgs
- ssl-dh-param-file
- ssl-propquery
- ssl-provider
- ssl-provider-path
- ssl-security-level
- ssl-server-verify
- ssl-skip-self-issued-ca
- stats
- stats-file
- strict-limits
- uid
- ulimit-n
- unix-bind
- unsetenv
- user
- wurfl-cache-size
- wurfl-data-file
- wurfl-information-list
- wurfl-information-list-separator
Performance tuning

Convert a Standalone to a Patroni Cluster

Mon, 01 Jan 0001 00:00:00 +0000

This section describes the process for converting a standalone PostgreSQL instance into a Patroni cluster.

To deploy a Patroni cluster without using a pre-existing PostgreSQL instance, see Running and Configuring instead.

Procedure

You can find below an overview of steps for converting an existing Postgres cluster to a Patroni managed cluster. In the steps we assume all nodes that are part of the existing cluster are currently up and running, and that you do not intend to change Postgres configuration while the migration is ongoing. The steps:

Stanza Create Command (stanza-create)

Mon, 01 Jan 0001 00:00:00 +0000

The stanza-create command must be run after the stanza has been configured in pgbackrest.conf. If there is more than one repository configured, the stanza will be created on each. Stanzas that have already been created will be skipped so it is always safe to run stanza-create, even when a new repository has been configured.

See Create the Stanza for more information and an example.

Command Options

Online Option (`--online`)

Create on an online cluster.

4. Proxies

Mon, 01 Jan 0001 00:00:00 +0000

Proxy configuration can be located in a set of sections:

defaults [<name>] [ from <defaults_name> ]
frontend <name> [ from <defaults_name> ]
backend <name> [ from <defaults_name> ]
listen <name> [ from <defaults_name> ]

A “frontend” section describes a set of listening sockets accepting client connections.

A “backend” section describes a set of servers to which the proxy will connect to forward incoming connections.

Stanza Delete Command (stanza-delete)

Mon, 01 Jan 0001 00:00:00 +0000

The stanza-delete command removes data in the repository associated with a stanza.

WARNING:

Use this command with caution — it will permanently remove all backups and archives from the pgBackRest repository for the specified stanza.

To delete a stanza:

Shut down the PostgreSQL cluster associated with the stanza (or use –force to override).
Run the stop command on the host where the stanza-delete command will be run.
Run the stanza-delete command.

Once the command successfully completes, it is the responsibility of the user to remove the stanza from all pgBackRest configuration files and/or environment variables.

Integration with other tools

Mon, 01 Jan 0001 00:00:00 +0000

Patroni is able to integrate with other tools in your stack. In this section you will find a list of examples, which although not an exhaustive list, might provide you with ideas on how Patroni can integrate with other tools.

Barman

Patroni delivers an application named patroni_barman which has logic to communicate with pg-backup-api, so you are able to perform Barman operations remotely.

This application currently has a couple of sub-commands: recover and config-switch.

5. Bind and Server Options

Mon, 01 Jan 0001 00:00:00 +0000

The “bind”, “server” and “default-server” keywords support a number of settings depending on some build options and on the system HAProxy was built on. These settings generally each consist in one word sometimes followed by a value, written on the same line as the “bind” or “server” line. All these options are described in this section.

5.1. Bind options

The “bind” keyword supports a certain number of settings which are all passed as arguments on the same line. The order in which those arguments appear makes no importance, provided that they appear after the bind address. All of these parameters are optional. Some of them consist in a single words (booleans), while other ones expect a value after them. In this case, the value must be provided immediately after the setting name.

Security Considerations

Mon, 01 Jan 0001 00:00:00 +0000

A Patroni cluster has two interfaces to be protected from unauthorized access: the distributed configuration storage (DCS) and the Patroni REST API.

Protecting DCS

Patroni and patronictl both store and retrieve data to/from the DCS.

Despite DCS doesn’t contain any sensitive information, it allows changing some of Patroni/Postgres configuration. Therefore the very first thing that should be protected is DCS itself.

The details of protection depend on the type of DCS used. The authentication and encryption parameters (tokens/basic-auth/client certificates) for the supported types of DCS are covered in settings.

Stanza Upgrade Command (stanza-upgrade)

Mon, 01 Jan 0001 00:00:00 +0000

Immediately after upgrading PostgreSQL to a newer major version, the pg-path for all pgBackRest configurations must be set to the new database location and the stanza-upgrade command run. If there is more than one repository configured on the host, the stanza will be upgraded on each. If the database is offline use the --no-online option.

Command Options

Online Option (`--online`)

Update an online cluster.

6. Cache

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy provides a cache, which was designed to perform cache on small objects (favicon, css…). This is a minimalist low-maintenance cache which runs in RAM.

The cache is based on a memory area shared between all threads, and split in 1kB blocks.

If an object is not used anymore, it can be deleted to store a new object independently of its expiration date. The oldest objects are deleted first when we try to allocate a new one.

HA multi datacenter

Mon, 01 Jan 0001 00:00:00 +0000

The high availability of a PostgreSQL cluster deployed in multiple data centers is based on replication, which can be synchronous or asynchronous (see replication modes).

In both cases, it is important to be clear about the following concepts:

Postgres can run as primary or standby leader only when it owns the leading key and can update the leading key.
You should run the odd number of etcd, ZooKeeper or Consul nodes: 3 or 5!

Synchronous Replication

To have a multi DC cluster that can automatically tolerate a zone drop, a minimum of 3 is required.

Start Command (start)

Mon, 01 Jan 0001 00:00:00 +0000

If the pgBackRest processes were previously stopped using the stop command then they can be started again using the start command. Note that this will not immediately start up any pgBackRest processes but they are allowed to run. See Starting and Stopping for more information and examples.

General Options

Allow Run as Root Option (`--allow-root`)

Allow the command to run as the root user.

7. ACLs and Sample Fetching

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is capable of extracting data from request or response streams, from client or server information, from tables, environmental information etc… The action of extracting such data is called fetching a sample. Once retrieved, these samples may be used for various purposes such as a key to a stick-table, but most common usages consist in matching them against predefined constant data called patterns.

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

In this section you will find answers for the most frequently asked questions about Patroni. Each sub-section attempts to focus on different kinds of questions.

We hope that this helps you to clarify most of your questions. If you still have further concerns or find yourself facing an unexpected issue, please refer to chatting and reporting_bugs for instructions on how to get help or report issues.

Comparison with other HA solutions

Why does Patroni require a separate cluster of DCS nodes while other solutions like repmgr do not?
There are different ways of implementing HA solutions, each of them with their pros and cons.

Stop Command (stop)

Mon, 01 Jan 0001 00:00:00 +0000

Does not allow any new pgBackRest processes to run. By default running processes will be allowed to complete successfully. Use the --force option to terminate running processes.

pgBackRest processes will return an error if they are run after the stop command completes. See Starting and Stopping for more information and examples.

Command Options

Force Option (`--force`)

Force all pgBackRest processes to stop.

8. Logging

Mon, 01 Jan 0001 00:00:00 +0000

One of HAProxy’s strong points certainly lies is its precise logs. It probably provides the finest level of information available for such a product, which is very important for troubleshooting complex environments. Standard information provided in logs include client ports, TCP/HTTP state timers, precise stream state at termination and precise termination cause, information about decisions to direct traffic to a server, and of course the ability to capture arbitrary headers.

Release notes

Mon, 01 Jan 0001 00:00:00 +0000

Version 4.1.5

Released 2026-08-12

Compatibility improvements

Compatibility with PostgreSQL 14.24, 15.19, 16.15, 17.11, 18.6 (Alexander Kukushkin)

Add the new output_plugin_libraries GUC, which restricts logical decoding output plugins.

Improvements

Log REST API connection resets at DEBUG instead of WARNING (Kyle McLaren)

Ensure the common “client went away mid-write” variants are silenced without affecting the handling of genuine (non-connection) errors.

Bugfixes

Fix thread_stack_size validation alignment (Sundong Kim)

Verify Command (verify)

Mon, 01 Jan 0001 00:00:00 +0000

Verify determines if the backups and archives in a repository are valid.

Command Options

Output Option (`--output`)

Output type.

The following output types are supported:

none - No verify output.
text - Output verify information to stdout.

default: none
example: --output=text

Set Option (`--set`)

Backup set to verify.

Verify all database and archive files associated with the specified backup set.

9. Supported Filters

Mon, 01 Jan 0001 00:00:00 +0000

Here are listed officially supported filters with the list of parameters they accept. Depending on compile options, some of these filters might be unavailable. The list of available filters is reported in haproxy -vv.

9.1. Trace

filter trace [name <name>] [random-forwarding] [max-fwd <max>] [hexdump]

Arguments:

<name>               is an arbitrary name that will be reported in
                     messages. If no name is provided, "TRACE" is used.

<quiet>              inhibits trace messages.

<random-forwarding>  enables the random forwarding of parsed data. By
                     default, this filter forwards all previously parsed
                     data. With this parameter, it only forwards a random
                     amount of the parsed data.

<max>                is the maximum amount of data that can be forwarded at
                     a time. "max-fwd" option can be combined with the
                     random forwarding. <max> must be an positive integer.
                     0 means there is no limit.

<hexdump>             dumps all forwarded data to the server and the client.

This filter can be used as a base to develop new filters. It defines all callbacks and print a message on the standard error stream (stderr) with useful information for all of them. It may be useful to debug the activity of other filters or, quite simply, HAProxy’s activity.

Contributing guidelines

Mon, 01 Jan 0001 00:00:00 +0000

Chatting

If you have a question, looking for an interactive troubleshooting help or want to chat with other Patroni users, join us on channel #patroni in the PostgreSQL Slack.

Reporting bugs

Before reporting a bug please make sure to reproduce it with the latest Patroni version! Also please double check if the issue already exists in our Issues Tracker.

Running tests

Requirements for running behave tests:

Reading from etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the get subcommand to read from etcd:

$ etcdctl --endpoints=$ENDPOINTS get foo
foo
Hello World!
$

where:

foo is the requested key
Hello World! is the retrieved value

Or, for formatted output:

$ etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo
{"header":{"cluster_id":289318470931837780,"member_id":14947050114012957595,"revision":3,"raft_term":4,
"kvs":[{"key":"Zm9v","create_revision":2,"mod_revision":3,"version":2,"value":"SGVsbG8gV29ybGQh"}]}}
$

where write-out="json" causes the value to be output in JSON format (note that the key is not returned).

Version Command (version)

Mon, 01 Jan 0001 00:00:00 +0000

Displays installed pgBackRest version.

Command Options

Output Option (`--output`)

Output type.

The following output types are supported:

text - Display the installed pgBackRest version as text.
num - Display the installed pgBackRest version as an integer.

default: text
example: --output=num

10. FastCGI Applications

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is able to send HTTP requests to Responder FastCGI applications. This feature was added in HAProxy 2.1. To do so, servers must be configured to use the FastCGI protocol (using the keyword “proto fcgi” on the server line) and a FastCGI application must be configured and used by the backend managing these servers (using the keyword “use-fcgi-app” into the proxy section). Several FastCGI applications may be defined, but only one can be used at a time by a backend.

11. Stick Tables and Peers

Mon, 01 Jan 0001 00:00:00 +0000

Stick-tables in HAProxy are a mechanism which permits to associate a certain number of information and metrics with a key of a certain type, and this for a certain duration after the last update. This can be seen as a multicolumn line in a table, where the line number is defined by the key value, and the columns all represent distinct criteria.

12. Other Sections

Mon, 01 Jan 0001 00:00:00 +0000

The sections described below are less commonly used and usually support only a few parameters. There is no implicit relation between any of them. They’re all started using a single keyword. None of them is permitted before a “global” section. The support for some of them might be conditioned by build options (e.g. anything SSL-related).

12.1. Traces

For debugging purpose, it is possible to activate traces on an HAProxy’s subsystem. This will dump debug messages about a specific subsystem. It is a very powerful tool to diagnose issues. Traces can be dynamically configured via the CLI. It is also possible to predefined some settings in the configuration file, in dedicated “traces” sections. More details about traces can be found in the management guide. It remains a developer tools used during complex debugging sessions. It is pretty verbose and have a cost, so use it with caution. And because it is a developer tool, there is no warranty about the backward compatibility of this section.

Management Guide

Mon, 01 Jan 0001 00:00:00 +0000

1. Prerequisites

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to start, stop, manage, and troubleshoot HAProxy, as well as some known limitations and traps to avoid. It does not describe how to configure it (for this please read configuration.txt).

In this document it is assumed that the reader has sufficient administration skills on a UNIX-like operating system, uses the shell on a daily basis and is familiar with troubleshooting utilities such as strace and tcpdump.

Writing to etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the put subcommand to write a key-value pair:

etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

where:

foo is the key name
"Hello World!" is the quote-delimited value

2. HAProxy Architecture

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means it uses event multiplexing to schedule all of its activities instead of relying on the system to schedule between multiple activities. Most of the time it runs as a single process, so the output of “ps aux” on a system will report only one “haproxy” process, unless a soft reload is in progress and an older process is finishing its job in parallel to the new one. It is thus always easy to trace its activity using the strace utility. In order to scale with the number of available processors, by default haproxy will start one worker thread per processor it is allowed to run on. Unless explicitly configured differently, the incoming traffic is spread over all these threads, all running the same event loop. A great care is taken to limit inter-thread dependencies to the strict minimum, so as to try to achieve near-linear scalability. This has some impacts such as the fact that a given connection is served by a single thread. Thus in order to use all available processing capacity, it is needed to have at least as many connections as there are threads, which is almost always granted.

3. Starting HAProxy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is started by invoking the “haproxy” program with a number of arguments passed on the command line. The actual syntax is:

$ haproxy [<options>]*

where [<options>]* is any number of options. An option always starts with ‘-’ followed by one of more letters, and possibly followed by one or multiple extra arguments. Without any option, HAProxy displays the help page with a reminder about supported options. Available options may vary slightly based on the operating system. A fair number of these options overlap with an equivalent one in the “global” section. In this case, the command line always has precedence over the configuration file, so that the command line can be used to quickly enforce some settings without touching the configuration files. The current list of options is:

4. Stopping and Restarting HAProxy

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy supports a graceful and a hard stop. The hard stop is simple, when the SIGTERM signal is sent to the haproxy process, it immediately quits and all established connections are closed. The graceful stop is triggered when the SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding from listening ports, but continue to process existing connections until they close. Once the last connection is closed, the process leaves.

5. File-Descriptor Limitations

Mon, 01 Jan 0001 00:00:00 +0000

In order to ensure that all incoming connections will successfully be served, HAProxy computes at load time the total number of file descriptors that will be needed during the process’s life. A regular Unix process is generally granted 1024 file descriptors by default, and a privileged process can raise this limit itself. This is one reason for starting HAProxy as root and letting it adjust the limit. The default limit of 1024 file descriptors roughly allow about 500 concurrent connections to be processed. The computation is based on the global maxconn parameter which limits the total number of connections per process, the number of listeners, the number of servers which have a health check enabled, the agent checks, the peers, the loggers and possibly a few other technical requirements. A simple rough estimate of this number consists in simply doubling the maxconn value and adding a few tens to get the approximate number of file descriptors needed.

6. Memory Management

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy uses a simple and fast pool-based memory management. Since it relies on a small number of different object types, it’s much more efficient to pick new objects from a pool which already contains objects of the appropriate size than to call malloc() for each different size. The pools are organized as a stack or LIFO, so that newly allocated objects are taken from recently released objects still hot in the CPU caches. Pools of similar sizes are merged together, in order to limit memory fragmentation.

How to get keys by prefix

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Get keys by prefix

$ etcdctl --endpoints=$ENDPOINTS get PREFIX --prefix

Global Options

--endpoints=[127.0.0.1:2379], gRPC endpoints

Options

--prefix, get a range of keys with matching prefix

Example

etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3

etcdctl --endpoints=$ENDPOINTS get web --prefix

7. CPU Usage

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy normally spends most of its time in the system and a smaller part in userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end connection setups and closes per second at 100% CPU on a single core. When one core is saturated, typical figures are:

95% system, 5% user for long TCP connections or large HTTP objects
85% system and 15% user for short TCP connections or small HTTP objects in close mode
70% system and 30% user for small HTTP objects in keep-alive mode

The amount of rules processing and regular expressions will increase the user land part. The presence of firewall rules, connection tracking, complex routing tables in the system will instead increase the system part.

8. Logging

Mon, 01 Jan 0001 00:00:00 +0000

For logging, HAProxy always relies on a syslog server since it does not perform any file-system access. The standard way of using it is to send logs over UDP to the log server (by default on port 514). Very commonly this is configured to 127.0.0.1 where the local syslog daemon is running, but it’s also used over the network to log to a central server. The central server provides additional benefits especially in active-active scenarios where it is desirable to keep the logs merged in arrival order. HAProxy may also make use of a UNIX socket to send its logs to the local syslog daemon, but it is not recommended at all, because if the syslog server is restarted while haproxy runs, the socket will be replaced and new logs will be lost. Since HAProxy will be isolated inside a chroot jail, it will not have the ability to reconnect to the new socket. It has also been observed in field that the log buffers in use on UNIX sockets are very small and lead to lost messages even at very light loads. But this can be fine for testing however.

9. Statistics and Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

It is possible to query HAProxy about its status. The most commonly used mechanism is the HTTP statistics page. This page also exposes an alternative CSV output format for monitoring tools. The same format is provided on the Unix socket.

Statistics are regroup in categories labelled as domains, corresponding to the multiple components of HAProxy. There are two domains available: proxy and resolvers. If not specified, the proxy domain is selected. Note that only the proxy statistics are printed on the HTTP page.

10. Easier Configuration Management

Mon, 01 Jan 0001 00:00:00 +0000

It is very common that two HAProxy nodes constituting a cluster share exactly the same configuration modulo a few addresses. Instead of having to maintain a duplicate configuration for each node, which will inevitably diverge, it is possible to include environment variables in the configuration. Thus multiple configuration may share the exact same file with only a few different system wide environment variables. This started in version 1.5 where only addresses were allowed to include environment variables, and 1.6 goes further by supporting environment variables everywhere. The syntax is the same as in the UNIX shell, a variable starts with a dollar sign (’$’), followed by an opening curly brace (’{’), then the variable name followed by the closing brace (’}’). Except for addresses, environment variables are only interpreted in arguments surrounded with double quotes (this was necessary not to break existing setups using regular expressions involving the dollar symbol).

11. Well-Known Traps to Avoid

Mon, 01 Jan 0001 00:00:00 +0000

Once in a while, someone reports that after a system reboot, the haproxy service wasn’t started, and that once they start it by hand it works. Most often, these people are running a clustered IP address mechanism such as keepalived, to assign the service IP address to the master node only, and while it used to work when they used to bind haproxy to address 0.0.0.0, it stopped working after they bound it to the virtual IP address. What happens here is that when the service starts, the virtual IP address is not yet owned by the local node, so when HAProxy wants to bind to it, the system rejects this because it is not a local IP address. The fix doesn’t consist in delaying the haproxy service startup (since it wouldn’t stand a restart), but instead to properly configure the system to allow binding to non-local addresses. This is easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This is also needed in order to transparently intercept the IP traffic that passes through HAProxy for a specific target address.

12. Debugging and Performance Issues

Mon, 01 Jan 0001 00:00:00 +0000

When HAProxy is started with the “-d” option, it will stay in the foreground and will print one line per event, such as an incoming connection, the end of a connection, and for each request or response header line seen. This debug output is emitted before the contents are processed, so they don’t consider the local modifications. The main use is to show the request and response without having to run a network sniffer. The output is less readable when multiple connections are handled in parallel, though the “debug2ansi” and “debug2html” scripts found in the examples/ directory definitely help here by coloring the output.

13. Security Considerations

Mon, 01 Jan 0001 00:00:00 +0000

HAProxy is designed to run with very limited privileges. The standard way to use it is to isolate it into a chroot jail and to drop its privileges to a non-root user without any permissions inside this jail so that if any future vulnerability were to be discovered, its compromise would not affect the rest of the system.

In order to perform a chroot, it first needs to be started as a root user. It is pointless to build hand-made chroots to start the process there, these ones are painful to build, are never properly maintained and always contain way more bugs than the main file-system. And in case of compromise, the intruder can use the purposely built file-system. Unfortunately many administrators confuse “start as root” and “run as root”, resulting in the uid change to be done prior to starting haproxy, and reducing the effective security restrictions.

How to delete keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Add or delete keys

del to remove the specified key or range of keys:

etcdctl del $KEY [$END_KEY]

Options

--prefix[=false]: delete keys with matching prefix
--prev-kv[=false]: return deleted key-value pairs
--from-key[=false]: delete keys that are greater than or equal to the given key using byte compare
--range[=false]: delete range of keys without delay

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key

etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix

How to make multiple writes in a transaction

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl.
A running etcd cluster.

Terminology

Here are definitions of some key terms used in the Example below.

Terms	Definition
etcdctl	The command line tool for interacting with the etcd server.
`txn` command	`txn` command is an abbreviation for “transaction”. It reads multiple etcd requests from standard input and applies them as a single atomic transaction. A transaction consists of list of conditions, a list of requests to apply if all the conditions are true, and a list of requests to apply if any condition is false. View etcdctl key-value commands for more information.
`compare`	The `compare` clause within a transaction (`txn`) serves as a conditional check that determines whether the transaction’s operations should proceed. It ensures changes are only applied if the current state of the key-value store matches expected conditions, thereby maintaining data consistency and preventing conflicts in concurrent environments. To see how the command is structured, view Perform a transaction section below.

Transactions

txn to process all the requests in one transaction:

How to watch keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Watching keys

watch to get notified of future changes:

etcdctl watch $KEY [$END_KEY]

Options

-i, --interactive[=false]: interactive mode
--prefix[=false]: watch on a prefix if prefix is set
--rev=0: Revision to start watching
--prev-kv[=false]: get the previous key-value pair before the event happens
--progress-notify[=false]: get periodic watch progress notification from server

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS watch stock1
etcdctl --endpoints=$ENDPOINTS put stock1 1000

etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20

How to create lease

Mon, 01 Jan 0001 00:00:00 +0000

lease to write with TTL:

etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)

etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS get sample

etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
# or after 300 seconds
etcdctl --endpoints=$ENDPOINTS get sample

How to create locks

Mon, 01 Jan 0001 00:00:00 +0000

LOCK acquires a distributed mutex with a given name. Once the lock is acquired, it will be held until etcdctl is terminated.

Prerequisites

Install etcd and etcdctl

Creating a lock

lock for distributed lock:

etcdctl --endpoints=$ENDPOINTS lock mutex1

Options

endpoints - defines a comma-delimited list of machine addresses in the cluster.
ttl - time out in seconds of lock session.

How to conduct leader election in etcd cluster

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Ensure etcd and etcdctl is installed.
Check for active etcd cluster.

Conduct Leader election

The etcdctl command is used to conduct leader elections in an etcd cluster. It makes sure that only one client can become leader at a time.

etcdctl --endpoints=$ENDPOINTS elect <election-name> [proposal]

etcdctl --endpoints=$ENDPOINTS elect election-name p1

Options

--endpoints : $ENDPOINTS

Address of each etcd cluster members.

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions to locally install, run, and test a single-member cluster of etcd:

Install etcd from pre-built binaries or from source. For details, see Install.

Warning

Important: Ensure that you perform the last step of the installation instructions to verify that etcd is in your path.

Launch etcd:

$ etcd
{"level":"info","ts":"2021-09-17T09:19:32.783-0400","caller":"etcdmain/etcd.go:72","msg":... }
⋮

Note

Note: The output produced by etcd are logs — info-level logs can be ignored.

How to check Cluster status

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Check Overall Status

endpoint status to check the overall status of each endpoint specified in --endpoints flag:

etcdctl endpoint status (--endpoints=$ENDPOINTS|--cluster)

Options

--cluster[=false]: use all endpoints from the cluster member list

Check Health

endpoint health to check the healthiness of each endpoint specified in --endpoints flag:

Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Auth

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

How to save the database

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Snapshot a database

snapshot to save point-in-time snapshot of etcd database:

etcdctl --endpoints=$ENDPOINT snapshot save DB_NAME

Global Options

etcdctl

--endpoints=[127.0.0.1:2379], gRPC endpoints

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

Install

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Before installing etcd, see the following pages:

Install pre-built binaries

The easiest way to install etcd is from pre-built binaries:

Download the compressed archive file for your platform from Releases, choosing release v3.7.0 or later.
Unpack the archive file. This results in a directory containing the binaries.
Add the executable binaries to your path. For example, rename and/or move the binaries to a directory in your path (like /usr/local/bin), or add the directory created by the previous step to your path.

Feature Gates

Mon, 01 Jan 0001 00:00:00 +0000

This page contains an overview of the various feature gates an administrator can specify on etcd.

See feature stages for an explanation of the stages for a feature.

Overview

Feature gates are a set of key=value pairs that describe etcd features. You can turn these features on or off using the --feature-gates command line flag on etcd.

etcd lets you enable or disable a set of feature gates. Use -h flag to see a full set of feature gates. To set feature gates, use the --feature-gates flag assigned to a list of feature pairs in commandline:

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

What is etcd?

etcd is a consistent distributed key-value store. Mainly used as a separate coordination service, in distributed systems. And designed to hold small amounts of data that can fit entirely in memory.

How do you pronounce etcd?

etcd is pronounced /ˈɛtsiːdiː/, and means “distributed etc directory.”

Do clients have to send requests to the etcd leader?

Raft is leader-based; the leader handles all client requests which need cluster consensus. However, the client does not need to know which node is the leader. Any request that requires consensus sent to a follower is automatically forwarded to the leader. Requests that do not require consensus (e.g., serialized reads) can be processed by any cluster member.

How to Add and Remove Members

Mon, 01 Jan 0001 00:00:00 +0000

member to add,remove,update membership:

# For each machine
TOKEN=my-etcd-token-1
CLUSTER_STATE=new
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_3=etcd-node-3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Note that third-party libraries and tools (not hosted on https://github.com/etcd-io) mentioned below are not tested or maintained by the etcd team. Before using them, users are recommended to read and investigate them.

Tools

etcdctl - A command line client for etcd
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcdloadtest - A command line load test client for etcd version 3.0 and above.
etcd-tui - A modern terminal user interface (TUI) for interacting with your etcd database. Navigate keys, view values, filter data, and manage your etcd cluster directly from your terminal.
etcdfinder - A lightning-fast, modern web UI for etcd with instant search. Supports both etcd v2 and v3.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-manager - A modern, efficient, multi-platform and free etcd 3.x GUI & client tool. Available for Windows, Linux and Mac.
etcd-backup-restore - Utility to periodically and incrementally backup and restore the etcd.
etcd-druid - A Kubernetes operator to deploy etcd clusters and manage day-2 operations.
etcdadm - A command-line tool for operating an etcd cluster.
etcd-defrag - An easier to use and smarter etcd defragmentation tool.
etcdhelper - An intellij platform plugin for etcd.

Libraries

The sections below list etcd client libraries by language.

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

If any part of the etcd project has bugs or documentation mistakes, please let us know by opening an issue. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

The default settings in etcd should work well for installations on a local network where the average network latency is low. However, when using etcd across multiple data centers or over networks with high latency, the heartbeat interval and election timeout settings may need tuning.

The network isn’t the only source of latency. Each request and response may be impacted by slow disks on both the leader and follower. Each of these timeouts represents the total time from request to successful response from the other machine.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

The protocol uses a new discovery token to bootstrap one unique etcd cluster. Remember that one discovery token can represent only one etcd cluster. As long as discovery protocol on this token starts, even if it fails halfway, it must not be used to bootstrap another etcd cluster.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery token and endpoint list.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Logging conventions

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses the zap library for logging application output categorized into levels. A log message’s level is determined according to these conventions:

DebugLevel logs are typically voluminous, and are usually disabled in production.
- Examples:
  - Send a normal message to a remote peer
  - Write a log entry to disk
InfoLevel is the default logging priority.
- Examples:
  - Startup configuration
  - Start to do snapshot
  - Add a new node into the cluster
  - Add a new user into auth subsystem
WarnLevel logs are more important than Info, but don’t need individual human review.

Golang modules

Mon, 01 Jan 0001 00:00:00 +0000

The etcd project (since version 3.5) is organized into multiple golang modules hosted in a single repository.

There are following modules:

go.etcd.io/etcd/api/v3 - contains API definitions (like protos & proto-generated libraries) that defines communication protocol between etcd clients and server.
go.etcd.io/etcd/pkg/v3 - collection of utility packages used by etcd without being specific to etcd itself. A package belongs here only if it could possibly be moved out into its own repository in the future. Please avoid adding here code that has a lot of dependencies on its own, as they automatically becoming dependencies of the client library (that we want to keep lightweight).

Data model

Mon, 01 Jan 0001 00:00:00 +0000

etcd is designed to reliably store infrequently updated data and provide reliable watch queries. etcd exposes previous versions of key-value pairs to support inexpensive snapshots and watch history events (“time travel queries”). A persistent, multi-version, concurrency-control data model is a good fit for these use cases.

etcd stores data in a multiversion persistent key-value store. The persistent key-value store preserves the previous version of a key-value pair when its value is superseded with new data. The key-value store is effectively immutable; its operations do not update the structure in-place, but instead always generate a new updated structure. All past versions of keys are still accessible and watchable after modification. To prevent the data store from growing indefinitely over time and from maintaining old versions, the store may be compacted to shed the oldest versions of superseded data.

etcd client design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Client Design

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Introduction

etcd server has proven its robustness with years of failure injection testing. Most complex application logic is already handled by etcd server and its data stores (e.g. cluster membership is transparent to clients, with Raft-layer forwarding proposals to leader). Although server components are correct, its composition with client requires a different set of intricate protocols to guarantee its correctness and high availability under faulty conditions. Ideally, etcd server provides one logical cluster view of many physical machines, and client implements automatic failover between replicas. This documents client architectural decisions and its implementation details.

etcd learner design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Learner

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Background

Membership reconfiguration has been one of the biggest operational challenges. Let’s review common challenges.

1. New Cluster member overloads Leader

A newly joined etcd member starts with no data, thus demanding more updates from leader until it catches up with leader’s logs. Then leader’s network is more likely to be overloaded, blocking or dropping leader heartbeats to followers. In such case, a follower may election-timeout to start a new leader election. That is, a cluster with a new member is more vulnerable to leader election. Both leader election and the subsequent update propagation to the new member are prone to causing periods of cluster unavailability (see Figure 1).

etcd v3 authentication design

Mon, 01 Jan 0001 00:00:00 +0000

Why not reuse the v2 auth system?

The v3 protocol uses gRPC as its transport instead of a RESTful interface like v2. This new protocol provides an opportunity to iterate on and improve the v2 design. For example, v3 auth has connection based authentication, rather than v2’s slower per-request authentication. Additionally, v2 auth’s semantics tend to be unwieldy in practice with respect to reasoning about consistency, which will be described in the next sections. For v3, there is a well-defined description and implementation of the authentication mechanism which fixes the deficiencies in the v2 auth system.

etcd API

Mon, 01 Jan 0001 00:00:00 +0000

This document is meant to give an overview of the v3 etcd APIs central design. This should not be mistaken with etcd v2 API, deprecated in etcd v3.5. It is by no means all encompassing, but intended to focus on the basic ideas needed to understand etcd without the distraction of less common API calls. All etcd APIs are defined in gRPC services, which categorize remote procedure calls (RPCs) understood by the etcd server. A full listing of all etcd RPCs are documented in markdown in the gRPC API listing.

etcd persistent storage files

Mon, 01 Jan 0001 00:00:00 +0000

This document explains the etcd persistent storage format: naming, content and tools that allow developers to inspect them. Going forward the document should be extended with changes to the storage model. This document is targeted at etcd developers to help with their data recovery needs.

Prerequisites

The following articles provide helpful background information for this document:

etcd data model overview
Raft overview (especially “5.3 Log replication” section).

Overview

Long leaving files

File name High level purpose

./member/snap/db

bbolt b+tree that stores all the applied data, membership authorization information & metadata. It’s aware of what's the last applied WAL log index ("consistent_index").

./member/snap/0000000000000002-0000000000049425.snap
./member/snap/0000000000000002-0000000000061ace.snap

Periodic snapshots of legacy v2 store, containing:

basic membership information
etcd-version

etcd API guarantees

Mon, 01 Jan 0001 00:00:00 +0000

etcd is a consistent and durable key value store. The key value store is exposed through gRPC Services. etcd ensures the strongest consistency and durability guarantees for a distributed system. This specification enumerates the API guarantees made by etcd.

APIs to consider

KV APIs
- Range
- RangeStream
- Put
- Delete
- Transaction
Watch APIs
- Watch
Lease APIs
- Grant
- [Revoke]
- Keep alive

KV API allows for direct reading and manipulation of key value store. Watch API allows subscribing to key value store changes. Lease API allows assigning a time to live to a key.

etcd versus other key-value stores

Mon, 01 Jan 0001 00:00:00 +0000

The name “etcd” originated from two ideas, the unix “/etc” folder and “d"istributed systems. The “/etc” folder is a place to store configuration data for a single system whereas etcd stores configuration information for large scale distributed systems. Hence, a “d"istributed “/etc” is “etcd”.

etcd is designed as a general substrate for large scale distributed systems. These are systems that will never tolerate split-brain operation and are willing to sacrifice availability to achieve this end. etcd stores metadata in a consistent and fault-tolerant way. An etcd cluster is meant to provide key-value storage with best of class stability, reliability, scalability and performance.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Alarm

The etcd server raises an alarm whenever the cluster needs operator intervention to remain reliable.

Authentication

Authentication manages user access permissions for etcd resources.

Client

A client connects to the etcd cluster to issue service requests such as fetching key-value pairs, writing data, or watching for updates.

Set up a local cluster

Mon, 01 Jan 0001 00:00:00 +0000

For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Starting a cluster

Run the following to deploy an etcd cluster as a standalone cluster:

$ ./etcd
...

If the etcd binary is not present in the current working directory, it might be located either at $GOPATH/bin/etcd or at /usr/local/bin/etcd. Run the command appropriately.

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

Users mostly interact with etcd by putting or getting the value of a key. This section describes how to do that by using etcdctl, a command line tool for interacting with etcd server. The concepts described here should apply to the gRPC APIs or client library APIs.

The API version used by etcdctl to speak to etcd may be set to version 2 or 3 via the ETCDCTL_API environment variable. By default, etcdctl on master (3.4) uses the v3 API and earlier versions (3.3 and earlier) default to the v2 API.

Why gRPC gateway

Mon, 01 Jan 0001 00:00:00 +0000

etcd v3 uses gRPC for its messaging protocol. The etcd project includes a gRPC-based Go client and a command line utility, etcdctl, for communicating with an etcd cluster through gRPC. For languages with no gRPC support, etcd provides a JSON gRPC gateway. This gateway serves a RESTful proxy that translates HTTP/JSON requests into gRPC messages.

Using gRPC gateway

The gateway accepts a JSON mapping for etcd’s protocol buffer message definitions. Note that key and value fields are defined as byte arrays and therefore must be base64 encoded in JSON. The following examples use curl, but any HTTP/JSON client should work all the same.

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

etcd provides a gRPC resolver to support an alternative name system that fetches endpoints from etcd for discovering gRPC services. The underlying mechanism is based on watching updates to keys prefixed with the service name.

Note that this feature is experimental because it depends on the google.golang.org/grpc/resolver package, which is still experimental in grpc-go.

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client:

Embedding etcd in a Go Application

Mon, 01 Jan 0001 00:00:00 +0000

The etcd embed go package provides a simple way to embed an etcd server directly into your application.

For more details, see the embed package documentation.

System limits

Mon, 01 Jan 0001 00:00:00 +0000

Request size limit

etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server.

Storage size limit

The default storage size limit is 2 GiB, configurable with --quota-backend-bytes flag. 8 GiB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.

etcd features

Mon, 01 Jan 0001 00:00:00 +0000

This document provides an overview of etcd features to help users better understand the features and related deprecation process. If you are interested in knowing about how features are developed in the etcd, please see these development guidelines.

The etcd features fall into three stages, experimental, stable, and unsafe. You can get the list of features by running etcd --help.

Experimental

In order to get early feedback, any new feature is usually added as an experimental feature. The experimental feature can be identified by looking at the flag name, which should have --experimental as a prefix. Please consider the following points while using an experimental feature:

API reference

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Auth` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
AuthEnable	AuthEnableRequest	AuthEnableResponse	AuthEnable enables authentication.
AuthDisable	AuthDisableRequest	AuthDisableResponse	AuthDisable disables authentication.
AuthStatus	AuthStatusRequest	AuthStatusResponse	AuthStatus displays authentication status.
Authenticate	AuthenticateRequest	AuthenticateResponse	Authenticate processes an authenticate request.
UserAdd	AuthUserAddRequest	AuthUserAddResponse	UserAdd adds a new user. User name cannot be empty.
UserGet	AuthUserGetRequest	AuthUserGetResponse	UserGet gets detailed user information.
UserList	AuthUserListRequest	AuthUserListResponse	UserList gets a list of all users.
UserDelete	AuthUserDeleteRequest	AuthUserDeleteResponse	UserDelete deletes a specified user.
UserChangePassword	AuthUserChangePasswordRequest	AuthUserChangePasswordResponse	UserChangePassword changes the password of a specified user.
UserGrantRole	AuthUserGrantRoleRequest	AuthUserGrantRoleResponse	UserGrant grants a role to a specified user.
UserRevokeRole	AuthUserRevokeRoleRequest	AuthUserRevokeRoleResponse	UserRevokeRole revokes a role of specified user.
RoleAdd	AuthRoleAddRequest	AuthRoleAddResponse	RoleAdd adds a new role. Role name cannot be empty.
RoleGet	AuthRoleGetRequest	AuthRoleGetResponse	RoleGet gets detailed role information.
RoleList	AuthRoleListRequest	AuthRoleListResponse	RoleList gets lists of all roles.
RoleDelete	AuthRoleDeleteRequest	AuthRoleDeleteResponse	RoleDelete deletes a specified role.
RoleGrantPermission	AuthRoleGrantPermissionRequest	AuthRoleGrantPermissionResponse	RoleGrantPermission grants a permission of a specified key or range to a specified role.
RoleRevokePermission	AuthRoleRevokePermissionRequest	AuthRoleRevokePermissionResponse	RoleRevokePermission revokes a key or range permission of a specified role.

service `Cluster` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
MemberAdd	MemberAddRequest	MemberAddResponse	MemberAdd adds a member into the cluster.
MemberRemove	MemberRemoveRequest	MemberRemoveResponse	MemberRemove removes an existing member from the cluster.
MemberUpdate	MemberUpdateRequest	MemberUpdateResponse	MemberUpdate updates the member configuration.
MemberList	MemberListRequest	MemberListResponse	MemberList lists all the members in the cluster.
MemberPromote	MemberPromoteRequest	MemberPromoteResponse	MemberPromote promotes a member from raft learner (non-voting) to raft voting member.

service `KV` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Range	RangeRequest	RangeResponse	Range gets the keys in the range from the key-value store.
Put	PutRequest	PutResponse	Put puts the given key into the key-value store. A put request increments the revision of the key-value store and generates one event in the event history.
DeleteRange	DeleteRangeRequest	DeleteRangeResponse	DeleteRange deletes the given range from the key-value store. A delete request increments the revision of the key-value store and generates a delete event in the event history for every deleted key.
Txn	TxnRequest	TxnResponse	Txn processes multiple requests in a single transaction. A txn request increments the revision of the key-value store and generates events with the same revision for every completed request. It is not allowed to modify the same key several times within one txn.
Compact	CompactionRequest	CompactionResponse	Compact compacts the event history in the etcd key-value store. The key-value store should be periodically compacted or the event history will continue to grow indefinitely.

service `Lease` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
LeaseGrant	LeaseGrantRequest	LeaseGrantResponse	LeaseGrant creates a lease which expires if the server does not receive a keepAlive within a given time to live period. All keys attached to the lease will be expired and deleted if the lease expires. Each expired key generates a delete event in the event history.
LeaseRevoke	LeaseRevokeRequest	LeaseRevokeResponse	LeaseRevoke revokes a lease. All keys attached to the lease will expire and be deleted.
LeaseKeepAlive	LeaseKeepAliveRequest	LeaseKeepAliveResponse	LeaseKeepAlive keeps the lease alive by streaming keep alive requests from the client to the server and streaming keep alive responses from the server to the client.
LeaseTimeToLive	LeaseTimeToLiveRequest	LeaseTimeToLiveResponse	LeaseTimeToLive retrieves lease information.
LeaseLeases	LeaseLeasesRequest	LeaseLeasesResponse	LeaseLeases lists all existing leases.

service `Maintenance` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Alarm	AlarmRequest	AlarmResponse	Alarm activates, deactivates, and queries alarms regarding cluster health.
Status	StatusRequest	StatusResponse	Status gets the status of the member.
Defragment	DefragmentRequest	DefragmentResponse	Defragment defragments a member’s backend database to recover storage space.
Hash	HashRequest	HashResponse	Hash computes the hash of whole backend keyspace, including key, lease, and other buckets in storage. This is designed for testing ONLY! Do not rely on this in production with ongoing transactions, since Hash operation does not hold MVCC locks. Use “HashKV” API instead for “key” bucket consistency checks.
HashKV	HashKVRequest	HashKVResponse	HashKV computes the hash of all MVCC keys up to a given revision. It only iterates “key” bucket in backend storage.
Snapshot	SnapshotRequest	SnapshotResponse	Snapshot sends a snapshot of the entire backend from a member over a stream to a client.
MoveLeader	MoveLeaderRequest	MoveLeaderResponse	MoveLeader requests current leader node to transfer its leadership to transferee.
Downgrade	DowngradeRequest	DowngradeResponse	Downgrade requests downgrades, verifies feasibility or cancels downgrade on the cluster version. Supported since etcd 3.5.

service `Watch` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Watch	WatchRequest	WatchResponse	Watch watches for events happening or that have happened. Both input and output are streams; the input stream is for creating and canceling watchers and the output stream sends events. One watch RPC can watch on multiple key ranges, streaming events for several watches at once. The entire event history can be watched starting from the last compaction revision.

message `AlarmMember` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
memberID	memberID is the ID of the member associated with the raised alarm.	uint64
alarm	alarm is the type of alarm which has been raised.	AlarmType

message `AlarmRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
action	action is the kind of alarm request to issue. The action may GET alarm statuses, ACTIVATE an alarm, or DEACTIVATE a raised alarm.	AlarmAction
memberID	memberID is the ID of the member associated with the alarm. If memberID is 0, the alarm request covers all members.	uint64
alarm	alarm is the type of alarm to consider for this request.	AlarmType

message `AlarmResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
alarms	alarms is a list of alarms associated with the alarm request.	(slice of) AlarmMember

message `AuthDisableRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthDisableResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthEnableRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthEnableResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleAddRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the role to add to the authentication system.	string

message `AuthRoleAddResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleDeleteRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
role		string

message `AuthRoleDeleteResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleGetRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
role		string

message `AuthRoleGetResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
header		ResponseHeader
perm		(slice of) authpb.Permission

message `AuthRoleGrantPermissionRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the role which will be granted the permission.	string
perm	perm is the permission to grant to the role.	authpb.Permission

message `AuthRoleGrantPermissionResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleListRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthRoleListResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
roles		(slice of) string

message `AuthRoleRevokePermissionRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
role		string
key		bytes
range_end		bytes

message `AuthRoleRevokePermissionResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthStatusRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthStatusResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
enabled		bool
authRevision	authRevision is the current revision of auth store	uint64

message `AuthUserAddRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string
password		string
options		authpb.UserAddOptions
hashedPassword		string

message `AuthUserAddResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserChangePasswordRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the user whose password is being changed.	string
password	password is the new password for the user. Note that this field will be removed in the API layer.	string
hashedPassword	hashedPassword is the new password for the user. Note that this field will be initialized in the API layer.	string

message `AuthUserChangePasswordResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserDeleteRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the user to delete.	string

message `AuthUserDeleteResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserGetRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string

message `AuthUserGetResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
roles		(slice of) string

message `AuthUserGrantRoleRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
user	user is the name of the user which should be granted a given role.	string
role	role is the name of the role to grant to the user.	string

message `AuthUserGrantRoleResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserListRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthUserListResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
users		(slice of) string

message `AuthUserRevokeRoleRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string
role		string

message `AuthUserRevokeRoleResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthenticateRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string
password		string

message `AuthenticateResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
token	token is an authorized token that can be used in succeeding RPCs	string

message `CompactionRequest` (api/etcdserverpb/rpc.proto)

CompactionRequest compacts the key-value store up to a given revision. All superseded keys with a revision less than the compaction revision will be removed.

API reference: concurrency

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Lock` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

The lock service exposes client-side locking facilities as a gRPC interface.

Method	Request Type	Response Type	Description
Lock	LockRequest	LockResponse	Lock acquires a distributed shared lock on a given named lock. On success, it will return a unique key that exists so long as the lock is held by the caller. This key can be used in conjunction with transactions to safely ensure updates to etcd only occur while holding lock ownership. The lock is held until Unlock is called on the key or the lease associate with the owner expires.
Unlock	UnlockRequest	UnlockResponse	Unlock takes a key returned by Lock and releases the hold on lock. The next Lock caller waiting for the lock will then be woken up and given ownership of the lock.

message `LockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
name	name is the identifier for the distributed shared lock to be acquired.	bytes
lease	lease is the ID of the lease that will be attached to ownership of the lock. If the lease expires or is revoked and currently holds the lock, the lock is automatically released. Calls to Lock with the same lease will be treated as a single acquisition; locking twice with the same lease is a no-op.	int64

message `LockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader
key	key is a key that will exist on etcd for the duration that the Lock caller owns the lock. Users should not modify this key or the lock may exhibit undefined behavior.	bytes

message `UnlockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
key	key is the lock ownership key granted by Lock.	bytes

message `UnlockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader

service `Election` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

The election service exposes client-side election facilities as a gRPC interface.

Authentication

Mon, 01 Jan 0001 00:00:00 +0000

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Note:

This is just a stub which needs to be filled and updated with more information on authentication. The text above is just a code example.

Configuration options

Mon, 01 Jan 0001 00:00:00 +0000

You can configure etcd through the following:

Command-line flags
Environment variables: every flag has a corresponding environment variable that has the same name but is prefixed with ETCD_ and formatted in all caps and snake case. For example, --some-flag would be ETCD_SOME_FLAG.
Configuration file

Warning

Caution: If you mix-and-match configuration options, then the following rules apply.

Command-line flags take precedence over environment variables.
If you provide a configuration file all command-line flags and environment variables are ignored.

Command-line flags

Flags are presented below using the format --flag-name DEFAULT_VALUE.

Role-based access control

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Authentication was added in etcd 2.1. The etcd v3 API slightly modified the authentication feature’s API and user interface to better fit the new data model. This guide is intended to help users set up basic authentication and role-based access control in etcd v3.

Special users and roles

There is one special user, root, and one special role, root.

Transport security model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication. Note that etcd doesn’t enable RBAC based authentication or the authentication feature in the transport layer by default to reduce friction for users getting started with the database. Further, changing this default would be a breaking change for the project which was established since 2013. An etcd cluster which doesn’t enable security features can expose its data to any clients.

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Starting an etcd cluster statically requires that each member knows another in the cluster. In a number of cases, the IPs of the cluster members may be unknown ahead of time. In these cases, the etcd cluster can be bootstrapped with the help of a discovery service.

Once an etcd cluster is up and running, adding or removing members is done via runtime reconfiguration. To better understand the design behind runtime reconfiguration, we suggest reading the runtime configuration design document.

Run etcd clusters as a Kubernetes StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

Below demonstrates how to perform the static bootstrap process as a Kubernetes StatefulSet.

Example Manifest

This manifest contains a service and statefulset for deploying a static etcd cluster in kubernetes.

If you copy the contents of the manifest into a file named etcd.yaml, it can be applied to a cluster with this command.

$ kubectl apply --filename etcd.yaml

Upon being applied, wait for the pods to become ready.

Run etcd clusters inside containers

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with Docker using the static bootstrap process.

Docker

In order to expose the etcd API to clients outside of Docker host, use the host IP address of the container. Please see docker inspect for more detail on how to get the IP address. Alternatively, specify --net=host flag to docker run command to skip placing the container inside of a separate network stack.

Failure modes

Mon, 01 Jan 0001 00:00:00 +0000

Failures are common in a large deployment of machines. A machine fails when its hardware or software malfunctions. Multiple machines fail together when there are power failures or network issues. Multiple kinds of failures can also happen at once; it is almost impossible to enumerate all possible failure cases.

In this section, we catalog kinds of failures and discuss how etcd is designed to tolerate these failures. Most users, if not all, can map a particular failure into one kind of failure. To prepare for rare or unrecoverable failures, always back up the etcd cluster.

Disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

etcd is designed to withstand machine failures. An etcd cluster automatically recovers from temporary failures (e.g., machine reboots) and tolerates up to (N-1)/2 permanent failures for a cluster of N members. When a member permanently fails, whether due to hardware failure or disk corruption, it loses access to the cluster. If the cluster permanently loses more than (N-1)/2 members then it disastrously fails, irrevocably losing quorum. Once quorum is lost, the cluster cannot reach consensus and therefore cannot continue accepting updates.

etcd gateway

Mon, 01 Jan 0001 00:00:00 +0000

What is etcd gateway

etcd gateway is a simple TCP proxy that forwards network data to the etcd cluster. The gateway is stateless and transparent; it neither inspects client requests nor interferes with cluster responses. It does not terminate TLS connections, do TLS handshakes on behalf of its clients, or verify if the connection is secured.

The gateway supports multiple etcd server endpoints and works on a simple round-robin policy. It only routes to available endpoints and hides failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future.

gRPC proxy

Mon, 01 Jan 0001 00:00:00 +0000

The gRPC proxy is a stateless etcd reverse proxy operating at the gRPC layer (L7). The proxy is designed to reduce the total processing load on the core etcd cluster. For horizontal scalability, it coalesces watch and lease API requests. To protect the cluster against abusive clients, it caches key range requests.

The gRPC proxy supports multiple etcd server endpoints. When the proxy starts, it randomly picks one etcd server endpoint to use. This endpoint serves all requests until the proxy detects an endpoint failure. If the gRPC proxy detects an endpoint failure, it switches to a different endpoint, if available, to hide failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future.

Hardware recommendations

Mon, 01 Jan 0001 00:00:00 +0000

etcd usually runs well with limited resources for development or testing purposes; it’s common to develop with etcd on a laptop or a cheap cloud machine. However, when running etcd clusters in production, some hardware guidelines are useful for proper administration. These suggestions are not hard rules; they serve as a good starting point for a robust production deployment. As always, deployments should be tested with simulated workloads before running in production.

Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Overview

An etcd cluster needs periodic maintenance to remain reliable. Depending on an etcd application’s needs, this maintenance can usually be automated and performed without downtime or significantly degraded performance.

All etcd maintenance manages storage resources consumed by the etcd keyspace. Failure to adequately control the keyspace size is guarded by storage space quotas; if an etcd member runs low on space, a quota will trigger cluster-wide alarms which will put the system into a limited-operation maintenance mode. To avoid running out of space for writes to the keyspace, the etcd keyspace history must be compacted. Storage space itself may be reclaimed by defragmenting etcd members. Finally, periodic snapshot backups of etcd member state makes it possible to recover any unintended logical data loss or corruption caused by operational error.

Monitoring etcd

Mon, 01 Jan 0001 00:00:00 +0000

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging.

Debug endpoint

If --log-level=debug is set, the etcd server exports debugging information on its client port under the /debug path. Take care when setting --log-level=debug, since there will be degraded performance and verbose logging.

The /debug/pprof endpoint is the standard go runtime profiling endpoint. This can be used to profile CPU, heap, mutex, and goroutine utilization. For example, here go tool pprof gets the top 10 functions where etcd spends its time:

Performance

Mon, 01 Jan 0001 00:00:00 +0000

Understanding performance

etcd provides stable, sustained high performance. Two factors define performance: latency and throughput. Latency is the time taken to complete an operation. Throughput is the total operations completed within some time period. Usually average latency increases as the overall throughput increases when etcd accepts concurrent client requests. In common cloud environments, like a standard n-4 on Google Compute Engine (GCE) or a comparable machine type on AWS, a three member etcd cluster finishes a request in less than one millisecond under light load, and can complete more than 30,000 requests per second under heavy load.

Design of runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two phase config changes keep the cluster safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member, first inform the cluster of the new configuration and then start the new member.

Runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Reconfiguration requests can only be processed when a majority of cluster members are functioning. It is highly recommended to always have a cluster size greater than two in production. It is unsafe to remove a member from a two member cluster. The majority of a two member cluster is also two. If there is a failure during the removal process, the cluster might not be able to make progress and need to restart from majority failure.

Supported platforms

Mon, 01 Jan 0001 00:00:00 +0000

Support tiers

etcd runs on different platforms, but the guarantees it provides depends on a platform’s support tier:

Tier 1: fully supported by etcd maintainers; etcd is guaranteed to pass all tests including functional and robustness tests.
Tier 2: etcd is guaranteed to pass integration and end-to-end tests but not necessarily functional or robustness tests.
Tier 3: etcd is guaranteed to build, may be lightly tested (or not), and so it should be considered unstable.

Current support

The following table lists currently supported platforms and their corresponding etcd support tier:

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the versions supported by the etcd project.

Service versioning and supported versions

etcd versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following Semantic Versioning terminology. New minor versions may add additional features to the API.

The etcd project maintains release branches for the current version and previous release. For example, when v3.5 is the current version, v3.4 is supported. When v3.6 is released, v3.4 goes out of support.

Data Corruption

Mon, 01 Jan 0001 00:00:00 +0000

etcd has built in automated data corruption detection to prevent member state from diverging.

Enabling data corruption detection

Data corruption detection can be done using:

Initial check, enabled with --experimental-initial-corrupt-check flag.
Periodic check of:
- Compacted revision hash, enabled with --experimental-compact-hash-check-enabled flag.
- Latest revision hash, enabled with --experimental-corrupt-check-time flag.

Initial check will be executed during bootstrap of etcd member. Member will compare its persistent state vs other members and exit if there is a mismatch.

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Two components of etcd storage consume physical memory. The etcd process allocates an in-memory index to speed key lookup. The process’s page cache, managed by the operating system, stores recently-accessed data from disk for quick re-use.

The in-memory index holds all the keys in a B-tree data structure, along with pointers to the on-disk data (the values). Each key in the B-tree may contain multiple pointers, pointing to different versions of its values. The theoretical memory consumption of the in-memory index can hence be approximated with the formula:

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Note

The watch features are under active development, and their memory usage may change as that development progresses. We do not expect it to significantly increase beyond the figures stated below.

A primary goal of etcd is supporting a very large number of watchers doing a massively large amount of watching. etcd aims to support O(10k) clients, O(100K) watch streams (O(10) streams per client) and O(10M) total watchings (O(100) watching per stream). The memory consumed by each individual watching accounts for the largest portion of etcd’s overall usage, and is therefore the focus of current and future optimizations.

Benchmarking etcd v3

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

Benchmarking etcd v2.2.0-rc-memory

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Benchmarking etcd v2.2.0-rc

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

Benchmarking etcd v2.2.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Bootstrap another machine, outside of the etcd cluster, and run the hey HTTP benchmark tool with a connection reuse patch to send requests to each etcd cluster member. See the benchmark instructions for the patch and the steps to reproduce our procedures.

Benchmarking etcd v2.1.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Downgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to downgrading etcd clusters and applications.

Downgrading an etcd v3.x cluster

Upgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to upgrading etcd clusters and applications.

Upgrade policy

Before upgrading, note that etcd only supports the following two upgrade cases:

Patch upgrade: Upgrading between patch releases within the same minor version (e.g. 3.7.0 - 3.7.1).
Minor upgrade: Upgrading one minor version at a time (e.g. 3.6 - 3.7). Upgrades that skip a minor version are not supported and will likely fail. Update to the most recent patch version before upgrading to the next minor version.

Upgrading an etcd v3.x cluster

Upgrading from etcd v2.3

Upgrade a v2.3 cluster to v3.0

Downgrade etcd from v3.7 to v3.6

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd v3.7 to v3.6 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd v3.7 processes and replace them with etcd v3.6 processes
after enabling the downgrade, new features in v3.7 are no longer available to the cluster

Before starting a downgrade, read through the rest of this guide to prepare.

Downgrade checklists

Highlighted differences between v3.7 and v3.6:

Upgrade etcd from v3.5 to v3.6

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd v3.5 to v3.6 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.5 processes and replace them with etcd v3.6 processes
after running all v3.6 processes, new features in v3.6 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Update 3.5

Important

Before upgrading to 3.6, make sure that all of your 3.5 members are updated to 3.5.32 or later. Patch releases 3.5.24 through 3.5.26 fix several potential upgrade blockers; 3.5.32 adds --v2-deprecation=write-only-skip-check and extends etcdutl check v2store to inspect WAL records as well as the v2 snapshot.

Downgrade etcd from 3.5 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd 3.5 to 3.4 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd 3.5 processes and replace them with etcd 3.4 processes
after starting any 3.4 processes, new features in 3.5 are not longer available to the cluster

Before starting a downgrade, read through the rest of this guide to prepare.

Downgrade checklists

content/enhttps://etcd.io/docs/v3.5/op-guide/authentication/rbac.md

Warning

If your cluster enables auth, rolling downgrade from 3.5 isn’t supported because 3.5 changes a format of WAL entries related to auth. You can follow the authentification instructions to disable auth, and delete all users first.

Downgrade etcd from v3.6 to v3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd v3.6 to v3.5 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd v3.6 processes and replace them with etcd v3.5 processes
after enabling the downgrade, new features in v3.6 are no longer available to the cluster

Before starting a downgrade, read through the rest of this guide to prepare.

Downgrade checklists

Highlighted breaking changes from v3.6 to v3.5:

Upgrade etcd from 3.4 to 3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.4 to 3.5 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.4 processes and replace them with etcd v3.5 processes
after running all v3.5 processes, new features in v3.5 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Warning

When migrating from v2 with no v3 data, etcd server v3.2+ panics when etcd restores from existing snapshots but no v3 ETCD_DATA_DIR/member/snap/db file. This happens when the server had migrated from v2 with no previous v3 data. This also prevents accidental v3 data loss (e.g. db file might have been moved). etcd requires that post v3 migration can only happen with v3 data. Do not upgrade to newer v3 versions until v3.0 server contains v3 data.

Upgrade etcd from 3.3 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Warning

Upgrade etcd from v3.6 to v3.7

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd v3.6 to v3.7 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.6 processes and replace them with etcd v3.7 processes
after running all v3.7 processes, new features in v3.7 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Update 3.6

Important

Before upgrading to 3.7, make sure that all of your 3.6 members are updated to 3.6.11 or later. Earlier 3.6 patch releases may not be compatible with a rolling upgrade to 3.7.

Upgrade etcd from 3.2 to 3.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Warning

Upgrade etcd from 3.1 to 3.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
after running all v3.2 processes, new features in v3.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Warning

Upgrade etcd from 3.0 to 3.1

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes
after running all v3.1 processes, new features in v3.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Warning

Upgrade etcd from 2.3 to 3.0

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes
after running all v3.0 processes, new features in v3.0 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Warning

Issue triage guidelines

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up issue management.

The etcd issues are listed at https://github.com/etcd-io/etcd/issues and are identified with labels. For example, an issue that is identified as a bug will eventually be set to label area/bug . New issues will start out without any labels, but typically etcd maintainers and active contributors add labels based on their findings. The detailed list of labels can be found at https://github.com/kubernetes/kubernetes/labels

Following are few predetermined searches on issues for convenience:

PR management

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up PR management.

The etcd PRs are listed at https://github.com/etcd-io/etcd/pulls A PR can have various labels, milestone, reviewer etc. The detailed list of labels can be found at https://github.com/kubernetes/kubernetes/labels

Following are few example searches on PR for convenience:

Scope

These guidelines serves as a primary document for managing PRs in etcd. Everyone is welcome to help manage PRs but the work and responsibilities discussed in this document is created with etcd maintainers and active contributors in mind.

Attribution and Licensing

Mon, 01 Jan 0001 00:00:00 +0000

pgsql.cc republishes documentation written by six upstream projects. Every documentation page carries a one-line credit at the end naming the work it is derived from, its copyright holder, its licence, and a link back to the section of this page that covers it.

This page is that record. It states what was imported, when, from which pinned revision, under which licence, and where the complete upstream notice is retained.

Note

pgsql.cc is an independent community documentation project. It is not affiliated with, endorsed by, or an official publication of any of the projects below, nor of the PostgreSQL Global Development Group.

Search

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL Community Components on PGSQL.CC

pgBadger 13.x Release Notes

v13.2 · 2025-12-29

v13.1 · 2025-03-16

Annotate Command (annotate)

Command Options

Backup Annotation Option (--annotation)

Dynamic Configuration Settings

Features

Introduction

Development Status

News

New Distribution Tarball

Starter Guide

YAML Configuration Settings

Global/Universal

pgBadger 12.x Release Notes

v12.4 · 2023-12-25

v12.3 · 2023-11-27

Archive Get Command (archive-get)

Configuration: pgbouncer.ini

Description

Generic settings

logfile

Download and Installation

Choose a distribution channel

Installation

Pre-requirements for Mac OS

Psycopg

Load Balancing Fundamentals

User Guide (Debian/Ubuntu)

Introduction

pgBadger 11.x Release Notes

v11.8 · 2022-04-08

What HAProxy Is and How It Works

Archive Push Command (archive-push)

Command-Line Reference

Usage and options

Environment Configuration Settings

Global/Universal

Log

Usage: pgbouncer command

Synopsis

Description

User Guide (RHEL)

Introduction

pgBadger 10.x Release Notes

v10.3 · 2019-02-14

Backup Command (backup)

Command Options

Backup Annotation Option (--annotation)

Basic Features

3.3.1. Basic features : Proxying

PgBouncer compilation and installation

Building

DNS lookup support

Patroni REST API

Health check endpoints

PostgreSQL Logging Configuration

Minimum query logging

pgBadger 9.x Release Notes

v9.2 · 2017-07-27

Check Command (check)

Configuration Reference

Introduction

Source Releases Download

PgBouncer 1.25

PgBouncer 1.24

PgBouncer 1.23

PgBouncer 1.22

PgBouncer 1.21

PgBouncer 1.20

PgBouncer 1.19

PgBouncer 1.18

PgBouncer 1.17

PgBouncer 1.16

PgBouncer 1.15

PgBouncer 1.14

PgBouncer 1.13

PgBouncer 1.12

Backup Annotation Option (`--annotation`)

Backup Annotation Option (`--annotation`)

Split one large file with `-j`