ClickHouse High Availability Architecture on 26.9: A Definitive 5-Layer Framework for Safe Failover

Reference architecture

The ClickHouse high availability architecture we recommend

The architecture in Exhibit 1 follows from a single principle: no failure domain may hold both copies of a shard or a majority of Keeper votes. With three ClickHouse availability zones, that means one replica of every shard in each of two data zones, and one Keeper member in each of the three zones. The third zone stores no table data. Its only role is to provide the deciding vote, so it can be a small zone or a modest instance in another region.

The cluster definition encodes the placement. Each shard lists one replica per data zone, and internal_replication leaves copying to ReplicatedMergeTree rather than to the Distributed engine. Macros carry the shard and replica identity, so the table DDL is identical on every server. That uniformity is what makes servers interchangeable during upgrades and rebuilds.

<!-- config.d/cluster.xml: ClickHouse high availability architecture, two shards across two data zones -->
<clickhouse>
    <remote_servers>
        <prod>
            <shard>
                <internal_replication>true</internal_replication>   <!-- replicas copy via Keeper log -->
                <replica><host>ch-s1r1.zone-a</host><port>9000</port></replica>
                <replica><host>ch-s1r2.zone-b</host><port>9000</port></replica>
            </shard>
            <shard>
                <internal_replication>true</internal_replication>
                <replica><host>ch-s2r1.zone-b</host><port>9000</port></replica>
                <replica><host>ch-s2r2.zone-a</host><port>9000</port></replica>
            </shard>
        </prod>
    </remote_servers>

    <!-- per server: these three values are the only difference between nodes in ClickHouse availability zones A and B -->
    <macros>
        <cluster>prod</cluster>
        <shard>01</shard>
        <replica>r1</replica>
    </macros>

    <!-- ClickHouse Keeper quorum: one member per zone, A, B and the witness zone C -->
    <zookeeper>
        <node><host>keeper1.zone-a</host><port>9181</port></node>
        <node><host>keeper2.zone-b</host><port>9181</port></node>
        <node><host>keeper3.zone-c</host><port>9181</port></node>
        <session_timeout_ms>30000</session_timeout_ms>
        <operation_timeout_ms>10000</operation_timeout_ms>
    </zookeeper>
</clickhouse>

-- ClickHouse high availability architecture: one logical table, two shards, two replicas each
CREATE DATABASE IF NOT EXISTS pay ON CLUSTER prod;

CREATE TABLE IF NOT EXISTS pay.txn_local ON CLUSTER prod
(
    txn_id       UInt64,
    merchant_id  UInt32,
    created_at   DateTime64(3, 'UTC'),
    amount       Decimal(18, 2),
    currency     LowCardinality(String),
    status       LowCardinality(String)
)
ENGINE = ReplicatedMergeTree(
    '/clickhouse/tables/{cluster}/{shard}/pay/txn_local',   -- one Keeper path per shard
    '{replica}'                                             -- one entry per replica under it
)
PARTITION BY toYYYYMM(created_at)
ORDER BY (merchant_id, created_at, txn_id)
SETTINGS index_granularity = 8192;

-- The entry point applications use; rows are routed to a shard by merchant
CREATE TABLE IF NOT EXISTS pay.txn ON CLUSTER prod AS pay.txn_local
ENGINE = Distributed('prod', 'pay', 'txn_local', cityHash64(merchant_id));

Replication between the two data zones is asynchronous by default. An insert is acknowledged once the receiving replica has written the part and recorded it in Keeper. The peer replica in the other zone fetches it moments later. For most analytical data that is the right trade. Where an acknowledged row must exist in both zones before the client is told it is safe, quorum inserts close the gap, at a cost that depends entirely on Keeper placement.

Change without downtime

A ClickHouse rolling upgrade with zero failed requests on 26.9

Most production downtime on mature clusters is planned: upgrades, configuration changes and host maintenance. That makes the ClickHouse rolling upgrade the most frequently exercised part of any ClickHouse high availability architecture. We started the cluster on ClickHouse 26.9.2.8 and upgraded it to 26.9.17.8 one server at a time, while a client wrote and read every 100 ms. The client failed over between all four servers, as a load balancer health-checking each node would. The upgrade order took zone B first, then zone A, so that each shard always kept one replica serving.

All 503 writes and 651 reads succeeded. 57 writes and 77 reads were retried on a second node, which is the client failover doing its job. Each server answered queries 2.4 to 3.3 seconds after starting and caught up within a further 1.3 to 2.6 seconds. One stop took 30.14 seconds rather than under a second: the server waited for six open connections to drain before shutting down. Change windows should allow for that, and load balancers should drain a node before it is stopped.

We automate the procedure with a gate before and after each server. The gate refuses to touch a node whose replicas are unhealthy, or whose shard peers are not active, because restarting it would take a shard offline. The script below ran unchanged against the lab cluster.

#!/usr/bin/env bash
# ClickHouse rolling upgrade / restart for a ClickHouse high availability architecture, one node at a time, with a gate before and after each node.
# Usage: rolling_restart.sh <node-host:tcp-port> <restart-command>
# Blast radius: one replica of one shard at a time. Rollback: restart the node on the previous binary.
set -euo pipefail
NODE="$1"; RESTART_CMD="$2"
HOST="${NODE%:*}"; PORT="${NODE#*:}"
CH="${CH_CLIENT:-clickhouse-client} --host ${HOST} --port ${PORT} --user ${CH_USER:-default}"
[ -n "${CH_PASSWORD:-}" ] && CH="${CH} --password ${CH_PASSWORD}"   # never hard-code the secret

# Gate 1: every replicated table on this node must be healthy and caught up before we touch it,
# and its peer replicas must be active, otherwise this restart takes a shard offline.
UNHEALTHY=$($CH -q "SELECT count() FROM system.replicas
                    WHERE is_readonly OR is_session_expired OR queue_size > 0 OR active_replicas < total_replicas")
if [ "${UNHEALTHY}" != "0" ]; then
    echo "ABORT: ${UNHEALTHY} replicated tables not healthy on ${NODE}; fix before restarting"; exit 1
fi
BEFORE=$($CH -q "SELECT version()")
echo "$(date -u +%T) ${NODE} healthy on ${BEFORE}, restarting"

# Restart (graceful stop drains connections; start may switch binary for an upgrade)
eval "${RESTART_CMD}"                     # e.g. "systemctl restart clickhouse-server"

# Gate 2: wait until the node answers, then until every replica queue is empty again
until $CH -q "SELECT 1" >/dev/null 2>&1; do sleep 1; done
until [ "$($CH -q "SELECT count() FROM system.replicas WHERE is_readonly OR queue_size > 0")" = "0" ]; do sleep 1; done
$CH -q "SYSTEM SYNC REPLICA pay.txn_local LIGHTWEIGHT"   # wait for the fetches already queued for this replica
echo "$(date -u +%T) ${NODE} back on $($CH -q "SELECT version()"), queues empty; safe to continue"

# Lab output for one node:
# 19:54:27 127.0.0.1:9502 healthy on 26.9.17.8, restarting
# 19:54:36 127.0.0.1:9502 back on 26.9.17.8, queues empty; safe to continue

Two controls make this safe at scale. Upgrade Keeper members separately from ClickHouse servers, one at a time, and never while a server upgrade is in progress. And hold each new version on one replica per shard for a soak period before the second replica follows. Patch releases on the 26.9 line have shipped almost daily in October, so the soak, not the mechanics, is the real governance decision.

Quantifying availability

An illustrative availability model for ClickHouse high availability architecture

Leadership teams usually ask for availability as a number. The model below shows how replica count and Keeper size combine, under one stated and deliberately simple assumption: each server and each Keeper member is independently available 99.5% of the time. The figures are illustrative arithmetic, not measurements and not a promise. Real outages are correlated: a zone, a deployment or an operator error takes several components at once. That is exactly why the zone-level tests above matter more than any table of nines.

# ClickHouse high availability architecture, illustrative availability arithmetic: independent failures, every component 99.5% available.
# Reads need one live replica per shard; writes also need a Keeper majority.
from math import comb

A = 0.995                                     # assumed availability of one server or Keeper member

def keeper_majority(n, a):                    # probability that a majority of n members is up
    k = n // 2 + 1
    return sum(comb(n, i) * a**i * (1 - a)**(n - i) for i in range(k, n + 1))

def shard_up(replicas, a):                    # at least one replica of the shard is up
    return 1 - (1 - a) ** replicas

for shards, replicas, keepers in [(2, 1, 0), (2, 2, 3), (2, 2, 5), (2, 3, 5)]:
    read = shard_up(replicas, A) ** shards
    write = read * (keeper_majority(keepers, A) if keepers else 1)
    print(shards, replicas, keepers,
          f"read {read:.5%} ({(1 - read) * 525600:.2f} min/yr)",
          f"write {write:.5%} ({(1 - write) * 525600:.2f} min/yr)")

Two points survive the simplifying assumption. First, a second replica buys almost two orders of magnitude of read availability, and it is the single largest improvement on the table. Second, once data is replicated, Keeper becomes the limiting factor for writes. Going from three to five members more than halves expected write downtime in this model. A five-member Keeper ensemble across three zones is laid out 2-2-1, which still leaves three of five votes when any one zone is lost.

Recommendations

A 90-day roadmap to a tested ClickHouse high availability architecture

Every change in this roadmap should be applied to a staging environment first, with a written rollback, and only then promoted to production. Replication protects availability; it does not replace backups. A bad mutation or an accidental TRUNCATE replicates to every replica in seconds.

Methodology

How we tested this ClickHouse high availability architecture, and the limits of the evidence

The test cluster ran four ClickHouse servers, two shards with two replicas each, and three ClickHouse Keeper members, all on one 2 vCPU host with 8 GiB of RAM. Zones were logical: a zone failure was emulated by stopping every process assigned to it. Keeper distance was emulated with a userspace TCP delay proxy between servers and Keeper. Raft traffic between Keeper members was not delayed. Servers started on 26.9.2.8 for the upgrade test and ran 26.9.17.8 afterwards; Keeper ran 26.9.17.8 throughout. The workload was a synthetic payments table of up to 7.1 million rows.

The evidence has clear limits. One host cannot reproduce real network partitions, packet loss, disk failure or the contention of production query load. Failure timings on real infrastructure will be longer, because detection depends on timeouts rather than refused connections. The availability model is arithmetic on an assumed component availability. We present these results as the minimum behaviour a well-configured cluster should show, and as a template for the drills every production estate should run on its own infrastructure.

Further reading

Related ChistaDATA research and sources

From ChistaDATA: eight failure drills on ClickHouse 26.9, our companion study of replica and Keeper failures, and ClickHouse performance observability, for the system tables used throughout this paper.

ClickHouse documentation: data replication, ClickHouse Keeper, the Distributed table engine and the changelog.

Measurements: ChistaDATA lab, October 2026, ClickHouse 26.9.2.8 and 26.9.17.8 on one 2 vCPU host, 4 servers and 3 Keeper members, synthetic data. Availability figures are illustrative arithmetic under stated assumptions. Test every recommendation on your own infrastructure before production use.