ClickHouse and Linux Swap: Diagnosing Swap-Induced Latency and Configuring Hosts to Avoid It

ClickHouse assumes it is running on hosts where RAM operations stay RAM operations. When the kernel starts swapping ClickHouse pages to disk, p99 latency degrades by orders of magnitude before any ClickHouse-level metric looks alarming — the server believes it is reading memory while the kernel quietly turns those reads into disk I/O. This post covers how to confirm swap is the problem in minutes, why it happens on ClickHouse hosts specifically, and the host- and server-level configuration that prevents it. Everything here applies to self-managed ClickHouse on Linux; version-sensitive settings are pinned inline.

The symptom

Queries that normally return in hundreds of milliseconds intermittently take tens of seconds. system.query_log shows elevated query_duration_ms with no corresponding growth in read_rows or read_bytes. Host CPU shows kswapd active, and iowait climbs while ClickHouse’s own disk reads look normal.

Triage: confirm swap in three commands

First, is the host swapping at all, and is it active right now?

free -h
vmstat 1 5

In vmstat, the columns that matter are si/so (swap-in/swap-out, KiB/s). Non-zero so sustained across samples means the kernel is actively evicting anonymous pages — swap in free -h merely being “used” can be historical and harmless; active si/so is the problem.

Second, is it ClickHouse being swapped, or something else on the host?

grep VmSwap /proc/$(pidof clickhouse-server)/status

Non-trivial VmSwap (hundreds of MiB or more) against the clickhouse-server process confirms it directly.

Third, on kernels 4.20+, memory pressure-stall information tells you how much wall-clock time processes are losing to memory reclaim:

cat /proc/pressure/memory

A rising some avg10 during latency incidents correlates reclaim pressure with your query slowdowns.

From inside ClickHouse, the server’s view of host memory (sourced from /proc/meminfo) is exposed in system.asynchronous_metrics:

SELECT metric, value
FROM   system.asynchronous_metrics
WHERE  metric ILIKE '%swap%' OR metric ILIKE '%memory%'
ORDER  BY metric;

For incident timelines, the same metrics are retained historically in system.asynchronous_metric_log, which lets you align swap growth with the exact query window in system.query_log.

One correction worth making explicitly, because an older version of this article propagated it: system.processes does not report per-query swap usage — it accounts ClickHouse-allocated memory (memory_usage/peak_memory_usage), which the kernel may or may not have resident. The operating system is the only source of truth for swap. Diagnose swap at the OS layer, then correlate with ClickHouse memory accounting.

Why ClickHouse hosts swap

Three mechanisms cover nearly every case we see in support:

1. Server memory cap vs. page cache pressure. ClickHouse caps its own allocations at max_server_memory_usage_to_ram_ratio (default 0.9 of physical RAM; max_server_memory_usage overrides it as an absolute value). MergeTree reads flow through the OS page cache, which competes for the same RAM. Under heavy scans plus near-cap allocations, the kernel reclaims — and with default vm.swappiness (60 on most distributions), it will swap anonymous ClickHouse pages rather than only dropping cache.
2. Co-tenants. Backup agents, log shippers, or a co-located service spike their working set; the kernel evicts the coldest anonymous pages, which are often ClickHouse hash tables from a long-running aggregation.
3. Confusing OS swap with ClickHouse spilling. Deliberate disk spilling via max_bytes_before_external_group_by / max_bytes_before_external_sort is controlled, observable (system.query_log.ProfileEvents shows ExternalAggregationWrittenFiles and related counters), and bounded. OS swap is none of those. If memory pressure is query-driven, you want spilling configured and swap disabled — not the reverse.

Fix

Immediate mitigation (running incident, host has swap in use):

sysctl -w vm.swappiness=1

This stops the kernel preferring swap over cache reclaim without a restart. Do not run swapoff -a mid-incident on a host with significant swap in use — it forces the entire swapped set back into RAM at once and can OOM the host. Verify used swap is small relative to available RAM (free -h) before considering it.

Proper fix (maintenance window, staged, reversible). For dedicated ClickHouse hosts, the ClickHouse documentation’s recommendation is to disable swap entirely. Persist the following:

Test in staging before production, and take a verified backup before host-level changes — standard caveat, and swap changes on a loaded host are exactly where it matters.

Verification after the change: rerun the triage set. vmstat si/so should sit at 0 under normal load; VmSwap for clickhouse-server should be ~0 and stay flat; heavy aggregations should show external-aggregation ProfileEvents in system.query_log instead of host swap activity — that is the intended trade: bounded, observable spilling instead of unbounded, invisible swapping.

Prevention

Alert on the leading indicators, not the outage: sustained non-zero so in your node exporter’s node_vmstat_pswpout; PSI some avg10 above a few percent; VmSwap growth for the clickhouse-server process; and in ClickHouse-native monitoring, watch system.asynchronous_metric_log memory metrics alongside MemoryTracking. Review host co-tenancy quarterly — the most common regression we see is a newly deployed agent quietly changing the memory math on database hosts.

If you’re operating latency-sensitive ClickHouse and want this class of host-level tuning reviewed systematically, ChistaDATA’s 24×7 support and performance audit cover exactly this surface.