ClickHouse Performance: Disabling Transparent Huge Pages (THP) in Linux

Tuning Linux Transparent Huge Pages (THP) for ClickHouse Performance

Table of Contents

Introduction

Transparent Huge Pages (THP) is a memory management feature in the Linux kernel that allows the operating system to manage memory pages more efficiently. It works by grouping smaller memory pages into larger pages, which can reduce the number of page table entries required by the operating system.

While THP can improve performance in certain workloads, it can also cause significant performance issues in databases like ClickHouse that perform a lot of memory mapping. This is because THP can cause delays when mapping and unmapping memory pages, which can result in increased CPU utilization, increased latency, and reduced throughput.

In ClickHouse, THP can cause problems when performing operations that involve memory mapping, such as reading and writing data to disk or executing queries. When THP is enabled, ClickHouse may experience higher CPU utilization, longer query execution times, and lower throughput, especially for workloads that involve large amounts of data.

To avoid these issues, it is recommended to disable THP for ClickHouse by setting the “transparent_hugepage” parameter to “never.” This can be done by adding the following lines to the system’s startup script, such as /etc/rc.local:

echo never &gt; /sys/kernel/mm/transparent_hugepage/enabled<br>echo never &gt; /sys/kernel/mm/transparent_hugepage/defrag

By disabling THP, ClickHouse can avoid the delays caused by memory mapping and achieve better performance for workloads that involve large amounts of data. However, it is important to note that disabling THP can increase memory usage and may require additional tuning of other system parameters, such as the size of the page cache and the amount of available memory.

Conclusion

In conclusion, while THP can improve performance in certain workloads, it can cause significant performance issues in databases like ClickHouse. Disabling THP can help improve performance for ClickHouse workloads that involve large amounts of data and memory mapping.

To read more about Linux and ClickHouse, do consider reading the following articles

About Shiv Iyer 211 Articles
Open Source Database Systems Engineer with a deep understanding of Optimizer Internals, Performance Engineering, Scalability and Data SRE. Shiv currently is the Founder, Investor, Board Member and CEO of multiple Database Systems Infrastructure Operations companies in the Transaction Processing Computing and ColumnStores ecosystem. He is also a frequent speaker in open source software conferences globally.