Exporting ClickHouse Metrics to Prometheus

Introduction

ClickHouse provides a way to expose metrics for scraping it from Prometheus. There are two steps involved in exposing ClickHouse metrics to Prometheus:

1. Configure ClickHouse to publish the metrics http endpoint.
2. Configure Prometheus to scrape the data using ClickHouse metrics endpoint.

ClickHouse configuration

<prometheus>
     <endpoint>/metrics</endpoint>
     <port>9126</port>
     <metrics>true</metrics>
     <events>true</events>
     <asynchronous_metrics>true</asynchronous_metrics>
 </prometheus>

I am running ClickHouse on Ubuntu 20.04 and the above prometheus section is added in ClickHouse configuration (/etc/clickhouse-server/config.xml).  While entering the port, use an unused TCP port and enable it in the Linux firewall.  Here the port is enabled for TCP traffic using the below command:

sudo ufw allow 9126/tcp

ClickHouse will start a new endpoint on the specified port in the format http://<clickhouse-server-ip>:9126/metrics

Here I am running ClickHouse on a virtual machine which has IP address 192.168.0.193 and prometheus is running as a docker container on the same machine.

Refer the ClickHouse documentation on below link for details about each configuration parameters:

https://clickhouse.com/docs/en/operations/server-configuration-parameters/settings/#server_configuration_parameters-prometheus

After adding the prometheus section in ClickHouse server configuration, restart the service.

sudo systemctl restart clickhouse-server

Once the server is restarted successfully, we can validate the metrics endpoint with below URL:

http://192.168.0.193:9126/metrics and you will get a result similar to the following:

Prometheus Configuration

I am setting up Prometheus as a docker container on the same ClickHouse host.

First step is to create a configuration file for prometheus on the host machine which will be mounted on the Prometheus docker container.  The file is created in the path /home/chistadata/clickhouse_exporter-master/docker/prometheus/prometheus.yml

---
global:
  scrape_interval:     1s
  evaluation_interval: 1s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: clickhouse
    static_configs:
      - targets: ['192.168.0.193:9126']

We can see two jobs added under scrape_configs section:

The job prometheus is for pulling metrics from prometheus itself and the second job clickhouse is the one we are interested in. You can see the target is pointed to the ClickHouse metrics endpoint 192.168.0.193:9126.

Now start the prometheus docker container:

docker run -p 9090:9090  –name prometheus   -v /home/chistadata/clickhouse_exporter-master/docker/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml  prom/prometheus

The Prometheus container port 9090 is forwarded to the host machine’s port 9090.  Once the container is up and running, we can access Prometheus UI using the URL http://192.168.0.193:9090/

To validate the scrape jobs are running, check Targets option under Status menu.  All the jobs should be shown in UP state.

Search for ClickHouse related metrics or use the Metrics Explorer button to view all the metrics imported to Prometheus.

In the below screenshot we are using ClickHouseAsyncMetrics_OSMemoryAvailable key for showing the graph:

Conclusion

With this simple runbook you are now able to export ClickHouse metrics to Prometheus.