Why I am not connected to my ClickHouse server?
There are several reasons for unsuccessful connection incidents to ClickHouse Server. The most common among those is “Connection reset by peer while reading from Socket” so how can troubleshoot such incidents? This blog post shares a few techniques/checklists to understand ClickHouse connection handling:
Step 1: Please check if your ClickHouse Server is running:
# systemctl status clickhouse-server
Step 2: Check for ClickHouse Server port configuration/conflicts issues:
- Are you using the appropriate port IP/interface?
- Confirm you are not attempting to connect with a secure port without tls/https or vice cersa
- By default, ClickHouse Server listens to only to localhost. So please confirm you are connecting to an interface which ClickHouse listens to
Note: Please remember to check SE Linux /firewall rules
Check the interfaces/ports ClickHouse listens to:
# sudo lsof -i -P -n | grep LISTEN
Note: If the ClickHouse configuration file is modified with uncommented <listen_host>0.0.0.0</listen_host> then ClickHouse server will listen to ipv4 interfaces and the localhost (used with clickhouse-client) will fail to connect with ClickHouse Server.
Step 3: Are you building a secured ClickHouse infrastructure? If yes, please follow the recommendations below:
- Confirm ClickHouse Service is using certificates which can be validated by the client successfully
- You can as well decide to disable certificate checks on the client (Note: This is risky from a data security perspective)
Step 4: Please read for anomalies in ClickHouser Server error log (var/log/clickhouse-server/clickhouse-server.err.log)
Step 5: ClickHouse Server has limitations on total number of open connections (by default this value is 4000), Please check for total number of sockets opened by ClickHouse Server:
# sudo lsof -i -a -p $(pidof clickhouse-server)
## Check for connection handling limitations with netstat
# system overall support limited number of connections it can handle
# netstat -st
Tcp: 4616277 active connections openings 2869861 passive connection openings 976920 failed connection attempts 22461 connection resets received 8 connections established 92372161 segments received 19885698 segments send out 9791132 segments retransmited 139 bad segments received. 62928 resets sent
# Finding Linux process ulimits (Max open files) cat /proc/$(pidof -s clickhouse-server)/limits