ClickHouse on Kubernetes – Part 2 | Running ClickHouse cluster on Amazon EKS

Kubernetes orchestration simplifies many common operational concerns like scheduling, auto-scaling, and failover. Usually, databases that support replication, sharding, and auto-scaling are well-suited for Kubernetes. ClickHouse and Kubernetes can perform better together.

At ChistaData, we are interested in writing the following series of blogs to explain the ClickHouse on Kubernetes topic.

  • ClickHouse on Minikube
  • ClickHouse on Google Kubernetes Engine ( GKE )
  • ClickHouse on Amazon Elastic Kubernetes Service ( Amazon EKS )

This is the second part of the series. You can find the first part here.

In this blog post, we will explain the complete details of the Installation and configurations of the ClickHouse cluster on Amazon EKS.

Overview of the Setup:

To complete the setup, we need to work on the following steps.

  • Creating IAM user and configurations
  • Installing AWS CLI and configurations
  • Installing eksctl
  • Installing Kubectl
  • Creating the Amazon EKS cluster
  • Creating Addon for the EBS CSI driver
  • Cloning cluster configs and configurations
  • Testing the connections and cluster status

Creating IAM user and configurations:

The first step we would say is to create the IAM user with proper configurations. The user should have the access to AWS management console and Access key/Secret keys. Make sure you already have the access to AWS management console, If not use the link to create one.

Once login the AWS console, open the IAM panel and click the create users section. Enter the user name and select the AWS management console and Access key/Secret key as shown below.

Then choose the AdministratorAccess as shown below in the permissions section, then click next.

Then complete the setup. Once you click the user name, you can see the Access key and Secret Key, as shown below.

Now, we are all good with the IAM user. The next step is we need to configure this in the local terminal using awscli.

Installing AWS CLI and configurations

The AWS Command Line Interface (AWS CLI) is a unified tool to manage your AWS services. Follow the below steps to Install the was client tool. ( link )

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Make sure you have the latest version of the awscli to avoid compatibility issues. After the installation, we need to configure the account using the command “aws configure” as shown below.

ubuntu@eksClick:~$ aws configure
AWS Access Key ID [None]: AKIAQY6KMRS4SHJUNVLM
AWS Secret Access Key [None]: xNvmA***********************************p3
Default region name [None]: us-east-2
Default output format [None]: json


ubuntu@eksClick:~$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************NVLM shared-credentials-file    
secret_key     ****************XGp3 shared-credentials-file    
    region                us-east-2      config-file    ~/.aws/config

Installing Eksctl:

Eksctl is a simple CLI tool for creating and managing clusters on EKS – Amazon’s managed Kubernetes service for EC2. The following steps can be used to install the eksctl client tool. ( link )

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

Installing Kubectl:

The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. The following steps can be used to install the “kubectl” client tool.

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Creating the Amazon EKS cluster

So far, we have installed and configured the necessary tools. The next step is, We need to create the Amazon EKS cluster. We will use the client tool “eksctl” to create the Amazon EKS cluster. The following configuration can be used to create the cluster.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eks-ClickHouseCluster
  region: us-east-2

managedNodeGroups:
  - name: eks-ClickHouse
    instanceType: t3.xlarge
    desiredCapacity: 1
    volumeSize: 60

privateCluster:
  enabled: false
  skipEndpointCreation: false

Copy this content and store them in the .yaml file. In our case, We have created the “AmazonEKS.yaml” file and stored that config. Once the file is created, we can run the following command to create the cluster.

eksctl create cluster -f AmazonEKS.yaml

This will create the EKS cluster for you. Below We are sharing the logs for the references.

2022-10-29 18:45:16 [ℹ]  eksctl version 0.115.0-dev+2e9feac31.2022-10-14T12:52:53Z
2022-10-29 18:45:16 [ℹ]  using region us-east-2
2022-10-29 18:45:17 [ℹ]  setting availability zones to [us-east-2c us-east-2b us-east-2a]
2022-10-29 18:45:17 [ℹ]  subnets for us-east-2c - public:192.168.0.0/19 private:192.168.96.0/19
2022-10-29 18:45:17 [ℹ]  subnets for us-east-2b - public:192.168.32.0/19 private:192.168.128.0/19
2022-10-29 18:45:17 [ℹ]  subnets for us-east-2a - public:192.168.64.0/19 private:192.168.160.0/19
2022-10-29 18:45:17 [ℹ]  nodegroup "eks-ClickHouse" will use "" [AmazonLinux2/1.23]
2022-10-29 18:45:17 [ℹ]  using Kubernetes version 1.23
2022-10-29 18:45:17 [ℹ]  creating EKS cluster "eks-ClickHouseCluster" in "us-east-2" region with managed nodes
2022-10-29 18:45:17 [ℹ]  1 nodegroup (eks-ClickHouse) was included (based on the include/exclude rules)
2022-10-29 18:45:17 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
2022-10-29 18:45:17 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
2022-10-29 18:45:17 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-east-2 --cluster=eks-ClickHouseCluster'
2022-10-29 18:45:17 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "eks-ClickHouseCluster" in "us-east-2"
2022-10-29 18:45:17 [ℹ]  CloudWatch logging will not be enabled for cluster "eks-ClickHouseCluster" in "us-east-2"
2022-10-29 18:45:17 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=us-east-2 --cluster=eks-ClickHouseCluster'
2022-10-29 18:45:17 [ℹ]  
2 sequential tasks: { create cluster control plane "eks-ClickHouseCluster", 
    2 sequential sub-tasks: { 
        wait for control plane to become ready,
        create managed nodegroup "eks-ClickHouse",
    } 
}
2022-10-29 18:45:17 [ℹ]  building cluster stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:45:19 [ℹ]  deploying stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:45:49 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:46:21 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:47:22 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:48:23 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:49:24 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:50:25 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:51:31 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:52:32 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:53:33 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:54:35 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:55:36 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:56:38 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-cluster"
2022-10-29 18:58:49 [ℹ]  building managed nodegroup stack "eksctl-eks-ClickHouseCluster-nodegroup-eks-ClickHouse"
2022-10-29 18:58:50 [ℹ]  deploying stack "eksctl-eks-ClickHouseCluster-nodegroup-eks-ClickHouse"
2022-10-29 18:58:50 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-nodegroup-eks-ClickHouse"
2022-10-29 18:59:21 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-nodegroup-eks-ClickHouse"
2022-10-29 19:00:17 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-nodegroup-eks-ClickHouse"
2022-10-29 19:01:26 [ℹ]  waiting for CloudFormation stack "eksctl-eks-ClickHouseCluster-nodegroup-eks-ClickHouse"
2022-10-29 19:01:27 [ℹ]  waiting for the control plane to become ready
2022-10-29 19:01:27 [✔]  saved kubeconfig as "/Users/sakthivel/.kube/config"
2022-10-29 19:01:27 [ℹ]  no tasks
2022-10-29 19:01:27 [✔]  all EKS cluster resources for "eks-ClickHouseCluster" have been created
2022-10-29 19:01:28 [ℹ]  nodegroup "eks-ClickHouse" has 1 node(s)
2022-10-29 19:01:28 [ℹ]  node "ip-192-168-1-0.us-east-2.compute.internal" is ready
2022-10-29 19:01:28 [ℹ]  waiting for at least 1 node(s) to become ready in "eks-ClickHouse"
2022-10-29 19:01:28 [ℹ]  nodegroup "eks-ClickHouse" has 1 node(s)
2022-10-29 19:01:28 [ℹ]  node "ip-192-168-1-0.us-east-2.compute.internal" is ready
2022-10-29 19:01:30 [ℹ]  kubectl command should work with "/Users/sakthivel/.kube/config", try 'kubectl get nodes'
2022-10-29 19:01:30 [✔]  EKS cluster "eks-ClickHouseCluster" in "us-east-2" region is ready

At the end, you might see the “EKS cluster <ClusterName> in <region> is ready” logs.

2022-10-29 19:01:30 [✔]  EKS cluster "eks-ClickHouseCluster" in "us-east-2" region is ready

from console,

Creating Addon for EBS CSI driver

Now we have the Amazon EKS cluster running. Before configuring the ClickHouse cluster, we need to make sure that we have configured the EBS CSI driver for the EKS cluster.

The Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver allows Amazon Elastic Kubernetes Service (Amazon EKS) clusters to manage the lifecycle of Amazon EBS volumes for persistent volumes.

The following steps can be used to add the EBS driver to cluster.

To create IAM Open ID Connect provider,

eksctl utils associate-iam-oidc-provider --region=us-east-2 --cluster=eks-ClickHouseCluster --approve

To create IAM role for service account,

eksctl create iamserviceaccount --name ebs-csi-controller-sa --namespace kube-system --cluster eks-ClickHouseCluster --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy --approve --role-only --role-name AmazonEKS_EBS_CSI_DriverRoleForClickHouse --region us-east-2

To create the add-on,

eksctl create addon --name aws-ebs-csi-driver --cluster eks-ClickHouseCluster --service-account-role-arn arn:aws:iam::01111111111111:role/AmazonEKS_EBS_CSI_DriverRoleForClickHouse --force --region us-east-2

The final output will be like below,

2022-10-29 23:48:48 [ℹ]  Kubernetes version "1.23" in use by cluster "eks-ClickHouseCluster"
2022-10-29 23:48:48 [ℹ]  using provided ServiceAccountRoleARN "arn:aws:iam::0111111111111:role/AmazonEKS_EBS_CSI_DriverRoleForClickHouse"
2022-10-29 23:48:48 [ℹ]  creating addon
2022-10-29 23:50:25 [ℹ]  addon "aws-ebs-csi-driver" active

Note: Make sure to replace the cluster name, region, user ID on the above commands. The ID, you can get it from AWS console user section.

You can use the following command to verify the EDS add-on status.

ubuntu@eksClick:~$ kubectl get pods -n kube-system | grep -i ebs
ebs-csi-controller-65d9ff4584-47nhn   6/6     Running   0          36h
ebs-csi-controller-65d9ff4584-ph4gc   6/6     Running   0          36h
ebs-csi-node-kwzlm                    3/3     Running   0          36h

From console,

Cloning cluster configs and configurations

The next step is, we need to configure the ClickHouse cluster. The configs are publicly available in our repository. You can clone them directly as following.

ubuntu@eksClick:~$ git clone https://github.com/ChistaDATA/clickhouse_lab.git
Cloning into 'clickhouse_lab'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 9 (delta 1), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (9/9), 29.61 KiB | 286.00 KiB/s, done.
Resolving deltas: 100% (1/1), done.

Once you have cloned the repository, you might see the following files under the folder “clickhouse_lab/ClickHouseCluster”.

ubuntu@eksClick:~$ cd clickhouse_lab/ClickHouseCluster/
ubuntu@eksClick:~/clickhouse_lab/ClickHouseCluster$ 
ubuntu@eksClick:~/clickhouse_lab/ClickHouseCluster$ ls -lrth
total 252K
-rw-rw-r-- 1 ubuntu ubuntu  254 Oct 28 15:13 create-zookeeper
-rw-rw-r-- 1 ubuntu ubuntu  241 Oct 28 15:13 create-operator
-rw-rw-r-- 1 ubuntu ubuntu  199 Oct 28 15:13 create-cluster
-rw-rw-r-- 1 ubuntu ubuntu 1.6K Oct 28 15:13 cluster.yaml
-rw-rw-r-- 1 ubuntu ubuntu 6.1K Oct 28 15:13 zookeeper.yaml
-rw-rw-r-- 1 ubuntu ubuntu 227K Oct 28 15:13 operator.yaml

Here you can see three .yaml files and the respective bash files. We can execute the bash files one by one, it will call the respective config and build the cluster.

First we need to call the operator script as shown below,

ubuntu@eksClick:~$ ./create-operator 
namespace/chista-operator created
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallations.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseinstallationtemplates.clickhouse.altinity.com created
customresourcedefinition.apiextensions.k8s.io/clickhouseoperatorconfigurations.clickhouse.altinity.com created
serviceaccount/clickhouse-operator created
clusterrole.rbac.authorization.k8s.io/clickhouse-operator-chista-operator created
clusterrolebinding.rbac.authorization.k8s.io/clickhouse-operator-chista-operator created
configmap/etc-clickhouse-operator-files created
configmap/etc-clickhouse-operator-confd-files created
configmap/etc-clickhouse-operator-configd-files created
configmap/etc-clickhouse-operator-templatesd-files created
configmap/etc-clickhouse-operator-usersd-files created
deployment.apps/clickhouse-operator created
service/clickhouse-operator-metrics created

Secondly, we need to call the zookeeper script,

ubuntu@eksClick:~$ ./create-zookeeper 
namespace/chista-zookeeper created
service/zookeeper created
service/zookeepers created
poddisruptionbudget.policy/zookeeper-pod-disruption-budget created
statefulset.apps/zookeeper created

Finally, need to call the cluster script as shown below,

ubuntu@eksClick:~$ ./create-cluster 
clickhouseinstallation.clickhouse.altinity.com/herc created

ClickHouse Cluster is created. We can verify this using the following command.

ubuntu@eksClick:~$ kubectl get pods -n chista-operator
NAME                                   READY   STATUS    RESTARTS   AGE
chi-herc-herc-cluster-0-0-0            2/2     Running   0          36h
chi-herc-herc-cluster-0-1-0            2/2     Running   0          36h
chi-herc-herc-cluster-1-0-0            2/2     Running   0          36h
chi-herc-herc-cluster-1-1-0            2/2     Running   0          36h
clickhouse-operator-58768cf654-h9nrj   2/2     Running   0          36h

From console,

As per the config ( cluster.yaml ), We have mentioned 2 replicas and 2 shards.

ubuntu@eksClick:~$ less cluster.yaml| grep -i 'repl\|shard'
          shardsCount: 2
          replicasCount: 2

Here,

  • “chi-herc-herc-cluster-0-0-0” and “chi-herc-herc-cluster-1-0-0” are the shards.
  • “chi-herc-herc-cluster-0-1-0” and “chi-herc-herc-cluster-1-1-0” are the respective replicas.

Shard 1:

chi-herc-herc-cluster-0-0-0
  \_chi-herc-herc-cluster-0-1-0

Shard 2:

chi-herc-herc-cluster-1-0-0
  \_chi-herc-herc-cluster-1-1-0

The cluster setup is completed!

Testing the connections and cluster status

You can use the following command to go directly login the ClickHouse shell.

ubuntu@ip-172-31-2-151:~$ kubectl exec -it chi-herc-herc-cluster-0-0-0 -n chista-operator -- clickhouse-client
Defaulted container "clickhouse" out of: clickhouse, clickhouse-log
ClickHouse client version 21.10.5.3 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 21.10.5 revision 54449.

chi-herc-herc-cluster-0-0-0.chi-herc-herc-cluster-0-0.chista-operator.svc.cluster.local :) show databases;

SHOW DATABASES

Query id: f0f16155-9967-479e-b439-e9babdae5799

┌─name────┐
│ default │
│ system  │
└─────────┘

2 rows in set. Elapsed: 0.013 sec.

You can also use the following method to login the OS shell then ClickHouse shell.

ubuntu@ip-172-31-2-151:~$ kubectl exec -it chi-herc-herc-cluster-0-0-0 /bin/bash -n chista-operator           
clickhouse@chi-herc-herc-cluster-0-0-0:/$ 
clickhouse@chi-herc-herc-cluster-0-0-0:/$ clickhouse-client 
ClickHouse client version 21.10.5.3 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 21.10.5 revision 54449.

chi-herc-herc-cluster-0-0-0.chi-herc-herc-cluster-0-0.chista-operator.svc.cluster.local :) show databases;

SHOW DATABASES

Query id: ee1e1378-4881-4c23-b76c-67ee3cc417c8

┌─name────┐
│ default │
│ system  │
└─────────┘

2 rows in set. Elapsed: 0.003 sec.

ClickHouse cluster status:

chi-herc-herc-cluster-0-0-0.chi-herc-herc-cluster-0-0.chista-operator.svc.cluster.local :) SELECT
:-]     cluster,
:-]     shard_num,
:-]     replica_num,
:-]     host_name
:-] FROM system.clusters
:-] WHERE cluster = 'herc-cluster'

SELECT
    cluster,
    shard_num,
    replica_num,
    host_name
FROM system.clusters
WHERE cluster = 'herc-cluster'

Query id: fc87564e-3602-4a35-a7ae-c39a2cd5109d

┌─cluster──────┬─shard_num─┬─replica_num─┬─host_name─────────────────┐
│ herc-cluster │         1 │           1 │ chi-herc-herc-cluster-0-0 │
│ herc-cluster │         1 │           2 │ chi-herc-herc-cluster-0-1 │
│ herc-cluster │         2 │           1 │ chi-herc-herc-cluster-1-0 │
│ herc-cluster │         2 │           2 │ chi-herc-herc-cluster-1-1 │
└──────────────┴───────────┴─────────────┴───────────────────────────┘

4 rows in set. Elapsed: 0.007 sec. 

From the above output, We have overall 4 nodes. For shard_num “1”, we have two nodes, and for shard_num “2” we have two nodes, and we can see the respective replicas as well. So, the configuration is perfect!

 

Conclusion:

Hopefully, this blog will help you understand the configurations involved in the ClickHosue cluster on Amazon EKS. We will continue with this series and come up with the final part ( GKE ) soon. Thank you!

 

 

 

 

 

About Sri Sakthivel M.D. 5 Articles
Oracle certified MySQL DBA. Have expertise knowledge on the MySQL and its related technologies. Love to learn the Open source databases. Currently focusing on Clickhouse and its internals. Active MySQL Blogger and Youtuber.
Contact: Website