Use S3 Storage in ClickHouse : AWS Configuration

In this post, we would like to configure an AWS S3 bucket for use in ClickHouse step by step. In the next part of this series, we’ll talk about configuration on the ClickHouse server side.

Therefore, we would like to create and configure AWS IAM user and S3 bucket.

For AWS configuration, we created detailed information with screenshots. You can access this information here.

Create and Configure IAM User

You can find steps for creating user:

• Under AWS IAM, select “ADD Users“

• Enter username with credential type to “Access key – Programmatic access”.

• Do not add user to any groups or do not add tags to user(accept default options until step 5).

• In step 5, note down the access key and secret access key for use in the ClickHouse storage configuration.

• Click the user that you created in the “Users“ screen.
• Copy the ARN (Amazon Resource Name) and save it for use when configuring the access policy for the bucket.

Create and Configure S3 Bucket

Now, it is time to create and configure S3 bucket. Here is the steps:

• Select “Create Bucket” in S3 section.
• At that point, enter a bucket name (All other options should be set to default) and click “Create Bucket” at the bottom of the page.
• After creating bucket, you should see your new bucket under “Buckets“
• Check your bucket and copy the ARN. (It is used in creating storage policies at the ClickHouse server)
• Select the bucket’s link in the “Buckets” section and select “Create Folder” to create a folder to use as ClickHouse storage.
• Enter a folder name and select Create folder. (All other options should be set to default.)
• Check the created folder and copy the URL for use in the ClickHouse storage configuration.
• Select the created bucket and choose the “Permissions” tab.
• Click on the “Edit” button in the “Bucket Policy” section and add the policy above with your credentials and information.

{
    "Version": "2012-10-17",
    "Id": "Policy1",
    "Statement": [
        {
            "Sid": "CHS3_1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<YOUR_IAM_ID>:user/<YOUR_USER>"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<YOUR_BUCKET>",
                "arn:aws:s3:::<YOUR_BUCKET>/*"
            ]
        }
    ]
}

In this part of the article, we discuss S3 configuration for use in ClickHouse. In the next post, we’ll configure ClickHouse to use S3 as a disk.

References

• https://clickhouse.com/docs/en/guides/sre/configuring-s3-for-clickhouse-use/