Removing Empty Parts In ClickHouse

The new feature of empty part removal was introduced in version 20.12. If TTL removes all rows from a part, older versions leave empty parts (with 0 rows) (see https://github.com/ClickHouse/ClickHouse/issues/5491). If you establish a TTL for your data, there are probably a lot of unused components in your system.

The updated version attempts to eliminate every space as soon as it is noticed. This one-time procedure is executed immediately following an upgrade. After that, TTL will automatically erase any empty portions.

When several replicas of the same table begin to delete empty spaces simultaneously, there is a problem. They can block one another due to the defect (https://github.com/ClickHouse/ClickHouse/issues/23292).

What we can do during an upgrade to prevent this issue:

1. Drop empty partitions before upgrading to reduce the number of vacant portions in the system.

SELECT concat('alter table ',database, '.', table, ' drop partition id ''', partition_id, ''';')
FROM system.parts WHERE active
GROUP BY database, table, partition_id
HAVING count() = countIf(rows=0)

2. One copy (in a shard) will be upgraded/restarted at a time. There won’t be a deadlock due to replicas waiting for one another if just one replica cleans empty portions. Restart the first replica, then wait for the replication queue to complete before restarting the second.

Removing empty parts can be disabled by adding remove_empty_parts=0 to the default profile.

$ cat /etc/clickhouse-server/users.d/remove_empty_parts.xml
<clickhouse>
    <profiles>
        <default>
            <remove_empty_parts>0</remove_empty_parts>
        </default>
    </profiles>
</clickhouse>