Data Fabric on Cloud-Native ClickHouse: 6 Proven Layers for Real-Time Analytics, AI and ML

Definition

What a data fabric is, and what a ClickHouse data fabric is not

The term is used loosely, so it is worth being exact. A data fabric is a metadata-driven integration and serving layer that spans distributed sources without forcing them into one physical store. It has six responsibilities that always travel together.

They are: connecting to sources without rewriting them, ingesting with a declared freshness and consistency contract, serving queries at the latency the consumers need, exposing a semantic layer so consumers do not need to know where a column came from, governing access and retention uniformly, and observing itself so freshness and cost are measured per dataset. Remove any one of those and what remains is a set of pipelines.

ChistaDATA’s data fabric puts ClickHouse at the serving core because the consumers that justify the investment, real-time dashboards, embedded product analytics, feature reads for machine learning, ad-hoc exploration over years of events, all need sub-second answers over columnar data at high concurrency. ClickHouse delivers that on commodity hardware and object storage, in your own cloud account, with no proprietary format between you and your data.

It is equally important to say what this data fabric is not. It is not a replacement for the transactional databases that own writes; those stay where they are and are read through change data capture. It is not a lakehouse replacement; Iceberg and Delta tables on object storage remain the long-term store of record for cold data and are queried in place or loaded from. And it is not a data mesh organisational model; a data fabric is the platform that a mesh, a central team or a single product team can run on. The distinction matters because the most expensive platform failures we review are platforms that tried to be all three.

Layer 2 · Ingestion

Four ingestion paths into the data fabric, and what each one guarantees

Ingestion is where these platforms most often go wrong, usually by treating every source the same way. ChistaDATA classifies each dataset by two questions: how fresh must it be, and how can the source be replayed if the pipeline fails? The answers select one of four paths, shown below with the consistency guarantee each actually provides rather than the one its documentation implies.

Path A: Kafka engine table with a materialized view

ClickHouse consumes the topic itself. A table with ENGINE = Kafka holds no data; a materialized view attached to it receives every polled block and writes it into a MergeTree table. Delivery is at-least-once: if a replica crashes between committing the Kafka offset and completing the insert, or the reverse, a block can be replayed. The platform handles that by declaring the dataset’s contract up front. Append-only events rely on insert-block deduplication (a block with the same hash is dropped by the replicated engine), and upserted entities use ReplacingMergeTree with a monotonic version.

-- ClickHouse 25.x / 26.x LTS: streaming ingestion for the data fabric
CREATE TABLE fabric.events_kafka ON CLUSTER '{cluster}'
(
    event_id    UUID,
    tenant_id   UInt32,
    entity_id   UInt64,
    event_type  LowCardinality(String),
    event_time  DateTime64(3, 'UTC'),
    payload     String
)
ENGINE = Kafka
SETTINGS
    kafka_broker_list        = '${KAFKA_BROKERS}',
    kafka_topic_list         = 'events.v2',
    kafka_group_name         = 'fabric-events-v2',
    kafka_format             = 'JSONEachRow',
    kafka_num_consumers      = 2,
    kafka_max_block_size     = 65536,
    kafka_handle_error_mode  = 'stream';

CREATE TABLE fabric.events ON CLUSTER '{cluster}'
(
    event_id    UUID,
    tenant_id   UInt32,
    entity_id   UInt64,
    event_type  LowCardinality(String),
    event_time  DateTime64(3, 'UTC') CODEC(DoubleDelta, ZSTD(3)),
    payload     String               CODEC(ZSTD(6)),
    ingested_at DateTime DEFAULT now()
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/fabric/events', '{replica}')
PARTITION BY toYYYYMMDD(event_time)
ORDER BY (tenant_id, entity_id, event_time)
TTL toDateTime(event_time) + INTERVAL 90 DAY TO VOLUME 'cold',
    toDateTime(event_time) + INTERVAL 24 MONTH DELETE
SETTINGS storage_policy = 'tiered', index_granularity = 8192;

CREATE MATERIALIZED VIEW fabric.events_mv ON CLUSTER '{cluster}'
TO fabric.events AS
SELECT event_id, tenant_id, entity_id, event_type, event_time, payload
FROM fabric.events_kafka;

Rows that fail to parse are routed by kafka_handle_error_mode = 'stream' to virtual columns (_error, _raw_message) that a second materialized view captures into a dead-letter table, so a malformed producer never stalls the consumer. Consumer lag is read from system.kafka_consumers and exported to Prometheus; the alert threshold is the dataset’s freshness contract, not a fixed number.

Path B: Kafka Connect ClickHouse sink

When Kafka Connect is already the organisation’s integration hub and a schema registry governs the topics, the official ClickHouse sink connector is the better fit. Its tasks scale independently of the cluster, poison records go to a dead-letter topic, and recent versions provide exactly-once delivery through a Keeper-backed state table; the connector’s release notes are the authority on which guarantee applies to the version deployed. This path is preferred for Debezium CDC topics, because the Connect cluster is already running the source connectors.

Path C: batch from object storage

For datasets where hourly freshness is enough and the lake is the source of truth, the platform loads partitions with INSERT INTO ... SELECT FROM s3() or iceberg() under an orchestrator. The correctness rule is idempotency by partition: a reload drops or replaces the partition rather than appending to it, so a retried job cannot double-count.

-- Idempotent partition load into the data fabric from Parquet on S3
INSERT INTO fabric.orders_staging
SELECT *
FROM s3('https://${LAKE_BUCKET}.s3.amazonaws.com/orders/dt=2026-09-24/*.parquet', 'Parquet')
SETTINGS max_threads = 16, input_format_parquet_filter_push_down = 1;

-- Validation query BEFORE swapping: expected row count and checksum
SELECT count(), sum(cityHash64(order_id, status, total)) FROM fabric.orders_staging;

ALTER TABLE fabric.orders REPLACE PARTITION '20260924' FROM fabric.orders_staging;

-- Validation query AFTER: the live partition matches staging
SELECT count() FROM fabric.orders WHERE toYYYYMMDD(order_date) = 20260924;

Path D: federation without a copy

Some sources should not be copied: an Iceberg table that is petabytes and queried monthly, or a regulated PostgreSQL schema that legally must stay in place. ClickHouse’s Iceberg, Delta Lake, PostgreSQL and MySQL table engines let the platform query these in place, pushing predicates down where the engine supports it. Every such query pays the source’s latency and network cost, so federation is recorded in the catalog as a deliberate choice for cold or rarely used data, never as a default for anything a dashboard depends on.

Cloud native deployment

Deployment topology: the data fabric on Kubernetes with object storage

Cloud native, for a data fabric, means something specific: the ClickHouse cluster is declared as a Kubernetes custom resource and reconciled by an operator; storage is split between local NVMe for hot data and object storage for cold data and backups; and every supporting component (Keeper, Kafka Connect, orchestrator, Prometheus) runs in the same cluster with its own scaling. The topology below is the one ChistaDATA deploys on EKS, GKE, AKS and on-premises Kubernetes alike.

# ClickHouseInstallation (Altinity clickhouse-operator CRD), abbreviated
apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: fabric
  namespace: data-fabric
spec:
  configuration:
    zookeeper:
      nodes:
        - host: keeper-0.keeper.data-fabric.svc
        - host: keeper-1.keeper.data-fabric.svc
        - host: keeper-2.keeper.data-fabric.svc
    clusters:
      - name: fabric
        layout:
          shardsCount: 2
          replicasCount: 3
        templates:
          podTemplate: ch-pod
          dataVolumeClaimTemplate: ch-nvme
  templates:
    podTemplates:
      - name: ch-pod
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - topologyKey: topology.kubernetes.io/zone
                  labelSelector:
                    matchLabels: { clickhouse.altinity.com/chi: fabric }
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:${CH_LTS_VERSION}
              resources:
                requests: { cpu: "16", memory: "64Gi" }
                limits:   { cpu: "16", memory: "64Gi" }
    volumeClaimTemplates:
      - name: ch-nvme
        spec:
          storageClassName: local-nvme
          accessModes: [ReadWriteOnce]
          resources: { requests: { storage: 2Ti } }

A few of the rules encoded there deserve explanation. Requests equal limits so that pods get Guaranteed QoS and are never evicted under node pressure; inside the container, max_server_memory_usage_to_ram_ratio is kept below 0.9 so ClickHouse leaves headroom for the page cache. Anti-affinity is required, not preferred, across zones, so a zone failure costs one replica per shard and never quorum.

Keeper runs as its own StatefulSet on separate PVCs; co-locating Keeper with data pods is the single most common cause of replication stalls we see in reviews. A PodDisruptionBudget of two of three per shard lets node upgrades proceed without a read outage. Backups run clickhouse-backup from a CronJob to a versioned bucket with cross-region replication, and the restore is drilled quarterly with a timed RTO.

Scaling follows the measurement: replicas are added for read concurrency or availability, shards only when a single replica’s NVMe or merge throughput is the bottleneck, which shows as a growing system.merges backlog or a rising parts count per partition. Ingestion workers scale separately, so a Kafka surge never competes with query CPU.

Layer 6 · Governance

Governance and security across the data fabric

Governance in a data fabric is only real if it is enforced by the serving engine rather than by convention in each tool. ClickHouse’s role-based access control gives the platform one model that every consumer inherits: identity from LDAP or SSO mapped to roles, grants per database and table, row policies that restrict what each role can see, settings profiles that cap what each role can spend, and quotas that cap how often. The audit trail is system.query_log and system.session_log, retained and shipped to the observability store.

-- Data fabric governance: one model, enforced by the engine
CREATE ROLE fabric_analyst ON CLUSTER '{cluster}';
CREATE ROLE fabric_ml_export ON CLUSTER '{cluster}';

GRANT SELECT ON fabric.* TO fabric_analyst;
GRANT SELECT ON fabric.events, fabric.features_daily TO fabric_ml_export;

-- Tenant isolation: analysts see only their tenant's rows
CREATE ROW POLICY tenant_scope ON fabric.events
    FOR SELECT USING tenant_id = toUInt32(getSetting('SQL_tenant_id'))
    TO fabric_analyst;

-- Spend limits per role
CREATE SETTINGS PROFILE analyst_limits
    SETTINGS max_memory_usage = 20000000000,
             max_execution_time = 60,
             max_result_rows = 1000000
    TO fabric_analyst;

CREATE QUOTA analyst_quota
    FOR INTERVAL 1 HOUR MAX queries = 5000, read_rows = 50000000000
    TO fabric_analyst;

-- Column masking for direct identifiers, exposed as a view
CREATE VIEW fabric.events_masked AS
SELECT event_id, tenant_id, cityHash64(entity_id) AS entity_key,
       event_type, event_time
FROM fabric.events;

Compliance obligations map onto the same primitives. Retention is a TTL expression, so the evidence that data older than the policy is gone is a query against system.parts. Erasure requests under GDPR or DPDP are lightweight deletes scoped by entity, followed by a proof-of-deletion query that is stored with the ticket. Encryption in transit is TLS on every port, including the Keeper ports; encryption at rest is the cloud KMS on the NVMe volumes and the buckets. SOC 2 and HIPAA evidence packs are generated from the same system tables the observability layer already reads.

Observability

How the data fabric measures itself

Every layer exports its health through ClickHouse’s own system tables and the Prometheus endpoint, rendered in Grafana and alerted against SLOs with error budgets. The metrics are per dataset, because a platform that is “99.9% healthy” overall can still be serving one stale dashboard to the board. The table below lists what is measured at each layer and where the number comes from.

-- Data fabric serving SLO check: p95 per fingerprint over the last hour, worst first
SELECT
    normalizedQueryHash(query)               AS fingerprint,
    count()                                  AS runs,
    quantile(0.95)(query_duration_ms)        AS p95_ms,
    formatReadableSize(avg(read_bytes))      AS avg_read,
    any(substring(query, 1, 90))             AS sample
FROM system.query_log
WHERE type = 'QueryFinish'
  AND event_time > now() - INTERVAL 1 HOUR
  AND user NOT IN ('default', 'monitoring')
GROUP BY fingerprint
HAVING runs >= 10
ORDER BY p95_ms DESC
LIMIT 20;

Delivery

How ChistaDATA delivers a data fabric

Every engagement follows the same four phases, each with an exit gate the customer signs off. The elapsed time for a first production dataset set is typically eight to twelve weeks; most of that is measurement and validation, not engineering.

ChistaDATA provides the phases as ClickHouse consulting for assessment and design, platform engineering for the build, and either 24×7 ClickHouse support for a customer-operated data fabric or ClickHouse managed services where ChistaDATA operates it. Migrations from Redshift, Snowflake, BigQuery, Druid, Elasticsearch or a legacy warehouse into the data fabric follow the ClickHouse migration method, one dataset at a time behind the same reconciliation gates.