Data Infrastructure for CTOs: Built, Run and Measured by ChistaDATA

Support targets are ChistaDATA SLA terms. Performance figures are from our published billion-row measurement on ClickHouse 26.7 and from named public deployments, each cited where it appears.

The data infrastructure decision a CTO is actually making

Every data infrastructure for CTOs conversation we join starts with the same three pressures. Event volume is growing faster than the budget that stores it. The business wants answers in seconds on data that arrived seconds ago, from dashboards, from customer-facing product features and from models that need fresh features. And the estate has to survive an audit, a region failure and a staff change without a vendor holding the keys. The engine choice, the schema, the topology and the operating model all follow from how those three are weighed, and so does the cost of the data infrastructure that results.

We are a vendor-neutral firm with a ClickHouse specialisation, which means the first thing we do is establish whether ClickHouse is the right engine for the workload at all. When it is not, we say so and route the work to MinervaDB’s PostgreSQL, MySQL and SQL Server practices. When it is, the rest of this page is the plan, and the numbers below are the reasons.

Table 1. The three pressures that shape data infrastructure for CTOs, translated into quantities we can put on a dashboard in week one.

The scale at which this data infrastructure has already been proven

A CTO evaluating data infrastructure deserves reference points larger than any proof of concept, and data infrastructure for CTOs at enterprise scale has a public record worth reading. The figures below are published by the operators themselves; we quote them with their source and we have not independently verified them. Our own billion-row measurement sits beside them so the mechanisms can be compared at two very different sizes.

Table 2. Public data infrastructure scale references beside our own measurement. Two orders of magnitude separate the lab from LogHouse; the mechanisms are the same.

The numbers that matter most to a CTO in that table are the 16x and the 190 million. Sixteen-fold compression on real telemetry is the difference between a storage line that is a rounding error and one that is a board topic. One hundred and ninety million rows per second of sustained ingest is the reason the same engine can serve the operational data warehouse, the product analytics API and the observability platform rather than three vendors doing one each.

What we build: the reference data infrastructure

The data infrastructure we design for most clients has four layers, and each one is chosen for a measurable reason. Sources feed a Kafka backbone through CDC or direct producers. ClickHouse holds the data as ReplicatedMergeTree tables across two or more shards with two replicas each, coordinated by a dedicated three-node ClickHouse Keeper quorum. Materialized views precompute what dashboards ask for; TTL moves cold parts to object storage; per-column codecs decide the bytes. Consumers read through distributed tables, tenant-scoped APIs and feature tables.

Three data infrastructure decisions carry most of the risk and most of the value, so they are made explicitly and documented before a single node is provisioned. The sort key of every large table, because it decides whether a tenant-scoped query reads 7 granules or 122,087. The insert path, because a thousand single-row inserts create a thousand parts while one insert of ten million rows creates one part in 0.17 seconds. And the Keeper placement, because a shared or under-provisioned Keeper is the most common cause of a ClickHouse outage we are called into.

CREATE TABLE analytics.events ON CLUSTER '{cluster}'
(
    tenant_id    UInt16,
    event_time   DateTime                CODEC(Delta(4), ZSTD(1)),
    event_date   Date MATERIALIZED toDate(event_time),
    user_id      UInt32                  CODEC(ZSTD(1)),
    event_type   LowCardinality(String),
    country      LowCardinality(String),
    device       LowCardinality(String),
    http_status  UInt16                  CODEC(T64, ZSTD(1)),
    latency_ms   UInt16                  CODEC(T64, ZSTD(1)),
    bytes_out    UInt32                  CODEC(T64, ZSTD(1)),
    INDEX idx_user_bf user_id TYPE bloom_filter(0.01) GRANULARITY 4
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/analytics/events', '{replica}')
PARTITION BY toYYYYMM(event_date)
ORDER BY (tenant_id, event_time)
TTL event_date + INTERVAL 90 DAY TO VOLUME 'cold',
    event_date + INTERVAL 730 DAY DELETE
SETTINGS index_granularity = 8192, storage_policy = 'hot_cold';

-- every choice above has a number behind it on the billion-row table:
-- ORDER BY (tenant_id, event_time): tenant-day query reads 7 of 122,127 granules, 8 ms
-- Delta + ZSTD on event_time: 1.0x under LZ4 alone, 4.2x with the codec chain
-- T64 + ZSTD on http_status: 8.2x → 11.4x; whole table 1.47x → 2.68x
-- bloom_filter on user_id: for the 2,875 ms non-key filter that PREWHERE alone cannot save

What we measured, so the numbers are ours to defend

Data infrastructure for CTOs is too often sold on benchmark slides nobody can rerun. We publish our measurements with the generator that produced the data, and we run the same discipline inside client estates. The figures below come from a billion-row multi-tenant events table on a two-core, 7 GB machine, larger than memory, with caches disabled where the engine allows, so they are conservative rather than flattering.

Table 3. Selected results from the billion-row lab that underpin our data infrastructure recommendations. Full method, DDL and generator are in the linked articles; every figure is a median of three runs with the query condition cache off unless stated.

Two companion measurements complete the picture for a CTO deciding where a workload belongs. On a 20-million-row copy of the same kind of table in PostgreSQL 16, a two-column aggregate read 3.85 GB of heap against 45.5 MiB in ClickHouse,

The same comparison measured the other side of the data infrastructure ledger: a primary-key point lookup ran at 0.1 ms in PostgreSQL against 5 to 9 ms in ClickHouse and a single-row UPDATE at 0.65 ms against an 8-second mutation. Those are the numbers behind our standing advice: the system of record stays on a row store, and everything that reads more than a few thousand rows to produce a few hundred moves to ClickHouse behind CDC. The full comparison is in columnar databases versus row-based databases, measured.

Where the budget goes: four levers, each with a measurement

Platform cost in data infrastructure for CTOs decomposes into bytes stored, bytes moved per query, cores kept busy and money paid for the right to use the software. ClickHouse gives an engineering team direct control of the first three and removes the fourth. The figure below places the published and measured numbers on each lever to scale.

How we run it: four phases and a 15-minute S1

The operating model is the part of data infrastructure for CTOs that decides whether year three looks like year one. We work across the whole lifecycle, from the first system.query_log extract to the quarterly business review, and every phase ends with an artefact the CTO can audit rather than a slide.

Table 4. Data infrastructure support envelope, 24×7×365, from the San Francisco Bay Area and eleven global offices. Targets are response times to a named engineer; resolution plans are agreed per contract.

The first thirty days of an engagement produce a specific set of numbers: the top twenty-five query shapes by p95 with their bytes read and granules selected, the compression ratio and codec of every column over 1 GiB, parts per partition and merge backlog per table, Keeper latency and queue depth, the last successful restore and how long it took, and the gap between each table’s ORDER BY and the predicates actually used against it. That set becomes the data infrastructure baseline against which every later change is measured, and it is why our recommendations arrive with a before-and-after rather than a promise.

Governance, compliance and the questions an auditor asks of data infrastructure for CTOs

Modern data infrastructure for CTOs is audited as often as it is benchmarked. ClickHouse provides the primitives, and we configure and evidence them: role-based access control with SQL-defined roles, row policies that scope every query to the tenant or region it is entitled to see, quotas that bound what any user or key can consume, settings profiles that stop a single analyst from taking a replica down, and system.query_log plus system.session_log as an audit trail that records who ran what against which table and how many rows it touched.

-- a customer-facing API role that can only see its own tenant
CREATE ROLE api_tenant_reader;
GRANT SELECT ON analytics.events TO api_tenant_reader;
CREATE ROW POLICY rp_events_tenant ON analytics.events
    FOR SELECT USING tenant_id = toUInt16(getSetting('SQL_tenant_id'))
    TO api_tenant_reader;

-- a quota that bounds any single key
CREATE QUOTA q_api_reader
    FOR INTERVAL 1 hour MAX queries = 100000, read_rows = 50000000000, execution_time = 3600
    TO api_tenant_reader;

-- the audit question: who read more than a billion rows from this table last week?
SELECT user, count() AS queries, sum(read_rows) AS rows_read, max(event_time) AS last_seen
FROM system.query_log
WHERE type = 'QueryFinish'
  AND has(tables, 'analytics.events')
  AND event_time >= now() - INTERVAL 7 DAY
GROUP BY user
HAVING rows_read > 1000000000
ORDER BY rows_read DESC;

On that foundation we map the estate to the frameworks the client is held to: GDPR and DPDP for personal data, HIPAA for health data, SOX for financial reporting, PCI DSS for card data and SOC 2 for the controls themselves. We do not hold certifications on a client’s behalf, and no data infrastructure vendor should claim to; we produce the technical evidence their auditors need, from encryption at rest and in transit through backup and restore records to access reviews generated from the system tables above.

When we will tell you not to use ClickHouse

The credibility of data infrastructure for CTOs advice rests on the cases where the answer is no. If the workload is dominated by primary-key point reads and row-level updates, the measurements above are decisive: 0.1 ms against 5 to 9 ms for the lookup, 0.65 ms against 8 seconds for the update. That workload belongs on PostgreSQL or MySQL, and MinervaDB runs those practices with the same 15-minute S1.

If the workload is large fact-to-fact joins without a dimensional model, the right-hand table has to fit in memory per query, and a redesign comes before an engine change. If the requirement is a transactional system of record, ClickHouse is not one and no configuration makes it one.

In every one of those cases the data infrastructure pattern we recommend is the same: keep the system of record where it is, stream changes into ClickHouse with CDC, and let each engine do the work it was built for. The measured cost of getting that split wrong is on this page; the measured benefit of getting it right is too.

How a data infrastructure for CTOs engagement starts

Further reading

• Why ClickHouse is so fast: 7 design choices, measured on 1 billion rows: the full method behind Figures 2 and 3.
• Columnar databases vs row-based databases, measured on 20 million rows: the PostgreSQL side of every comparison above.
• Building high-performance real-time analytics applications and Crafting the right data strategy.
• ChistaDATA GenAI support with ClickHouse and Data warehousing support and managed services.
• ChistaDATA University: ClickHouse training for the engineering teams who will own the platform.

ChistaDATA Inc. is not affiliated with ClickHouse, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc. Third-party scale figures are quoted from the linked publications and have not been independently verified. Test every change in staging before applying it to production and maintain a tested disaster-recovery posture.