Introducing CAS: Drop-in Compute-Storage Separation for ClickHouse® MergeTree Tables
TL;DR: Content Addressed Storage (CAS), new in Antalya 26.6, is a new extension to the ClickHouse storage model that lets you store MergeTree data in shared object storage. Most importantly, it lets you scale storage without scaling compute.
“Bang your head against it long enough, and something’s bound to work.”
– English saying
Compute-storage separation is a critical problem in modern analytic systems. Analytic databases follow the lead of Snowflake and place a single copy of table data on shared object storage. ClickHouse® has migrated in this direction over the course of several years, but until now open source builds did not yet offer a complete solution.
Altinity Antalya 26.6 introduces a new extension to the ClickHouse storage model, Content Addressed Storage (CAS), that offers a simple way to adopt shared object storage for MergeTree tables. Users can turn it on by upgrading to the latest Antalya build and making a simple configuration change. It works with all MergeTree table engines.
The Shared Storage Problem
Most databases are designed for single-node architecture. In order to scale up such architectures, the memory, the disk and the number of CPUs in the system must increase. But there is a limit to how much these can increase within a single server.
That’s where shared-nothing architecture comes in. In it, each node has its own processing power and its own allocated slice or copy of data. This provides massive parallelism, as it distributes the data and processing across multiple nodes. Databases adopted this architecture under a massive parallel processing name (MPP).
The perfect example of an MPP database is open source ClickHouse. Every server comes with CPUs and its own local storage. Multiple servers in a cluster improve compute capacity by replication and sharding. Every node has its own local storage.
But there is one critical limitation to shared-nothing architecture: it cannot scale storage and compute independently.
Users and database developers were looking for an architecture where a single copy of the data could be attached to multiple database nodes that would distribute query workload. SnowFlake pioneered this approach. DataBricks followed the same pattern a few years later. In such architectures, there is a pool of data, usually placed on object storage, and multiple query processing nodes that fetch data from this pool for query processing. Processing nodes can be added or removed on demand. Storage lives on its own.
Image source – Snowflake blog
In 2020 ClickHouse started to transition to using object storage as a storage option for MergeTree tables named S3 disks. It went through multiple iterations. Object storage has different characteristics compared to block storage. In particular, the latency is much higher, which requires different data organization and fetching techniques. Over the years, ClickHouse support for object storage rapidly evolved. Compact parts, multiple caches, prefetch and other tricks were closing the performance gap between local storage and object storage. But one problem remained unsolved for a long time – data sharing. Despite using object storage, ClickHouse kept using a shared nothing approach where every replica and shard uses its own data copy!
The challenge was apparent, and ClickHouse developers responded with zero-copy replication. Zero-copy replication means that once one ClickHouse replica places data to an S3 disk, it does not need to be replicated to other ClickHouse replicas. Multiple replicas connect to the same shared data on S3. Problem solved? Well, not quite.
Unfortunately, the underlying design of ClickHouse S3 disk makes zero-copy replication unreliable. In short, ClickHouse S3 disk storage consists of three systems that must coordinate in a transactional way: data references are stored on local storage and replicated as a regular ClickHouse table, data itself is stored on object storage, and replication state is tracked in Keeper separately for every replica. You may refer to the MergeTree over S3 architecture article for more detail. Keeping those three data sources in sync is almost impossible in real life. Most users presumably suffered in silence, so the practical problems are not well known. Our friends at Tinybird released a fascinating story about their experiences. Most prominently, the design does not delete orphan files properly. The lack of effective garbage collection is a key drawback in the zero-copy replication approach.
The problems of zero-copy replication led the upstream ClickHouse team to deprecate it in 2025 (#82250). The code is there, but all tests are removed.
The Arrival of CAS
It took time to arrive at a better approach to the compute-storage separation problem in ClickHouse. We started with RFC 54644 trying to fix zero-copy replication but ran into multiple roadblocks. A year later we launched Project Antalya. Project Antalya adds Apache Iceberg as a storage layer to ClickHouse, and stateless swarm nodes as scalable compute. It allows sharing data between ClickHouse nodes and external systems via Iceberg, and therefore uses Iceberg as a single data copy not just for ClickHouse but for other external systems as well. It’s a great architecture that enables access to data from potentially thousands of query services.
Antalya works really well, but it also adds complexity in exchange for making storage broadly accessible to other applications. We needed a simpler ClickHouse-only solution. Mikhail Filimonov, Altinity Principal engineer and one of the best software engineers I ever worked with, racked his brains about this problem a lot. He locked himself into a room for a month with a plethora of AI tools. This is how CAS was born.
The concept of content addressed storage is not new. It works by referring to files using a hash of the content instead of a conventional file name. Git works this way, for example. ClickHouse already used CAS for some internal use cases, like block deduplication. We just took it to the file level. We built it as an extension of the existing storage policy model in the simplest way possible.
The main idea is that every MergeTree table data file is referenced by a hash of its content. Many servers share one object storage pool. There is no byte duplication, no zero-copy bookkeeping in Keeper, no per-replica local disk reference state that grows with data volume, and no mutable per-blob refcount. Only the hash. The map between hashes and content is stored in immutable manifest files and an in-memory reftable. Nothing is stored locally, and everything is on object storage.
The architecture resembles Iceberg, and with good reason. Keeping all information “on disk” establishes a single source of truth and simplifies problems like recovery. It’s a scalable and proven solution.
Instead of replacing MergeTree with a new engine, we added a different metadata type CAS for object storage disks. That significantly simplifies both implementation and migration.
But that’s enough theory. Let me show you how it works, then we’ll get back to the technical details.
CAS in Action
CAS extends the ClickHouse architecture without taking anything away, so it can be applied to working ClickHouse systems without any redesign. In order to start using it, just upgrade to Antalya builds 26.6 and above. There was a format change in 26.6.4, so please use the latest build.
Once you upgrade to the build that supports CAS nothing immediately happens. In order to start using it, the CAS disk must be defined in ClickHouse storage configuration:
config.d/cas_disk.xml
—--------------------
<clickhouse>
<storage_configuration>
<disks>
<cas>
<type>object_storage</type>
<object_storage_type>s3</object_storage_type>
<metadata_type>cas</metadata_type>
<cas_server_root_id>{replica}</cas_server_root_id>
<endpoint>https://s3.us-east-1.amazonaws.com/altialya-2fv4arm7-chi-clickhouse-data/cas/{cluster}</endpoint>
</cas>
<cas_cache>
<type>cache</type>
<disk>cas</disk>
<path>cas_cache/</path>
<max_size>50Gi</max_size>
</cas_cache>
</disks>
</storage_configuration>
</clickhouse>As a seasoned ClickHouse user can see, the configuration is not much different from a regular S3 disk. The key difference is the new metadata_type. ClickHouse supports several metadata types already, so we added a new one – cas. Also, it needs to distinguish different ‘clients’ of CAS, and those are signed by cas_server_root_id which needs to uniquely identify a cluster node. It is similar to {replica} in ReplicatedMergeTree tables. (Instead of the {replica} macro, one could use {server_uuid} as well.)
Since CAS is a disk, it needs to be added to storage policies to be used. For example, here is a policy that stores data on CAS only:
config.d/cas_policy.xml
—----------------------
<clickhouse>
<storage_configuration>
<policies>
<cas>
<volumes>
<cas>
<disk>cas_cache</disk>
</cas>
</volumes>
</cas>
</policies>
</storage_configuration>
</clickhouse>Or the tiered policy that adds CAS as a second tier:
config.d/cas_tiered.xml
—----------------------
<clickhouse>
<storage_configuration>
<policies>
<cas_tiered>
<volumes>
<default>
<disk>default</disk>
<volume_priority>1</volume_priority>
</default>
<cas>
<disk>cas_cache</disk>
</cas>
</volumes>
</cas_tiered>
</policies>
</storage_configuration>
</clickhouse>All ClickHouse features are fully supported. We can create tables on CAS disks, move parts and partitions between local disks, regular S3 and CAS, or we can use a CAS disk as a cold tier with table TTL expressions.
For the purpose of this article, I will load the “ontime” dataset to the CAS disk into a single node ClickHouse in Altinity.Cloud. Later I will scale it to more nodes and run some other experiments. In order to load the test dataset, one can use the IMPORT DATASET feature of Altinity.Cloud, or run it in plain SQL as follows:
CREATE TABLE ontime
ENGINE = ReplicatedMergeTree
PARTITION BY Year ORDER BY (Carrier, FlightDate)
SETTINGS storage_policy = 'cas', min_bytes_for_wide_part = '1G'
AS SELECT * FROM s3('https://altinity-clickhouse-data.s3.amazonaws.com/airline/data/ontime_bin/*.bin.*', 'NOSIGN', 'Native')
SETTINGS max_threads=8, max_insert_threads=8, input_format_parallel_parsing=0;Note, that the engine is ReplicatedMergeTree, but replication is only used for coordination. The data is only written once.
It takes less than 4 minutes to load 200M rows, and the table is ready to run queries. No other magic, it just works.
In order to see how data is actually stored, we can use system tables. We may start with system.parts to confirm all the data is on CAS:
select disk_name, sum(bytes), sum(rows) from system.parts where active and table='ontime' group by 1
┌─disk_name─┬──sum(bytes)─┬─sum(rows)─┐
1. │ cas_cache │ 16694819251 │ 201575308 │ -- 201.58 million
└───────────┴─────────────┴───────────┘There are also three new CAS-specific tables:
cas_mounts– resemblessystem.replicasbut at a disk level. It shows servers and disks mounted to CAS poolscas_logandcas_gc_log– helpful to track what is happening inside CAS
For example, cas_log grouped by event_type can give you a hint of what was happening inside CAS:
┌─event_type─────────┬─count()─┐
1. │ blob_delete │ 237 │
2. │ blob_put │ 5776 │
3. │ blob_retire │ 237 │
4. │ blob_reuse_adopt │ 275 │
5. │ build_publish │ 858 │
6. │ build_start │ 858 │
7. │ gc_fence │ 126 │
8. │ gc_fold_begin │ 24 │
9. │ gc_fold_end │ 24 │
10. │ gc_recheck_verdict │ 237 │
11. │ gc_retire_observe │ 237 │
12. │ indegree_zero │ 237 │
13. │ manifest_delete │ 711 │
14. │ manifest_put │ 858 │
15. │ precommit │ 858 │
16. │ ref_drop │ 474 │
17. │ ref_repoint │ 237 │
18. │ ref_resolve │ 2210 │
19. │ root_add │ 13828 │
20. │ root_remove │ 474 │
└────────────────────┴─────────┘Let’s test adding a new replica to confirm we can scale compute nodes easily. The Altinity Operator for ClickHouse and Altinity.Cloud make it super easy to add replicas. With regular ReplicatedMergeTree storage, ClickHouse would need to replicate data over the network to a new replica that takes quite a lot of time. With CAS, it took only 14 seconds to initialize the ontime table on a new replica. One can see it from cas_log once replica goes online:
SELECT
now() - uptime() started,
min(event_time) - started cas_start,
max(event_time) - started cas_end
FROM system.cas_log ┌─────────────started─┬─cas_start─┬─cas_end─┐
1. │ 2026-09-03 17:51:06 │ 13 │ 27 │
└─────────────────────┴───────────┴─────────┘On bigger datasets it may take longer. We still need to measure how it works on multi-terabyte data.
Architecture Highlights
Under the hood, CAS is a MetadataStorage backend that stores each MergeTree part file once, using its content hash as the object key. ClickHouse nodes share the same object storage pool and publish references to immutable manifests. CAS coordination data lives in the bucket rather than in Keeper.
Let me explain the main architecture decisions in more detail.
How Does CAS Store a Part?
Let’s start with the basic object model. A MergeTree part is a directory containing many files. CAS calculates a content hash for every file and stores it under a key derived from that hash. If another part contains the same file, it resolves to the same object, so there is no need to store another copy.
The part name does not point directly to these blobs. Instead, it points to a mutable ref, which points to an immutable manifest. The manifest lists the files belonging to the part and where their contents are stored:
part name → mutable ref → immutable manifest → immutable hashed blobs
Small metadata files, such as count.txt or columns.txt, can be embedded directly into the manifest. Larger files are stored as separate content-addressed blobs.
How Is a Part Published?
So how does a new part become visible? ClickHouse cannot publish the ref first and upload the data later. A crash in between would leave a visible part pointing to missing files.
CAS uses a carefully ordered publication sequence. First, the writer creates an immutable manifest. It then records a durable precommit ref before uploading or adopting the required blobs. Once all blobs are available, the precommit is promoted to a committed ref and the part becomes visible to normal readers.
Conditional object writes provide the coordination. If two servers try to create the same blob, only one write succeeds. The other server finds the existing blob and adopts it instead of uploading another copy. There is no separate metadata service involved in this decision. The coordination happens entirely within object storage.
Reading follows the chain in the opposite direction. ClickHouse resolves the part ref, validates the manifest, and then reads the required blob ranges from object storage. Files embedded in the manifest do not need an additional object request.
Object Storage As the Only Source of Truth
CAS puts more responsibility on object storage than a regular S3-backed disk. The bucket stores not only data, but also refs, manifests, mount leases, fencing state, and garbage collection metadata.
This does not remove Keeper from ReplicatedMergeTree. Keeper still coordinates the replication log and part-set consensus, as described below.
Every server has its own identity, writer epoch, and renewable mount lease. If a server loses the lease, a local fence prevents it from issuing more writes. Ambiguous operations fail closed: ClickHouse retries them instead of assuming that an uncertain write did not happen.
This requires real support for conditional object creation and replacement, ranged reads, resumable listing, stable object tokens, and exact-token deletion. It is not enough for an S3-compatible implementation to accept the corresponding headers—it has to implement the logic correctly.
AWS S3 and Google Cloud Storage support the required operations. Azure has not been validated yet.
Replication Behavior
CAS does not replace ClickHouse replication. ReplicatedMergeTree still uses Keeper for the replication log and part-set consensus, which is how other replicas learn that a new part has arrived. What CAS changes is how the receiving replica obtains the part data.
How does this differ from ClickHouse zero-copy replication? The goal is similar, but the ownership model is fundamentally different. Zero-copy replication says:
“Replica 2 should reference the same remote object that Replica 1 already uses.”
CAS says:
“The file’s hash identifies the object. Any replica can independently publish a reference to it.”
With zero-copy replication, replicas share remote object paths and use Keeper bookkeeping to coordinate ownership. CAS shares content identities instead.
A zero-copy part fetch can avoid transferring bytes by giving the receiving replica metadata that points to the existing remote files. The difficult part is proving that those files remain alive while ownership moves between replicas.
CAS handles this with a three-step relink process:
- The receiving replica records that it intends to use the shared files.
- It checks that the sending replica still owns the same part.
- Once confirmed, it publishes the part locally.
If any step has an uncertain result, CAS retries instead of guessing what happened. It downloads the part bytes only when it can confirm that the relink did not complete. This prevents a replica from publishing a part with missing data or publishing the same part twice.
Garbage Collection
This is arguably the biggest architectural difference, and addresses the orphan file problems that users have encountered in production use of zero-copy replication.
Zero-copy must track which replicas still reference a remote object. RFC 62936 (one of multiple attempts to fix zero-copy replication) describes WAL, ZooKeeper queues and shared snapshots intended to make that tracking safer.
CAS takes a different approach. It determines what is still needed by following the reachability chain:
- A committed or in-progress ref keeps its manifest alive.
- A live manifest keeps all of its blobs alive.
- When a ref is removed, its blobs may become eligible for cleanup if nothing else references them.
- Garbage collection marks an unreferenced blob and checks it again in a later round.
- It deletes only the exact object version that was marked, so a newer replacement remains safe.
When safety is uncertain, CAS delays deletion and checks again in later GC rounds. Cleanup happens automatically once the object is proven unused.
Important software engineering note: Mikhail proved the efficacy of this algorithm using TLA+ with help from Claude. This approach is a quantum step forward in developing reliable algorithms related to distributed storage management. We’ll have more to say about the CAS engineering process in future.
Hints and Tips
We are still collecting operational experience for CAS disks. Let me share a few hints.
CAS may suffer from the same issues as any S3-backed MergeTree. Requests are expensive, so if one writes a lot of data directly to CAS, AWS may apply rate limiting. You will see the following message in the log as well as a generic slowdown:
2026.09.04 07:21:28.037323 [ 872 ] {} <Error> AWSClient: Response status: 503, Slow Down
Here are a few hints that can make it more effective:
- Do not write frequently to CAS disks. Instead use local storage for high frequency inserts and merges and then move less fragmented data to CAS using TTL rules
- Use compact parts more aggressively. The default
min_bytes_for_wide_partis only 10M, it can be safely increased to 100M or more. There is alsomin_level_for_wide_partthat can be set to 3 or 4 to keep all low generation parts in compact format. - Shard the pool for big deployments. If cluster sharding is fixed, it can be done by adding the
{shard}macro to the endpoint. Also different tables may live in different pools.
See also other configuration tricks in the in-tree documentation.
Before CAS reaches GA we plan to create a detailed user guide and also apply additional server optimizations to make user experience smooth. It is going to be integrated into Altinity.Cloud as well.
Status and Future Plans
CAS disks are an experimental feature as of the Antalya 26.6 release. It has been extensively tested by all possible means. It passes all regular ClickHouse tests when using CAS instead of regular S3. We also developed our own test suite for CAS-specific features. The implementation is functionally complete for AWS S3 and GCP GCS. It still needs to be adapted to work with Azure blob storage.
Our next focus is performance. We have tested insert and query performance and will share results in a separate blog post. It works great for queries, but is not fully performant for inserts yet. Therefore it can be used for cold tier, but not as the only storage model.
Along the way we’ll also provide comparisons between storing data in CAS, which is readable only through ClickHouse, and putting table data in Iceberg, which is generally accessible to a wide range of query engines. These solve different use cases and have different trade-offs. The big advantage of CAS is that it’s a drop-in extension for existing ClickHouse applications.
We will keep experimenting with CAS in-house and work on improvements. We welcome everybody to try it out and report issues. It is open source and will always be that way. We are looking forward to having a GA version of CAS for compute-storage separation in OSS ClickHouse by the end of 2026!
ClickHouse® is a registered trademark of ClickHouse, Inc.; Altinity is not affiliated with or associated with ClickHouse, Inc.