ClickHouse® Keeper: The Long Road from “ZooKeeper in C++” to Something More

Engineering progress is rarely a single heroic rewrite. More often it is a long sequence of uncomfortable measurements, wrong hypotheses, small fixes, and one more benchmark run.

This article is a story about ClickHouse Keeper: why ClickHouse needed ZooKeeper in the first place, why replacing it was much harder than “just use Raft”, why Keeper solved many operational problems, why it still disappointed some users on performance, and how a recent round of investigation and optimization changed that picture.

It is also a story about not stopping at “good enough”. Keeper has been production-ready for many users for a long time. But for performance-sensitive workloads, the road from “works correctly” to “is obviously fast enough to replace ZooKeeper everywhere” turned out to be much longer.

Where the story starts: replication needs coordination

ClickHouse replication is eventually consistent by design.

A replicated table is not a system where every replica is guaranteed to be perfectly synchronized at every point in time. One replica may already have a new part, another may still be fetching it, a third may be merging something else. That is normal. The important part is that replicas eventually converge.

But even in an eventually consistent system, some decisions need stronger consistency guarantees.

Someone has to allocate block numbers. Someone has to decide which merge should be assigned. Distributed DDL queries need a single order. Some tasks need a single executor. In other words, ClickHouse can tolerate replicas being temporarily out of sync, but it still needs a small, highly reliable place where the cluster can agree on things.

Historically, that place became Apache ZooKeeper.

ZooKeeper entered ClickHouse history in March 2014. The first actual ZooKeeper-related code appears to be 5ceab75, authored by Michael Kolupaev on 2014-03-07.

The replication engine itself appeared a little later. 750cb44 added the first StorageReplicatedMergeTree skeleton, but without real ZooKeeper calls.

The first real use of ZooKeeper in the replication engine seems to be 7b6ce30, also authored by Michael Kolupaev, on 2014-03-22. That is where StorageReplicatedMergeTree started using ZooKeeper for actual replication metadata: table and replica znodes, /metadata, /replicas, /blocks, /block-numbers, and an ephemeral /is_active node for the active replica.

That historical detail is important. ClickHouse did not merely pick “some consensus service.” It adopted ZooKeeper’s coordination model: sessions, watches, ephemeral nodes, sequential nodes, a hierarchical namespace, and multi-operations.

That distinction became important later.

Why ZooKeeper worked so well for ClickHouse

ZooKeeper is, at its core, a distributed state machine with a filesystem-like data model. You have a tree of znodes. A znode can have data, children, metadata, versions. Clients have sessions. Some nodes can be ephemeral and disappear when the session dies. Clients can set watches and be notified when something changes.

For ClickHouse, this was a very convenient abstraction.

ReplicatedMergeTree could store replication queues, block numbers, part metadata, merge assignments, and other small pieces of coordination state in ZooKeeper. Distributed DDL could store a log of queries and let nodes execute them in a consistent order. The amount of data was small, but the correctness requirements were high.

This is the crucial split in ClickHouse architecture: the data itself can be replicated in an eventually consistent way, but the coordination metadata must be agreed on much more strictly.

ZooKeeper provided that.

It also had good read performance. Reads in ZooKeeper are local to the server the client is connected to, while writes go through consensus. That is not full linearizability for every read, but it is exactly the tradeoff many ZooKeeper-based systems were built around: linearizable writes, ordered session semantics, fast reads, and a simple API.

For a long time this was a very reasonable design.

Why ZooKeeper became painful

Over time, though, ZooKeeper became one of those components that people keep because it works, but nobody is particularly happy to operate.

The first complaint was obvious: Java.

ClickHouse is a fast C++ server. It is usually deployed by people who like the fact that it is a single native binary with predictable performance characteristics. ZooKeeper meant adding a JVM-based service next to it: heap sizing, garbage collector behavior, Java runtime, separate operational habits, separate tuning.

That alone was not enough to replace it. ZooKeeper was production-tested and reliable. But it made many ClickHouse users ask the same question again and again: why do I need to run this Java thing next to my C++ database?

Then there were the more operationally unpleasant problems.

One of them is zxid.

Strictly speaking, ZooKeeper’s zxid is a 64-bit value split into a 32-bit epoch and a 32-bit counter. Operationally, the painful part is the counter. After enough write transactions in one epoch, the counter rolls over and ZooKeeper needs to move to a new epoch. That means leader election. For clients, that means interruption.

For ClickHouse, interruption of ZooKeeper is not just a harmless reconnect. ClickHouse has to reconnect sessions and reconcile replicated table state with the coordination state. I do not mean that the ClickHouse server process restarts. I mean that table replicas effectively go through a restart/reconciliation path similar in spirit to SYSTEM RESTART REPLICA.

During that time, tables may still be available for reads, but writes can fail because the replicated table is temporarily read-only. Worse, this is not a scheduled maintenance window. It happens when the counter reaches the limit. Depending on write load, that may be days, weeks, or longer, but it is still a bad kind of failure: predictable in theory, annoying in practice, and very easy to forget until it happens again.

There is a similar but smaller problem with xid, the request identifier inside a client session. When that overflows, it affects one session rather than the whole ZooKeeper ensemble. That is better, but for ClickHouse a session reconnect can still trigger replica reconciliation and temporary write unavailability.

Then there are tuning issues. ZooKeeper has jute.maxbuffer, about 1 MB by default. ClickHouse can produce large requests, and in real installations this often needs to be increased. There are log and snapshot retention settings, heap settings, disk layout questions, and many small operational choices that become important only after something goes wrong.

And then there are the scary cases, like digest mismatch.

If one ZooKeeper node has damaged or divergent local state, the system can enter a deeply confusing mode where writes appear to succeed, but reads from another node do not show the expected state. From an operator’s point of view it looks like ZooKeeper stopped obeying the rules. For ClickHouse this is especially unpleasant, because ZooKeeper is the thing you use to avoid disagreement. If the coordination layer itself becomes suspicious, troubleshooting becomes very hard.

Finally, there was the maintenance story. ZooKeeper was not dead, but for a long time it did not feel like a fast-moving project. It was common to find old issues with stack traces that looked exactly like your production problem, followed by years of silence or “me too” comments. That does not inspire confidence when the component is sitting at the center of your replicated database.

So the desire to replace ZooKeeper was understandable.

Why not etcd or Consul?

The obvious question was: if ZooKeeper was painful, why not replace it with Consul or etcd?

People asked that very early. In 2017, issue #479 proposed Consul as a drop-in replacement for ZooKeeper. A year later, issue #1765 asked the same question about etcd.

On the surface, both suggestions made sense. Consul and etcd were widely used, easier to deploy in many environments, and did not require running a JVM next to ClickHouse.

But ClickHouse was not using ZooKeeper only as “a place with consensus.” It had grown around ZooKeeper semantics.

Replicated tables depended on sessions, watches, ephemeral nodes, sequential nodes, multi-operations, reconnect behavior, and the hierarchical znode model. Replacing ZooKeeper with another coordination system would mean either rewriting a large part of ClickHouse replication logic, or building a compatibility layer that reproduced enough ZooKeeper behavior to be trusted in production.

That compatibility-layer idea was tried too. There was an attempt to use zetcd, a ZooKeeper-compatible layer on top of etcd, discussed in #777. It was an attractive idea: keep the ZooKeeper protocol on the outside, use etcd underneath. But in practice, “mostly compatible” was not enough. The experiment ran into missing or incomplete semantics, crashes, and failures on real replicated-table operations.

Later, direct etcd support was attempted inside ClickHouse itself. PR #10376 added an experimental etcd-based implementation, and PR #17495 rebased that work. But the experiment was eventually abandoned. The hard part was not talking to etcd. The hard part was preserving the ZooKeeper behavior ClickHouse already relied on, especially around watches and session semantics.

So the lesson was not “etcd is bad” or “Consul is bad.” Both are good systems. They just expose a different model.

ClickHouse needed something very specific: a ZooKeeper-compatible coordination service that could preserve the existing replication model, pass the same kind of correctness tests, and still be developed in the direction ClickHouse needed.

That made the next step almost inevitable: instead of translating ZooKeeper semantics onto another system, implement those semantics directly.

Birth of ClickHouse Keeper

The direction that became ClickHouse Keeper started in issue #15090, opened by Alexey Milovidov in September 2020. The title was modest: “Add NuRaft to TestKeeper.” The body was even shorter: “And look what will happen.”

That is a very ClickHouse way of starting a large project.

The idea was different from the etcd attempt. Keeper would not translate ZooKeeper operations into another coordination model. It would keep the ZooKeeper protocol and the ZooKeeper-style data model on the outside, while replacing the internals with a C++ implementation based on Raft.

NuRaft was chosen as the first Raft implementation to try. It was small, had relatively few dependencies, had usable documentation, and was easy enough to start experimenting with. But choosing Raft was only the beginning.

This was not easy. The first implementation took close to a year of work by Alexander Sapin. It was not enough to “plug in Raft.” Keeper had to implement the ZooKeeper protocol, sessions, watches, snapshots, logs, write-ahead logs, leader/follower behavior, and all the details needed to behave like ZooKeeper from ClickHouse’s point of view.

The testing effort was also serious. Keeper was tested not only with ordinary ClickHouse tests, but also with Jepsen: crashes, network partitions, disk corruption, slow networks, and other failure modes that tend to expose incorrect distributed systems.

That part is important. A slow coordination system is annoying. An incorrect coordination system is catastrophic.

By August 2021, Alexander Sapin presented ClickHouse Keeper publicly at ClickHouse Meetup 54. The presentation is available as slides, and the recording starts at the Keeper part here.

In retrospect, the positioning in that presentation was already clear: Keeper was meant to be a ZooKeeper replacement compatible with the existing client protocol, but implemented in C++, using Raft, with better integration into ClickHouse and stronger control over logs, snapshots, checksums, compression, and testing.

The early Keeper work was mostly successful on correctness. Later problems were much more often operational or performance-related than “Keeper violates consensus.”

From working system to mature system

It would be nice if the story ended there: Keeper is written, Jepsen test passes, ZooKeeper problems disappear, everyone opens champagne.

Of course, that is not what happened.

Once Keeper started being used seriously, it became clear that correctness is only one part of production readiness. The first implementation gave ClickHouse a working ZooKeeper-compatible coordination service, but the next few years were about turning it into something operators could trust, debug, scale, reconfigure, and run under very different production conditions.

This is also where Keeper maintenance gradually moved from the original implementation work by Alexander Sapin to a longer, deeper stream of work led mostly by Antonio Andelic. A lot of what made Keeper mature after the first release was not one dramatic rewrite, but a sequence of architectural, operational, and performance changes.

Some of that work was about basic production readiness: compressed logs in #29223, four-letter-word commands in #28981, better behavior for those commands before quorum in #35992, and Jepsen checks in CI in #32998.

Some of it was deeper. One of the most important architectural changes was request preprocessing and real-time digest support in #37036 and #37555. Keeper request handling was split so operations could be preprocessed before commit, and replicas could continuously cross-check that they held the same state using a real-time digest.

There were also experiments that did not survive. Faster linearizable reads were tried in #38200, but that change was quickly reverted in #41436. That is a useful reminder: read performance in a coordination system is not just a performance problem. It is also a semantics problem.

Other work was about operations. Keeper learned to upload snapshots to S3 in #41342. Later, #50098 generalized Keeper logs and snapshots to use ClickHouse IDisk, rather than being tied directly to raw filesystem paths. Dynamic cluster membership arrived with Keeper reconfig in #49450. Keeper gained a native CLI client in #47414, and later an HTTP REST API and embedded UI in #78181.

Memory and storage became their own long-running theme. Keeper gained a memory soft limit in #57271. There was also a serious attempt to support RocksDB as an alternative Keeper storage backend in #56626. That experiment is worth mentioning even though RocksDB did not become the final direction and was later removed in #108000. It shows what problems Keeper was starting to face as it matured: large node counts, memory usage, snapshot size, storage layout, and recovery behavior.

So the first reality check was not “Keeper is incorrect.” The early Keeper work was mostly successful on correctness. Later problems were much more often about operational maturity, scale, performance, tooling, and edge cases.

That distinction matters. An incorrect coordination system is catastrophic. A correct but immature coordination system is painful, but it can be improved — and that is exactly what happened over the next several years.

Keeper stopped being just a ZooKeeper replacement

At first, Keeper was mostly “ZooKeeper, but written for ClickHouse.”

That was already valuable. It removed the JVM, avoided some ZooKeeper operational limits, gave ClickHouse control over the implementation, and made it possible to ship and test the coordination layer together with the database.

But over time Keeper became something more important: a coordination service that could evolve together with ClickHouse.

The first step in that direction was not a large marketing announcement, but a small protocol change. Keeper-specific operations started appearing because ClickHouse could do better than the generic ZooKeeper API for some of its own coordination patterns. One early example was FILTERED_LIST, added in v22.8, which allowed ClickHouse to list only selected children instead of fetching a full znode child list and filtering it on the client side.

That sounds like a minor optimization, but it changes the model. ZooKeeper can list all children under a znode. ClickHouse often needs only a subset: for example, children matching some condition around parts, replicas, or ephemeral state. When there are many children and many replicas repeatedly asking the same questions, moving that filtering into Keeper avoids unnecessary network traffic and client-side work.

More Keeper-specific operations followed. MULTI_READ, added in v22.10, allowed batched read operations. CHECK_NOT_EXISTS, added in v23.5, gave ClickHouse a negative version check. These early extensions were originally organized through a monotonic Keeper API version mechanism introduced in #39096.

That linear versioning was later replaced with independent, negotiable feature flags in #50796. This was a better fit for a distributed system where clients and servers may not all be upgraded at the same time. Instead of assuming one global API level, Keeper could advertise concrete capabilities.

After that, new operations were added directly in the feature-flag model. CREATE_IF_NOT_EXISTS landed in v23.9, and native recursive subtree deletion, REMOVE_RECURSIVE, landed in v24.10. By 2025, these extensions were mature enough that #83488 enabled the Keeper feature flags by default.

This is the important part: these were not random one-off hacks. They were a mechanism for evolving ClickHouse coordination beyond the Apache ZooKeeper API while still preserving compatibility where possible.

There was another side of the same story. Keeper was not trying to become a perfect, byte-for-byte clone of every ZooKeeper feature before it could be useful. Issue #59489, opened after running zookeeper-client-c and kazoo tests against Keeper, listed remaining compatibility gaps: persistent watches, removing watches, container and TTL nodes, null znode values, some ACL differences, SASL authentication, read-only mode without quorum, and other edge cases.

That is the balance Keeper had to strike: compatible enough to replace ZooKeeper for ClickHouse, but not restricted by ZooKeeper compatibility when ClickHouse needed a better primitive.

As long as Keeper only imitated ZooKeeper, users could reasonably think of it as one backend among others. But once ClickHouse started adding Keeper-specific protocol capabilities, Keeper became the place where ClickHouse coordination could be optimized for ClickHouse itself.

That did not mean ZooKeeper immediately stopped working. ClickHouse still tried to preserve compatibility where possible. But the center of gravity shifted. Newer features and optimizations increasingly had a natural home in Keeper, not in ZooKeeper.

This is why staying on ZooKeeper forever became less attractive by 2025. The old arguments against ZooKeeper were still there — Java, operations, limits, external dependency — but now there was another argument: ClickHouse itself was gradually moving beyond the ZooKeeper API.

By 2025, this became one of the reasons Altinity started thinking more actively about migration. We added Keeper support in the Altinity clickhouse-operator, and started testing. We did not want to maintain two different “correct” cluster architectures indefinitely. Operationally, having one target architecture is much better: one deployment model, one monitoring model, one runbook set, one set of recommendations.

But there was one big blocker.

Performance.

The performance problem

It is important to say this carefully: Keeper was not “unusable” before 2026. That would be wrong. It had already been production-ready for many users, and the official ClickHouse blog post “ClickHouse Keeper: A ZooKeeper alternative written in C++” described its use in ClickHouse Cloud and highlighted major advantages such as lower memory usage and operational simplicity.

But production-ready does not mean perfect for every workload.

For years, many people who tried Keeper in performance-sensitive scenarios saw the same disappointing result: it was slower than ZooKeeper.

Sometimes much slower.

This was especially frustrating because everyone expected the opposite. Keeper was C++. It was written by ClickHouse developers. It was designed specifically for ClickHouse. ZooKeeper was Java, older, more general-purpose. Surely Keeper should be faster?

But in real tests, Keeper could be three or four times slower, sometimes worse. At Altinity we saw the same thing. We had customers where ZooKeeper performance was already a concern, and replacing it with a slower Keeper was not a responsible decision.

And this was not just an internal Altinity observation. Users had been reporting similar patterns for years. In #37974, a user testing ClickHouse 22.3 saw a small gap in standalone mode, but a much larger drop with a three-node Keeper cluster. In #41045, another user reported a degenerate concurrent-write scenario where Keeper was dramatically slower than ZooKeeper. In #53798, the same pattern appeared again in high-concurrency testing on ClickHouse 23.7.

Not every report was a clean apples-to-apples comparison. Some involved unusual settings, old Keeper versions, or benchmark shapes that were not representative of normal ClickHouse workloads. But the repeated pattern was hard to ignore: for some write-heavy or high-concurrency scenarios, users could reproduce serious performance problems.

This created an awkward situation. Strategically, Keeper was the direction ClickHouse was moving. Operationally, we wanted to standardize on it. But practically, performance-sensitive users had a very good reason to stay on ZooKeeper.

In early 2026, after another round of disappointing benchmarks, I decided to stop waiting and investigate.

Partly this was a practical goal: we needed to know whether the Keeper could be fixed. Partly it was curiosity. And partly it was a good experiment in using AI models as an assistant while working with a large, unfamiliar, performance-sensitive C++ codebase.

First benchmark: make ClickHouse hurt ZooKeeper/Keeper

The first test was intentionally simple.

Create a ReplicatedMergeTree table, effectively with one partition, and insert many tiny blocks. The goal was to make local disk writes cheap, while creating a huge number of parts. Every part needs coordination metadata. Many small parts also create merges, and merges create more coordination activity.

This is not a full production workload, but it is a useful stress model for the coordination layer. It amplifies a real mechanism: Keeper load grows with cluster size, part count, insert rate, merge activity, and the number of replicas watching and updating metadata.

The result was immediately familiar: on the same hardware, ZooKeeper was much faster. In one run ZooKeeper 3.9 was around 7.5k requests/sec with median latency around 18 ms, while Keeper from master was around 2.8k requests/sec with median latency around 41 ms.

That was enough to reproduce the problem, but not convenient enough to investigate it. Running a full ClickHouse workload means running ClickHouse, Keeper or ZooKeeper, tables, inserts, merges, and then trying to separate coordination effects from everything else.

Luckily, ClickHouse already had keeper-bench, a tool for benchmarking ZooKeeper/Keeper-style workloads directly. With a smaller benchmark, it became possible to iterate quickly.

The benchmark reproduced the same conclusion: Keeper was significantly slower.

The next question was: where is the time going?

The system was not busy. It was waiting.

The first surprising observation was that there was no obvious CPU bottleneck.

Keeper was not burning a core. The disk was not saturated. The machine was mostly idle. Something was waiting for something else.

That is often the real performance problem in a coordination system. Linearizable writes are hard to scale horizontally because ordering matters. You cannot simply process dependent requests in arbitrary parallel threads. The wins usually come from batching, pipelining, avoiding unnecessary synchronization, and making sure unavoidable waits — especially fsync — do not create extra waits around them.

The first clear problem was read-after-write handling.

Keeper, like ZooKeeper, must preserve session semantics. If a client writes something and then reads from the same session, it must see its own write. That guarantee is important.

But Keeper was effectively implementing this too globally.

In a mixed workload, read requests acted as global batch separators. Imagine one session is sending writes and Keeper is building a batch. Then another unrelated session sends a read. Instead of just serving that read if it had no dependency on the writes from the first session, Keeper would flush the accumulated write batch, wait for it to commit, serve the read, and only then continue.

One read from one session could stop batching for everyone.

That is stronger than ZooKeeper semantics require. There is no useful causal relationship between two arbitrary sessions unless the application creates one. If session B did not perform the write and has no synchronization with session A, there is no reason its read should force all writes from A to complete first.

The correct barrier is per session.

Reads should wait for previous writes from the same session. They should not globally stop the pipeline for unrelated sessions.

This became PR #100125: introduce per-session read barriers instead of global ones. The PR was not merged, but it was a useful proof of the problem. In my tests, it improved RPS by about 30% and reduced latency by roughly 20–25% on the mixed workload.

That was a good result, but not enough.

Keeper was still behind ZooKeeper.

The deeper problem: the pipeline

After the read barrier improvement, it became clear that the problem was not one bad if.

The whole request pipeline needed attention.

A request goes through many stages: TCP handler, session state, request queue, batching, forwarding to leader if the current node is a follower, Raft preprocessing, log append, follower replication, commit, applying the state machine, response routing, and finally the TCP handler again.

In the ideal world, this is a pipeline. While one batch is being fsynced, another can be prepared. While followers are processing one append, the leader can prepare the next. While one stage waits on disk, another can do useful work.

In practice, the pipeline was often underfilled.

Batching existed, and the code was not obviously careless. But in many realistic latency-bound scenarios, the batching component simply did not receive enough work at the right moment. It formed small batches. Small batches meant more Raft rounds, more fsyncs, more waits, and worse latency.

I spent a lot of time trying to tune batching and backpressure. Most of those experiments produced only small improvements. The problem was not just the batch size setting. The problem was the interaction between queues, barriers, response delivery, Raft, and fsync.

This led to a larger experimental PR, #102419.

That PR tried to refactor Keeper request processing around a per-session lifecycle: strict FIFO ordering per session, sharded queues, fewer global synchronization points, direct response delivery, and more explicit request states. It touched many parts of the path from receiving a request to returning a response.

The benchmark result was much better:

This was a serious improvement.

But even after that, Keeper still did not fully catch ZooKeeper in my latency-bound test. The remaining bottleneck seemed to be deeper than Keeper’s own dispatcher.

It was in NuRaft.

The fsync clue

One of the most useful ways to think about the latency was through fsync.

A coordination system that confirms writes durably must eventually pay the cost of making data durable on disk. If fsync takes, say, 8–10 ms, then write latency is strongly tied to that number.In a simple model, a request should wait somewhere between one and two fsyncs.

Best case: it arrives just before a batch is flushed and waits for that fsync.

Worst normal case: it arrives just after a previous fsync starts, waits for that one to finish, gets into the next batch, and waits for another fsync.

So the expected latency is roughly one to two fsyncs plus network and processing overhead.But the observed latency was often more like three or four fsyncs.That meant we were not merely paying unavoidable durability cost. We were adding extra serialized waits between the unavoidable waits.

The pipeline was stopping in places where it should have continued.

NuRaft and follower forwarding

One issue involved requests sent to followers.

A client can connect to any Keeper node. If it connects to a follower and sends writes, those writes need to be forwarded to the leader. But the ordering of requests within a session must be preserved.

That creates a real constraint. You cannot simply send requests from the same session over many independent connections and hope for the best. They may be reordered, and session ordering is critical for ZooKeeper-compatible behavior.NuRaft had a way to do more parallel sending, but it did not preserve the ordering Keeper needed. The safe path preserved ordering, but created a bottleneck.

Another issue was leader-to-follower append processing.

When the leader sends log entries to followers, you want the follower pipeline to stay full. If a follower is doing fsync, the leader should still be able to send or queue more work so the follower can continue immediately after the current sync finishes.

But in the problematic path, the follower could effectively block while waiting for durability, and the leader would not continue filling the pipeline for that follower. Again, the system was not CPU-bound. It was waiting.

This also explained why some streaming-mode experiments behaved strangely. Enabling streaming could reduce latency on followers, but it could also destroy batching and cause many tiny fsyncs, hurting throughput badly. Removing one kind of backpressure exposed another problem: too many small tasks.

At this point I did not really want to dive deep into NuRaft correctness myself. The code is non-trivial, and this is exactly the kind of place where a small mistake can break consensus. Also, the time I had planned for this investigation was already running out.

Fortunately, another thread of work appeared at exactly the right time.

Michael Kolupaev’s PR

A few days before my larger PR, Michael Kolupaev opened PR #101757, “KeeperDispatcher overhaul.”

It was clearly working in the same area. His PR was large, ambitious, and in some places more elegant or more aggressive than my approach. It introduced a new dispatcher design with better batching, forwarding, pipelining, ordering, flow control, and read execution.

At first, in my latency-bound benchmark, his branch did not show a major improvement over the simpler per-session barrier work. That was confusing until we realized we were optimizing different scenarios.

Michael was mostly looking at throughput. I was mostly looking at latency.That difference matters a lot.

If you run with a very deep pipeline, you can get good throughput while hiding latency. But some ClickHouse workloads are latency-bound: requests are not infinitely queued, CPU is not saturated, and every unnecessary wait matters.

We discussed the findings with Mikhail. The agreement was that Michael would continue the main work, incorporate the useful findings from my investigation, and add a latency-bound benchmark scenario so that the optimization would not be guided only by throughput-oriented tests.

That turned out to be the right path.

Michael quickly confirmed and refined the NuRaft issues. One of the important fixes was in NuRaft itself: allow the follower to continue receiving and processing more append_entries messages while waiting for fsync, and send responses when the corresponding entries become durable. That avoids forcing every append message to map too directly to a separate fsync and makes streaming mode useful instead of destructive.

After those changes, the picture changed dramatically.

In my latency-bound test, Keeper finally caught ZooKeeper!

In throughput benchmarks, Keeper now confidently beats ZooKeeper!

This is the point where the “Keeper isn’t a valid choice for production workloads” story ended.

What landed in 26.6

The main merged ClickHouse PR was #101757. The changelog describes it as various Keeper changes that make it around 2x faster overall: better batching, pipelining messages to the leader, and pipelining log appends.

That is a short changelog entry for a fairly deep change.

Under the hood, this was not “make one function faster.” It was about keeping the pipeline full without breaking ordering guarantees:

• preserve request order inside a client session;
• allow requests from different sessions to interleave more freely;
• batch where batching helps;
• pipeline messages to the leader;
• pipeline log appends;
• avoid unnecessary work in the commit path;
• improve flow control so queues do not explode;
• make reads local where they safely can be local;
• avoid global stalls caused by unrelated sessions.

There were also related improvements around the same time: running consecutive reads in parallel, reducing profiled lock overhead, and the NuRaft follower append fix.

Together, these changes moved Keeper from “often slower than ZooKeeper in the workloads we care about” to “comparable in latency-bound tests and faster in throughput tests.”

That is a big shift.

What is still moving

The 26.6 work did not close the book on Keeper performance. It changed the baseline.

Once the request pipeline became much healthier, the next bottlenecks moved to lower layers: storage layout, memory usage, snapshot loading, changelog reads, startup time, and small sources of unnecessary CPU usage or lock contention.

This is where ongoing work by both Michael Kolupaev and Antonio Andelic becomes important.

Michael has been working on the next generation of Keeper storage. The old experimental RocksDB backend was removed in #108000, with the note that a better on-disk storage was coming. The new direction is a specialized Keeper storage model, closer to an LSM-tree but designed around Keeper’s own data: znodes, paths, metadata, snapshots, and the access patterns of a coordination service. The early work is visible in #107261 and the follow-up refactoring in #108583.

Antonio has been working on the surrounding durability and recovery paths: snapshot locking, parallel snapshot loading, changelog read-ahead, startup reads, and reducing contention around the places where Keeper has to touch disk. One good example is #107595, which avoids holding the snapshot lock while doing slow snapshot serialization and disk I/O. Another is #108473, which works on changelog read-ahead for follower catch-up and commit reads.

These are not cosmetic changes. They are the next layer of the same performance story.

A coordination service spends a lot of time doing things that look boring from the outside: writing logs, reading logs, loading snapshots, transferring state to followers, rebuilding in-memory structures after restart, and keeping enough metadata around to answer requests quickly. If any of those paths holds the wrong lock, reads the wrong thing twice, or serializes work that could be pipelined, the result eventually shows up as slow startup, slow catch-up, latency spikes, or wasted memory.

So the next chapter is not another “replace ZooKeeper” story. It is making Keeper cheaper to run, faster to recover, and harder to disturb under real production conditions.

Stay tuned; it should get even better.

And personally, kudos to the ClickHouse Keeper developers. A lot of this work is deep, difficult, and not very glamorous. Users usually notice coordination systems only when they are slow or broken. Making one faster, safer, and easier to operate is exactly the kind of engineering that deserves more credit than it usually gets.

So, should you migrate to Keeper now?

For many users, ClickHouse Keeper has already been the production default for a long time. So the message is not “Keeper is finally production-ready.” That would be wrong and unfair to all the people already running it successfully.

The more precise message is:

If you wanted to move from ZooKeeper to Keeper, but performance was the reason you could not proceed earlier, it is time to try again.

Keeper already solved many of the old ZooKeeper problems: JVM dependency, operational mismatch with ClickHouse, practical zxid overflow, packet-size limitations, compressed logs and snapshots, checksums, ClickHouse-specific protocol extensions, and better integration with the ClickHouse ecosystem.

The missing piece for some users was performance.

With 26.6, the missing enamel piece was found. For the class of users who were waiting for Keeper performance to catch up, the moment has arrived.

If your old staging test showed Keeper losing badly to ZooKeeper, rerun it. Use your actual workload. Test both latency and throughput. Watch fsync. Test leader and follower paths. Do not rely only on synthetic benchmarks with huge pipeline depth if your production workload is latency-sensitive.

But do test it.

Try ClickHouse 26.6, bring your benchmark results, and report what still does not work well. Keeper is no longer just “ZooKeeper compatibility in C++.” It is becoming the coordination layer ClickHouse can actually optimize for itself.