Blog
View as Markdown
Open in Claude
Open in ChatGPT

The S3Queue Keeper Case: Lag, Pressure, and the Fix That Held

ByDiego Nieto

Introduction: Why This Case Matters

ClickHouse® ingestion lag can have many causes: overloaded consumers, object storage delays, replication bottlenecks, resource contention, inefficient transformations, or simply more incoming data than the system can process. In this case, those familiar explanations turned out to be only symptoms.

The real pressure was in Keeper/ZooKeeper.

S3Queue is one of ClickHouse’s lesser-known table engines, but it behaves similarly to streaming engines such as Kafka or RabbitMQ. It is designed to continuously ingest files from S3-compatible object storage. It works like a queue over files in a bucket: ClickHouse discovers new objects matching a path or pattern, reads them in a configured format such as JSON, CSV, Parquet, or TSV, and tracks which files have already been processed so they are not consumed repeatedly.

S3Queue is commonly used with a materialized view that moves incoming data from the S3Queue table into a regular MergeTree table. That makes it useful for streaming-style ingestion from object storage. But there is an important operational detail: S3Queue is not only reading files — it is also coordinating work between cluster replicas. It uses Keeper/ZooKeeper to track file ownership, processed state, retries, failures, locks, and commits. At enough scale, that metadata traffic can compete with other cluster-critical coordination work, including ReplicatedMergeTree metadata.

That was the core of this incident. The queue lag was visible first, but Keeper pressure was driving the pattern. This article walks through how we traced recurring S3Queue lag back to Keeper pressure, why tuning alone was not enough, and how adding an auxiliary Keeper gave S3Queue its own coordination path.

From a reader perspective, the practical takeaways are:

  • When S3Queue lag repeats or recovers only temporarily, check Keeper pressure early.
  • Compare lag windows with ZooKeeperWaitMicroseconds and Keeper in-flight request metrics.
  • Remember that unordered mode, tracked_files_limit, replicas, and many active queues can increase coordination cost.
  • Tuning can reduce pressure, but it does not isolate S3Queue from other Keeper-backed workloads.
  • Auxiliary Keeper can be the right fix when S3Queue coordination competes with the default Keeper plane.

The Night the Lag Came Back

The incident started with a familiar alert: replication and ingestion lag were climbing on a large customer cluster. At first, the system looked like it might recover on its own. Lag rose, then partially dropped. We checked the usual suspects: ingest health, cluster health, and whether the pattern was isolated or repeating. Then the alert came back with the same shape and timing — no longer a transient spike but a recurring pressure pattern. The goal became simple: find where the queue was waiting and why it kept happening.

At Altinity Engineering, we do not stop at triage. We run the full arc: detection, root-cause analysis, issue creation, implementation, validation with peer review, and deployment in customer environments.

Separate Ingestion Lag From Coordination Pressure

When ingestion lag rises, it is natural to check compute, storage, consumers, materialized views, and bad input files first. Those checks are still important, but they should not be the end of the investigation. In this case, CPU, storage I/O, replica health, materialized views, and bad file batches did not explain the repeating lag pattern. For example, a quick check on replica queue depth and S3Queue consumer status returned nothing alarming:

-- Replica queue depth 
SELECT database, table, queue_size, inserts_in_queue 
FROM system.replicas 
WHERE queue_size > 0
ORDER BY queue_size DESC 
LIMIT 5; 
 
-- S3Queue consumer status (recent hour) 
SELECT database, table, status, count() AS files 
FROM system.s3queue_log 
WHERE event_time >= now() - INTERVAL 1 HOUR 
GROUP BY database, table, status;

Both looked unremarkable. The stronger signal was in Keeper/ZooKeeper metrics. The key question became: is the queue slow because it cannot read and process files, or because it is waiting on coordination? For S3Queue, that distinction matters. The engine reads from object storage, but it also uses Keeper/ZooKeeper to coordinate ownership, processed state, failures, locks, retries, and commits. When that coordination layer slows down, replicas can spend less time ingesting and more time waiting.

The useful signals were:

  • ZooKeeperWaitMicroseconds rose during lag windows
  • KeeperOutstandingRequests / ZooKeeperRequest stayed elevated
  • S3Queue backlog increased in the same time windows
  • Retry-related noise increased around the same periods
  • Some Keeper-backed system table queries became slow or stuck.

A useful first diagnostic query is:

SELECT 
    toStartOfMinute(event_time) AS minute, 
    sum(ProfileEvent_ObjectStorageQueueListedFiles) AS listed_files, 
    sum(ProfileEvent_ObjectStorageQueueFilteredFiles) AS filtered_files, 
    sum(ProfileEvent_ObjectStorageQueueReadFiles) AS read_files, 
    sum(ProfileEvent_ObjectStorageQueueProcessedFiles) AS processed_files, 
    sum(ProfileEvent_ObjectStorageQueueFailedFiles) AS failed_files, 
    sum(ProfileEvent_ObjectStorageQueueTrySetProcessingRequests) AS set_processing_requests, 
    sum(ProfileEvent_ObjectStorageQueueTrySetProcessingSucceeded) AS set_processing_succeeded, 
    sum(ProfileEvent_ObjectStorageQueueTrySetProcessingFailed) AS set_processing_failed, 
    sum(ProfileEvent_ObjectStorageQueueFailedToBatchSetProcessing) AS batch_set_processing_failed, 
    sum(ProfileEvent_ObjectStorageQueueCommitRequests) AS commit_requests, 
    sum(ProfileEvent_ObjectStorageQueueSuccessfulCommits) AS successful_commits, 
    sum(ProfileEvent_ObjectStorageQueueUnsuccessfulCommits) AS unsuccessful_commits, 
    max(CurrentMetric_ZooKeeperRequest) AS max_zookeeper_inflight_requests, 
    sum(ProfileEvent_ZooKeeperWaitMicroseconds) AS zookeeper_wait_microseconds, 
    sum(ProfileEvent_ZooKeeperTransactions) AS zookeeper_transactions, 
    sum(ProfileEvent_ZooKeeperMulti) AS zookeeper_multi, 
    sum(ProfileEvent_ZooKeeperMultiRead) AS zookeeper_multi_read, 
    sum(ProfileEvent_ZooKeeperMultiWrite) AS zookeeper_multi_write, 
    max(CurrentMetric_KeeperOutstandingRequests) AS max_keeper_inflight_requests, 
    sum(ProfileEvent_KeeperCommitWaitElapsedMicroseconds) AS keeper_commit_wait_microseconds, 
    sum(ProfileEvent_KeeperCommits) AS keeper_commits, 
    sum(ProfileEvent_KeeperCommitsFailed) AS keeper_commits_failed, 
    sum(ProfileEvent_KeeperTotalElapsedMicroseconds) AS keeper_total_elapsed_microseconds, 
    sum(ProfileEvent_KeeperProcessElapsedMicroseconds) AS keeper_process_elapsed_microseconds 
FROM system.metric_log 
WHERE event_time >= now() - INTERVAL 1 HOUR 
GROUP BY minute 
ORDER BY minute DESC 
LIMIT 60;

The query returns one row per minute with two groups of columns. The first group covers S3Queue activity: how many files were listed, filtered, read, processed, failed, and whether set-processing and commit operations were succeeding. The second group covers Keeper-side cost: how long requests waited, how many were in-flight, how long commits took.

The pattern that points to Keeper pressure is a correlation between the two groups across the same time windows:

  • zookeeper_wait_microseconds rising sharply while processed_files drops
  • max_zookeeper_inflight_requests or max_keeper_inflight_requests staying elevated for multiple consecutive minutes
  • set_processing_failed or unsuccessful_commits increasing — S3Queue retrying because Keeper operations timed out or were rejected
  • keeper_commit_wait_microseconds growing disproportionately relative to keeper_process_elapsed_microseconds — commits spending most of their time waiting, not executing

If those columns move together during a lag window, the bottleneck is coordination, not ingestion capacity. The takeaway from this step was simple: lag was the symptom, but Keeper pressure was the pattern.

Identify What Is Amplifying Keeper Work

After the metrics pointed toward coordination, the next question was why S3Queue was generating enough Keeper pressure to matter. Several factors compounded in this incident:

  • Metadata operation volume. Every file S3Queue processes requires a sequence of Keeper calls — exists, get, getChildren, create, set, remove. These are expected: they are how the engine discovers work, claims files, records progress, handles failures, and commits state. The problem is that across many queues, replicas, and pipelines, those calls accumulate fast.
  • Shared coordination plane. S3Queue metadata shared the same Keeper ensemble as ReplicatedMergeTree metadata and background replication tasks — meaning S3Queue coordination traffic was not isolated and competed directly with other cluster-critical operations.
  • Durable write pressure. Metadata writes add transaction-log, fsync, and snapshot cost. When metadata churn rises, Keeper latency follows.
  • Large tracked-file state. In unordered mode, tracked_files_limit controls how many processed files Keeper tracks to prevent duplicates. The default is 1,000 per table. This customer raised it to 10,000 — sensible for their workload, but it increased the znode count and made the S3Queue ZooKeeper subtree heavier to traverse.
  • Multiple pipelines and replicas. Each additional S3Queue table, replica, or pipeline adds coordination traffic. At high concurrency the pressure is cumulative, not isolated to any single table.
  • Stale metadata from unordered mode. Switching to ordered mode reduces the ongoing metadata footprint, but old znodes do not disappear automatically — they keep the ZooKeeper subtree heavy until explicitly cleaned up. This is a known issue addressed in 26.2 (ClickHouse#94412).

The pattern was clear: S3Queue lag moved with Keeper-side pressure windows, not with any single broken consumer, bad batch, or CPU spike.

Choose The Fix That Matches the Failure Mode

By this point, two things were clear:

  1. S3Queue lag was strongly correlated with Keeper pressure.
  2. The pressure came from normal S3Queue coordination activity, not a broken component.

That changed the type of fix we needed. The goal was not to repair something, but to reduce or isolate contention.

Option 1: Tune S3Queue

Adjusting queue settings or reducing metadata churn can lower Keeper load and help during an incident. However, S3Queue metadata still shares the same Keeper infrastructure as replication and other coordination workloads. Tuning reduces pressure but does not isolate it.

Option 2: Reduce ingestion concurrency

Lowering concurrency reduces the number of Keeper operations generated per second. This can stabilize the cluster quickly, but at the cost of throughput. The contention remains and can return as ingestion volume grows.

Option 3: Use an auxiliary Keeper

This was the long-term fix: instead of sharing the default Keeper infrastructure, S3Queue metadata could be moved to an auxiliary Keeper ensemble. An S3queue table can specify an auxiliary Keeper directly:

keeper_path = 'aux_name:/path'

The prefix identifies which Keeper service owns the metadata. This does not eliminate coordination work, but it moves that work to a separate coordination plane so it no longer competes directly with replication metadata and other cluster-critical workloads.

Why an auxiliary Keeper was the right fix

The problem was not ingestion volume alone — it was contention. Tuning and throttling could both reduce pressure, but neither could isolate the workload. Auxiliary Keeper could. By moving S3Queue metadata to a separate Keeper ensemble, it prevented direct competition with replication and other critical metadata operations while preserving existing S3Queue behavior.

The remaining challenge was implementation: ensuring every S3Queue metadata operation consistently used the selected Keeper and never silently fell back to the default one.

Make Keeper Identity Follow the Whole S3Queue Path

Auxiliary Keeper support was not a new concept in ClickHouse. ReplicatedMergeTree already supports storing its metadata in a separate Keeper ensemble, and the underlying infrastructure for connecting to auxiliary Keeper services was already in place. What S3Queue lacked was the plumbing to carry that Keeper selection through its own metadata lifecycle.

The existing code had a quiet assumption: S3Queue metadata belongs to the default Keeper. Every place that opened a Keeper connection, cached a metadata object, registered a factory key, or reconstructed a table definition was defaulting to the primary ensemble. Supporting auxiliary Keeper meant making the Keeper name travel alongside the Keeper path at every one of those points. A path like /clickhouse/s3queue/foo says where the metadata is stored. Without the name, the code cannot tell which Keeper service owns it.

The implementation touched six areas:

  • Initialization — resolve the Keeper name and path together so the table has a complete metadata address before any operations start. (source)
  • Metadata objects — store the Keeper name inside the metadata object so that discovery, claiming, processing, commits, and cleanup all route to the same ensemble. (source)
  • Runtime calls — obtain Keeper clients by name instead of defaulting to the primary. Every exists, get, getChildren, create, set, and remove call goes to the selected Keeper. (source)
  • Retry loops — reacquire the Keeper client inside each retry body so reconnects after session failures still use the correct ensemble. (source)
  • Metadata factory keys — include the Keeper name in the in-memory registry key so two tables sharing the same path string but pointing to different Keeper ensembles do not collide. (source)
  • DDL output — preserve the auxiliary prefix in SHOW CREATE TABLE so operators can always see where the metadata actually lives.

After the change, the full keeper_path setting is reflected in the table definition:

SHOW CREATE TABLE my_s3_queue; 
CREATE TABLE default.my_s3_queue 
( 
    `event_time` DateTime, 
    `user_id`    UInt64, 
    `event_type` String, 
    `payload`    String 
) 
ENGINE = S3Queue( 
    'https://my-bucket.s3.amazonaws.com/events/*.json', 
    'JSONEachRow' 
) 
SETTINGS 
    mode = 'unordered', 
    keeper_path = 'aux_keeper:/clickhouse/s3queue/my_s3_queue', 
    tracked_files_limit = 10000;

The full implementation is in the upstream PR S3Queue auxiliary ZooKeeper support (ClickHouse/ClickHouse#95203).

Review And Release

The implementation went through the usual development cycle: coding, testing, peer review, and validation. The review focused on the areas most likely to cause problems:

  • Hidden assumptions that defaulted to the primary Keeper
  • Incomplete propagation of Keeper identity through S3Queue metadata paths
  • Metadata keys that used only the path and ignored the Keeper name
  • Retry paths that could reuse stale Keeper clients
  • DDL output that could hide the auxiliary Keeper prefix.

The change was merged upstream first:

References:

Thanks to the reviewers and contributors who helped refine and validate the implementation:

What We Learned and Closing the Case

This case started with a recurring lag alert and ended with an upstream code change. Along the way, it reinforced a simple lesson: coordination has a cost. S3Queue was doing exactly what it was designed to do. It continuously ingests files from object storage, tracks progress, avoids duplicates, and coordinates work across replicas. None of those behaviors were bugs. The challenge was the amount of coordination required at scale. At small scale, that cost is easy to miss. At larger scale, especially with many S3Queue tables, high churn, unordered mode, large tracked_files_limit, and multiple replicas, Keeper metadata can become part of the ingestion bottleneck.

The failure may not look like Keeper at first. It may look like:

  • Ingestion lag
  • Slow materialized-view progress
  • Replication delay
  • Retries
  • Hanging system-table queries
  • Intermittent recovery followed by another spike.

In this case, the queue was waiting on coordination. The important result was not a workaround or a collection of tuning recommendations that would need to be rediscovered during the next incident. The durable fix was auxiliary Keeper support for S3Queue metadata. It isolated the S3Queue coordination workload from the default Keeper plane, preserved the default behavior for existing users, and made the selected Keeper identity explicit through the full metadata path. The fix also gave operators a cleaner architecture for high-churn S3Queue workloads:

  • Keep replication metadata on the default Keeper
  • Move S3Queue coordination metadata to auxiliary Keeper when the workload requires isolation
  • Preserve Keeper identity from initialization to runtime to DDL visibility.

That is the production engineering arc we want: find the pressure, identify the cause, validate the solution, and ship it so customers can use it. And like many good detective stories, the ending was simpler than it first appeared. We chased ingestion lag, examined consumers, reviewed storage, and questioned every usual suspect. In the end, the culprit was hiding in plain sight.

The butler was ZooKeeper.

Join our Slack

ClickHouse® is a registered trademark of ClickHouse, Inc.; Altinity is not affiliated with or associated with ClickHouse, Inc.

Table of Contents:

Related:

Leave a Reply

Your email address will not be published. Required fields are marked *