Delivering Production ClickHouse® Apps on the Altinity Kubernetes Operator

Recorded: March 24 @ 08:00 am PDT
Presenters: Robert Hodges and Alexander Zaitsev

In this webinar, Altinity CEO Robert Hodes and CTO Alexander Zaitsev walk through the Altinity Kubernetes Operator for ClickHouse®, the open-source operator that powers Altinity.Cloud and hundreds of production deployments worldwide. The session opens with a live demo of spinning up a replicated ClickHouse® cluster in under three minutes using a local Kubernetes environment. From there, the speakers dive deep into the architecture behind the operator’s custom resources (CHI and CHK), key production deployment considerations, and advanced platform-building features.

Topics covered include: managed Kubernetes options (EKS, GKE, AKS), cluster autoscaling, storage class configuration, the volume reclaim policy, backup via Altinity Backup for ClickHouse®, ClickHouse Installation Templates (CHIT), service templates, zero-downtime rescaling and rolling upgrades, Prometheus-based monitoring with Grafana dashboards, and automation via Terraform and Helm. The session closes with a roadmap preview covering post-start and pre-stop hooks, tighter CHI and CHK integration, persistent volume snapshot-based replica provisioning, and an extensible plugin architecture. A Q&A segment covers topics including NVMe support, KEDA autoscaling, the relationship between the Altinity Operator and ClickHouse Inc.’s newer operator, and Project Antalia’s swarm clusters.

Key Moments (Timestamps)

Key moments generated with AI assistance.

• 00:00 – Welcome and housekeeping
• 01:42 – Introduction to Altinity and its history with ClickHouse on Kubernetes
• 03:18 – Live demo: spinning up a replicated ClickHouse cluster in under three minutes
• 07:43 – ClickHouse and Kubernetes architecture overview
• 09:03 – How the operator pattern works: custom resources and reconciliation
• 13:27 – Installing the Altinity Kubernetes Operator and its components
• 15:23 – CHI resource deep-dive: cluster layout, pod templates, and storage
• 20:07 – Production deployment tips: managed Kubernetes, autoscaling, and node affinity
• 24:54 – Storage classes, volume expansion, and the reclaimPolicy safety net
• 26:29 – Backup with Altinity Backup for ClickHouse and the sidecar pattern
• 28:26 – Advanced features: service templates and ClickHouse Installation Templates
• 35:17 – Rescaling: adding and removing replicas and shards with no downtime
• 39:35 – Upgrades, configuration management, and rolling restarts
• 43:51 – Maintenance attributes: stop, restart, suspend, and troubleshoot mode
• 47:25 – Monitoring: Prometheus metrics exporter and Grafana dashboards
• 50:06 – Deployment automation: Terraform, Helm, and Argo CD
• 54:14 – Altinity Operator vs. the ClickHouse Inc. operator
• 59:15 – 2025–2026 roadmap highlights
• 1:01:54 – Q&A: NVMe, KEDA, Project Antalia, and community contributions

Webinar Transcript

[00:00] – Welcome and Housekeeping

Robert: Welcome, everybody. We are going to deliver a webinar on delivering production ClickHouse® apps on the Altinity Kubernetes Operator. I hope the whole webinar goes better than that first line. Welcome again, and thank you so much for joining us today.

I am going to be working with my colleague Alexander Zaitsev, who is the Altinity CTO. I am Robert Hodes, Altinity CEO. Before we start the webinar, I would like to kick off with a couple of housekeeping items so that you will not have to take frantic notes.

First, this webinar is being recorded right now. We will send you a copy of the recording, or a link to it, along with a link to the slides right after the webinar finishes. You will get it sometime later today. No need to worry. As long as you have signed up, the information will be forwarded to your email.

Second, we do have time for questions, and in fact we encourage them. You can enter them in the Q&A box. Since there are two of us, we may be able to answer them in the background, or just answer them as part of the presentation. We do have a lot to cover, so we will probably reserve some of the questions for the end of the presentation. There is also a chat here. Some people like to put questions there. It does not matter to us. Type them in anywhere you want and we will answer them.

With that, let us dive in.

[01:42] – Introduction to Altinity

Robert: Just a little background information on Altinity. We are a vendor for ClickHouse®, the best analytic database on the planet. We have been around since 2017 and we have a lot of experience running ClickHouse on Kubernetes.

We run Altinity.Cloud, which has been up and running since 2020, and at this point we have customers running on at least five cloud platforms. We also have a large support business, and we support a lot of people who run on Kubernetes as well.

This relationship with Kubernetes started relatively early on. We began to get into Kubernetes in a big way starting at the end of 2018. That led us to write the Altinity Kubernetes Operator for ClickHouse®, which was first released in early 2020 and has been the foundation of many customer deployments as well as our own Altinity.Cloud.

This talk is going to introduce you to the operator and show you some of the features that enable you not just to do development, but also to build full-scale data platforms, in some cases consisting of hundreds of nodes. With that, I would like to do a quick introduction and demo. For that, I am going to turn this over to my colleague Alexander Zaitsev.

[03:18] – Live Demo: Spinning Up a Replicated Cluster

Alexander: So we start with the mini deployment that I have on my laptop. The beauty of Kubernetes and the operator is that you can try production configurations on your laptop before deploying them to real production.

Let me create a namespace first. This is a test namespace. Then I will deploy the ClickHouse operator using the simplest deployment mechanism. We can see that a lot of resources have been created. We will probably discuss those a bit later. Let me double-check that the operator is up and running. Good.

Now we can go to ClickHouse itself. I have pre-created a simple manifest that contains a ClickHouseKeeper installation. You can see it here, and there is also a ClickHouseInstallation. So we create two resources. The ClickHouseInstallation in turn is referring to ClickHouseKeeper. You can see it in the keeper section. There is a host, keeper-my-chk, which is a service name for ClickHouseKeeper.

Let us create those resources. I just run kubectl create in the test namespace. Two resources were created, and we can track the progress. The operator reads those resources in the background, reads the definitions of how many shards and replicas need to be created, what needs to be running, and so on. You can see that it started the first ClickHouse pod and now it is starting CHK, which is ClickHouseKeeper. The second ClickHouse pod is starting, so it is all more or less in parallel.

Now everything is running. Let me double-check. It is still in progress, but we can already connect to ClickHouse using kubectl exec. I am now in the ClickHouse pod on one of the replicas. I should mention that we started a replicated cluster, which is why we have a Keeper. We can now create a table, and I will use a schema from an S3 bucket. We have some Parquet data. It is now connecting to S3 to get the schema.

The schema has been created on two nodes. It is still an empty table, and I can immediately load a small portion of the data. I will just read 100 rows to make it faster, because the object storage is somewhere far away from this mini installation.

You can see it reads quite a few rows. Let me check one row just to show you. We have data in our ClickHouse deployed by the Kubernetes operator. Exactly 100 rows, and those rows are on two nodes because we have replication. This is as simple as it gets. It took me 3 minutes to spin up a cluster with replication. If you have bare-bones ClickHouse it will take you quite a lot of time to figure out how to configure it properly. Now let us go back to the presentation.

[07:43] – ClickHouse and Kubernetes Architecture Overview

Robert: Let us look at the architecture of ClickHouse, because we are going to represent this in Kubernetes. In the examples we are using here, we have ClickHouse, which as most people on this call know is an analytic server with a shared-nothing architecture and attached storage connected by a network.

Here is an example where we have two servers set up as replicas, and we have a ClickHouse Keeper, which is used to maintain consensus between those replicas so they can figure out which one has which parts and get notified when new parts arrive.

Alexander: So, what we are going to do is use an operator to control these clusters. An operator is a special programming paradigm implemented by Kubernetes. It allows you to create what are called custom resources. Resources are the way we model container-based applications in Kubernetes. What the operator does is allow you to define any kind of resource and, equally important, define the processing on that resource.

The way it works, using an example from the Altinity Operator, is that you feed in this custom resource using kubectl, which is the standard command to send it to Kubernetes. Kubernetes reads it, looks up the resource type, and performs what is called reconciliation. It looks at the resource, which contains a definition of what you would like to have representing your ClickHouse server, and looks at what resources are already allocated in Kubernetes. Anything not there will be created. Anything different will be adjusted to match the new version of the custom resource you provided.

This is a very powerful mechanism, and it is particularly good for databases because databases do not just have a layout; they also have complicated procedures to move between states.

[09:03] – Custom Resources: CHI, CHK, and What Gets Created

Robert: Here is an example of the installation you can use to set up a demo. We have two types of resources. The first is a ClickHouseKeeper installation. Since Keeper is used to maintain consensus, we want to get that up first, because when the ClickHouse servers arrive they are going to need it. It is relatively simple. It allocates a small amount of storage and does not even specify what pod image to use, because it will default to the latest build.

This is a very simple ClickHouse cluster resource. It does not even include the image; it is going to pull the latest ClickHouse build. It is going to have two replicas and 10 GB of storage for development purposes. When you are doing this on a laptop or development host, this is all you need to get both Keeper and ClickHouse set up and talking to each other.

Let us look at what is actually happening underneath. We have these different types of resources. We sometimes call the ClickHouseInstallation a CHI resource and the ClickHouseKeeperInstallation a CHK. We can use those abbreviations in Kubernetes. In each case, when they lay out resources in Kubernetes to create a cluster, they first create a resource called a stateful set. This is a type of resource in Kubernetes that knows how to manage storage and comes up with a predetermined name. Both qualities are very useful for databases: we need to be able to find them, and they need to be able to find their storage if they are restored or rescheduled to another host.

The storage itself is represented as a persistent volume claim, which is a request to Kubernetes to allocate storage. When that storage is allocated, it becomes represented as a persistent volume. That is a real patch of storage somewhere in the cloud, or on local NVMe SSD, that you can write to. In the end these are just processes running on hosts, but the Kubernetes modeling with these resources allows us to tell Kubernetes what we would like to see, and Kubernetes takes care of representing it.

[13:27] – Installing the Altinity Operator

Robert: Let us dig into what is going on behind the scenes, and specifically some of the things we can do as we set up these CHI and CHK resources.

First, installing the Altinity Operator. This is a very simple operation that on a typical network connection will take 10 seconds or less. The Altinity Operator is a container like everything else that does work in Kubernetes. If you just type the install command, press Enter, and are pointing to a Kubernetes cluster, it will pull down the piece of YAML that contains the full definition of the operator and all the things it needs to operate.

Those include a number of things. For example, config maps, which supply parameters and other types of information the operator needs to function. We also set up a service account with appropriate roles so we have the ability to manipulate things within the namespace or namespaces we are operating. A really important part of operators is what is called a custom resource definition. The Altinity Operator has four: the CHK resource, the CHI resource, and a couple of different kinds of templates that Alexander will talk about. Finally, there is the operator itself as well as an additional sidecar service that exports metrics. All of this comes down and gets pushed into the Kubernetes system namespace.

[15:23] – CHI Resource Deep-Dive: Cluster Layout, Pod Templates, and Storage

Robert: Here we are going to dig in a little to the CHI resource and show you some of the key elements you would want to know even in a development environment. These cover talking to Zookeeper, the layout of your cluster, how you define the container you want (your pod template), and how you define storage (your volume claim template).

So here we have cluster layouts and locating Keeper. This is usually the first section in any of these. You have the ClickHouseInstallation resource and then some very straightforward information: number of replicas, number of shards. This is all you need to set up replicas. The ClickHouse operator is pretty smart about them. If you put schema in them and use ON CLUSTER commands, the operator will automatically transfer your schema to new replicas or even new shards when you add them. Then there is the Zookeeper location shown below.

There is an additional section worth knowing about. If you run ClickHouse directly on bare metal or in the cloud, you know you often have to set up configuration files or change the number of maximum concurrent queries. These can be added to the definition and will be automatically set up in the configuration when the container lands on a host.

Pod definitions. In the development form we are showing here, this is very simple. You have a spec section. You list your containers. You can actually have multiple containers there; we are just going to use one for now. Give it a name and specify what image you want. The Altinity Kubernetes Operator for ClickHouse® runs with basically any ClickHouse build you can find. We do our own Altinity Stable® Builds for ClickHouse®. There are of course ClickHouse official builds as well as the Altinity project builds. All you have to do is name the image and we will get it for you.

Moreover, the Altinity Operator is very agnostic about server versions. I think the oldest version we have running in production right now is 21.11. They should probably upgrade, but basically whatever version you are comfortable with, you can just put it into the operator and it should run just fine.

Storage definitions. This is again very straightforward. The volume claim template has two settings that are useful to know. First, how much storage you want. If you do not include this section in the definition, you will be running on ephemeral storage, which means it will evaporate if your container storage gets deleted. There is another important item here: the reclaim policy. What that says is: if we drop the cluster, do not delete the storage allocations in Kubernetes. Specifically, the persistent volume claim will be preserved. What that does is: if you accidentally delete something, like a shard or the whole cluster, the PVCs will stay up. When you realize the error of your ways, you can just reinstall and it will find the storage again and keep running. It is a really important safety lock to keep you from getting into trouble.

We are not going to talk about the ClickHouseKeeper installation very much other than to note that it uses the same ClickHouse binary as the ClickHouse servers. As a result the CHK resource is very similar and you have things like number of keepers, a pod template, and a volume claim template. It is pretty straightforward.

[20:07] – Production Deployment Tips

Robert: All of the stuff I have shown you so far is all the knowledge you need to begin developing and playing with ClickHouse on your laptop or a development machine. Now, as you think about actually turning this into a production application, there are a number of things you need to do.

First, a really important thing is that most people do not want to run Kubernetes themselves. If you are running in the cloud, this is actually one of the best bargains around. Virtually every cloud provider, and some non-cloud providers, have their own managed Kubernetes, and you just use that. For example, EKS, GKE, and AKS are all on the big three. They cost on the order of $70 to $100 a month to run, which is practically free, and they take care of all the grunt work: keeping Kubernetes up to date, making sure security patches are delivered, making sure things are backed up, and giving you an automated way to bring clusters up and put them away. This is something you should absolutely use unless you are very deep in Kubernetes, in which case you may want to manage it yourself.

The second thing is you want to learn how to wire this up. There are two really key things you have to handle. Kubernetes will schedule things and knows when it cannot schedule something and needs to go allocate more VMs so it has additional capacity. That is done through something called cluster autoscaler, at least on Amazon. We can use Terraform to set this up, but basically this tool will configure the underlying cloud infrastructure and configure Kubernetes so that it is correctly wired with the correct authorization, so that when it needs to allocate or delete a pod, it can spin up or spin down VMs at will.

The same thing applies to storage. Storage is defined in something called a storage class. Your persistent volume claim refers to a storage class. You have a storage provisioner that reads the persistent volume claims and allocates matching amounts of storage. Somebody in the organization needs to understand how it works.

Some key things we have found which are really important for running ClickHouse in production: first, you want to allocate a separate host for each ClickHouse server. Here are the operator settings that allow you to do this. First, you specify what kind of VM image type you want. We show an M7G, which is a Graviton on Amazon. High performance, lower cost than equivalent Intel. You specify that image. Second, pod distribution: it is kind of like the Highlander principle, there can only be one. It uses anti-affinity, which is a Kubernetes setting we build on top of, to push these CHI resources onto different VMs. We do not want two ClickHouse nodes sharing the same host; they would be competing for resources, and it would also be bad if that host went down. Third, tolerations: this allows us to say that the hosts we are going to be running on have certain taints, and these tolerations say we can be scheduled on those hosts. So it allows you to have pools of resources that are only for ClickHouse servers and not for, say, ClickHouseKeeper or other applications.

[24:54] – Storage Classes, Volume Expansion, and Backup

Robert: Storage classes are actually really easy to understand and have improved considerably over the last few years. This is an example of the storage class for GP3 EBS, which is the current most advanced form of elastic block storage on Amazon. It includes encryption. It includes settings like file type, disk throughput, and IOPS. We can also allow volume expansion to be done online. These actually can be changed online.

You can go ahead and change storage on live systems. It is pretty simple to go from 50 to 100 GB by patching the volume claim template. It is a little bit trickier to scale up disk throughput, but that can also be done. We wrote a blog article on the way we do it in Altinity.Cloud. We have a special controller that handles that. Getting storage correctly wired is one of the most common ways that people upgrade systems, either to increase storage (probably the number one way systems upscale) or to change disk performance in response to new conditions.

Backup. Since we are talking about storage, you might think: what if I lose it? Backup is many-splendored. We run Altinity Backup for ClickHouse®, which lives in a project called clickhouse-backup out on our GitHub. Here is how it works. We have the ClickHouseInstallation, and then what we use is what is called a sidecar: instead of having one container running in the pod, we have two. The first is your ClickHouse server. The second is a container running the clickhouse-backup binary. Because it is running in the same pod, it is in the same process space; it can see the storage, which is necessary because it needs to be able to see the storage and play around with hard links to do backups. Then it can write to object storage. What you do is allocate a bucket and make sure that the IAM is correctly wired so that you can write to it and read from it. That is how you store your backups and restore them on demand. You can also use the internal backup commands, which do not require an extra container. As Alexander will discuss, we have roadmap activity that will improve this further over time.

[28:26] – Advanced Features: Service Templates and ClickHouse Installation Templates

Alexander: Let me introduce you to some more advanced features of the operator that allow you to build very complex production systems and data platforms.

First is templating. We already talked about storage templates and pod templates, which are standard Kubernetes abstractions. We have not invented them; they have existed for a long time. But what we added in the operator, which does not exist in Kubernetes by itself, are two things: service templates and ClickHouse Installation Templates.

Service templates help you define network configuration, and ClickHouse Installation Templates help you define anything in the spec and enable a lot of things that look like magic.

Let us start with service templates, because they are pretty simple. In the operator we support several different types of services that can be deployed. It starts with just a service template, which is a load balancer for the ClickHouse installation. If you have multiple shards and replicas, you may use this service and it will randomly connect to any of your ClickHouse nodes. There are also other types of service templates. For example, one for internal load balancer, another for external load balancer.

There is also what we call a replica service template. The name is a bit confusing. In fact, this is a service template for every ClickHouse node, and using it you may have a service that allows you to connect to a specific ClickHouse node from Kubernetes. Sometimes you do need to connect to a specific node. Finally there is a cluster service template that allows you to connect to a specific cluster. One ClickHouseInstallation can actually work as an enclosure for multiple ClickHouse clusters.

Here is an example of a service template. It is very simple. You have a name and a generateName. The generateName allows you to use macros so you do not have to hardwire the name. You can just specify that you want your service named like service- and the name of your ClickHouseInstallation. And then the port definitions, which are exposed.

If you work with some cloud provider’s Kubernetes, you will use a service template to specify cloud-provider-specific load balancer settings. For example, on AWS you will put annotations in the service template that instruct Kubernetes how to expose your service if you want to, what kind of load balancer to use, and what security settings to apply. For production operation, this is a must.

ClickHouse Installation Templates. For these we have a separate Kubernetes resource, which we call a CHIT for simplicity. Structure-wise it is identical to a ClickHouseInstallation. It has pretty much the same structure and the same objects you can define. The purpose of a ClickHouseInstallation Template is that you can define parts of your configuration. For example, you can define what kind of pod template to use and then use that as a default or as a shared configuration between multiple ClickHouseInstallations in your environment. You can also use it to override behavior. If you have a ClickHouseInstallation that you deploy with some automation and then want to inject some specific behavior by other automation, a CHIT lets you do that. We use it quite often in Altinity.Cloud. This is a very powerful feature.

Here is a very simple example. I define a ClickHouseInstallation on the left side and add a special useTemplates reference. This is an array, so I can use multiple templates. In this example I use clickhouse-version-template. This template, described on the right side, defines what kind of ClickHouse version to use for any installation that uses it. So in my ClickHouseInstallation I do not have to specify the version anymore. I will just use the one provided by the template. If a new version comes, I can change the template. I do not need to change every installation. This is really convenient.

Here is a more complex example. This is an installation template that opens up the PostgreSQL port in ClickHouse, and it has to do multiple things: open a port on the container, define a special service template for the PostgreSQL connection, and open a PostgreSQL port in ClickHouse itself. On top of that, we specify that this template applies to a single ClickHouse cluster and applies automatically using the policy: auto attribute. For that cluster, we do not even need to reference useTemplates. This is an instruction to automatically apply this template to all ClickHouseInstallations that match the selector. This is a kind of inversion of control. It allows you to not touch anything in your ClickHouseInstallation but still deploy some behavior or changes to your running system. This is a very powerful mechanism.

[35:17] – Rescaling: Adding and Removing Replicas and Shards

Alexander: If you run in production, you usually start small but then tend to grow bigger. You may add replicas, and our operator does it very smartly with no downtime and no service interruption.

First it creates new pods and storage, as you would expect. Then it creates schema on the new replicas automatically. If you have tables on your cluster, those will be automatically created on the new replicas. Finally, it waits for replication to catch up. If you have a big cluster with hundreds of terabytes of data, you cannot just start using a new replica immediately. It will be lagging behind and your queries may get incorrect results. Therefore, the operator waits for replication to catch up before including the new replica in the load balancer and making it available for queries. This is one more reason to use a load balancer: if you connect to a replica directly, you do not have this control over replication lag.

Adding a replica does not require any restarts. It can be done on a running cluster, transparently.

The same applies for sharding. If you want to add more shards, you can do the same. Just increase the count in your ClickHouseInstallation and the operator will take care of adding pods, adding storage, creating schema on the shards, and waiting for replication to catch up if you have replicas for those shards. What it does not do is automatic data redistribution. If you add new shards, those new shards are empty. This is a conscious decision going back to the beginning of ClickHouse, because the data sizes are typically big and resharding is typically very expensive. So if you add more shards, you just wait for new data to be distributed across the new shards.

Scaling down is the same. If you had three replicas and go to two, the operator will handle this change without service interruption. It removes nodes from the load balancer, removes resources, and finally it even cleans data in ZooKeeper or Keeper, because ClickHouse does not do it automatically when you remove a node.

Rescaling storage. Robert already touched on that. You can add more storage in your ClickHouseInstallation. In fact the operator has two ways to manage storage. There is the traditional way, when you define a volume claim template as part of the stateful set. If you change the volume claim template in the stateful set, you will have to reprovision the pod and recreate the stateful set, which requires downtime. This is why we implemented a different way to manage storage called the provisioner operator. In this case, the operator creates PVCs directly without using stateful set logic anymore, and because of that the operator can modify the storage directly with no downtime if the storage class allows volume expansion. This is very powerful. Unfortunately, by default the stateful set provisioner is used because it is the standard, and we cannot change the default without affecting running installations. But the two modes are totally interchangeable.

We also support multiple volumes, but you will have to do a lot of wiring in pods and in ClickHouse configuration manually, because there are many nuances that the operator does not automate. All of it is available if you know what to do in Kubernetes.

[39:35] – Upgrades, Configuration Management, and Rolling Restarts

Alexander: The next important thing that every production user will have to deal with is upgrades and configuration management: how you upgrade your ClickHouse, how you change configuration.

An image change, if you change the version, requires a restart of the pod, which is expected. If you change something in a pod template or a volume claim template, the operator will recreate the stateful set because in many cases those changes cannot be done on existing objects. This is a Kubernetes limitation, but when stateful sets are recreated it is performed in a rolling fashion. Finally, changes to ClickHouse configuration itself, depending on the nature of the change, may be a very fast restart using SYSTEM SHUTDOWN, which does not require any pod recreation, or they may be done with no restart at all if the configuration change can be taken by the ClickHouse server automatically.

Speaking about restarts, we have invested a lot of effort to make sure that restarts are as smooth as possible and do not affect your application. The process involves multiple steps. It starts by removing the node from the load balancer. Then the node is removed from remote_servers. This is an internal ClickHouse cluster configuration. It makes sure that distributed queries, which may hit a different node and be executed on many nodes, will also skip this node. Finally, the operator waits for running queries to finish. We want to make sure your clients do not get interrupted, and only after that does it restart or recreate the pod.

Once the operation is completed, it performs everything in the reverse order. It adds the node back to remote_servers, then adds it back to the load balancer.

On big installations, we have examples of ClickHouse installations with hundreds of shards. Multiple shards can be processed in parallel to speed things up. You can imagine that if you need to upgrade 200 or 300 nodes one by one, it will take quite a lot of time, 3 to 5 minutes per node. But you may do multiple shards in parallel without service interruption, and the amount of parallelism can be configured.

A couple of notes about configuration. Different parts of configuration deploy differently. For example, config.d files and users.d files are common for all ClickHouse installations. Any change is propagated to all nodes. The conf.d setting looks the same but is mapped separately for every node, which allows per-replica customization. For example, you can provide different settings for different replicas in order to specify availability zones in which ClickHouse is running.

[43:51] – Maintenance Attributes: Stop, Restart, Suspend, and Troubleshoot Mode

Alexander: If you are running the operator with your cluster, you need to do some maintenance tasks, and we have support for many of them.

Stop: If you have a development system, you do not want to run it 100% of the time. You can stop it, and the operator will scale down all stateful sets to zero. The pods are deleted, but all the storage and everything else is there. As soon as you need to start it back, you just remove the stop attribute or set it to null and your cluster is up again with the same storage and the same data.

Restart attribute: Sometimes you may want to restart your cluster because something goes wrong. You can do it by specifying a restart task. The operator cannot perform tasks; it can only process Kubernetes resources. To emulate tasks, we have to do some tweaks. In this case, we use the start attribute paired with a task ID. Task ID is a way to specify that I want to execute this task, and the ClickHouse operator tracks executed tasks so each task is executed only once. If you want a new task, you change the task ID.

There are also two attributes for troubleshooting. There is a suspend attribute that stops reconciliation. Any changes you make to your ClickHouseInstallation when it is suspended are not propagated to Kubernetes resources. You may do multiple changes and then turn on the operator and deploy them all. And there is troubleshoot mode. It is important if your ClickHouse is in a crash loop and you do not understand why it is failing. You can turn on troubleshoot mode and the operator will leave the container running after the ClickHouse process terminates. So you can still exec into the pod, check the logs, try to start the process again, and figure out what was wrong.

There are a lot of controls to configure how the operator processes the reconciliation loop. You can configure concurrency, inclusion and exclusion rules, which probes to check when things are deployed, timeouts, and a lot more to tune your operation and make it as safe as possible. Many users require different things, and the operator provides all the ways to customize it. There is also a dedicated ClickHouse Operator Configuration resource, and in the operator documentation there is an example that contains all possible parameters with comments.

[47:25] – Monitoring: Prometheus Metrics Exporter and Grafana Dashboards

Alexander: The last thing, which is very important for production, is monitoring. You need to know how your ClickHouse is running. The operator provides this information using a special sidecar called the metrics exporter. It exposes a Prometheus endpoint on port 8888 and collects a lot of data from ClickHouse, including metrics, event data, table sizes, and much more.

ClickHouse itself has its own Prometheus endpoint, which was developed later and actually provides less information than our operator’s exporter. Therefore, we still stick with our approach where the operator queries ClickHouse in order to gather monitoring data and then exports that data to the endpoint.

It is worth mentioning that many users want to define their own metrics, and they can do so using a custom metrics table or any other table. This table should match the structure of the system.metrics ClickHouse table: same columns, same types. But you can define different things, maybe application-specific metrics. Usually it is done via a SQL view that extracts different information from your application data and exposes it to the same Prometheus endpoint.

For standard ClickHouse monitoring, we have Grafana dashboards that include dashboards for ClickHouse metrics, query analysis, ZooKeeper and Keeper, and a few others. Here is an example of one of the dashboards. You can see queries that are running right now, how many queries started per second, same for inserts, how many rows inserted per second. In this example you can see it is a pretty heavily loaded system, about three million rows per second. This dashboard may have a couple dozen different charts, and they were created by our support engineers to monitor ClickHouse.

[50:06] – Deployment Automation: Terraform, Helm, and Argo CD

Robert: There are a bunch of ways to automate deployment, and I am listing four of them here. Some folks are still using Ansible. There is a lot of Terraform use. Helm is really popular. And plain old manifests with a bash script are also a really great way for lots of custom deployments. We still see those. What I would like to focus on is Terraform and Helm as the most popular ways to deploy.

Terraform. The cool thing about Terraform is that it can deploy both your cloud infrastructure and the stuff that runs inside Kubernetes. Of all the ways you can deploy, it is the most versatile. You have to manage the state carefully, but if you are going to pick a single API, this is the one to use.

There are a couple of Terraform modules that we have that will help with this. If you are on Amazon, we built, with the help of our friends at Amazon starting about a year and a half ago, the Altinity EKS Terraform module. We unfortunately do not have this extended to GKE and Azure yet, but it is definitely good for AWS. If you are running this and want broader platform coverage or want something that does everything soup to nuts, there is Altinity.Cloud. This is our cloud platform for ClickHouse. It has a great Terraform provider that sets up what we call a bring-your-own-cloud model, where ClickHouse is going to run inside your environment using your data, using a Kubernetes cluster that we set up for you, and there is a BYOC Terraform module that handles that setup. Then you are able to manage it not just on the command line but also via the Altinity.Cloud UI.

Helm. People love Helm, and we do have Helm charts. An important one, very convenient if you are just setting up the operator, is a Helm chart to set the operator up the first time. You just add our repo, update, and install the operator. We also have a sample application for ClickHouse. What I like about Helm is that for managing the manifests for ClickHouse clusters, it is a pretty convenient way to do it. A lot of people use Helm and couple it with something like Argo CD, which is really good at running Helm charts at the right time.

One thing I want to point out on Helm: if you are not already aware, Helm does not deal with operators very well. The problem is that it cannot upgrade an operator because it cannot properly update the CRDs, these custom resource definitions, because what they do is change the behavior of the API of the thing it is trying to upgrade, and Helm just throws up its hands. So there is a special procedure: if you are going to use Helm for the operator upgrade, you actually have to load the new CRDs manually and then run helm upgrade. Just a tip for the unwary.

[54:14] – Altinity Operator vs. the ClickHouse Inc. Operator

Alexander: For years we were the only operator for ClickHouse, and actually the Altinity ClickHouse Operator is the most popular operator for databases after PostgreSQL on GitHub. We collect statistics on that. It has about 2,500 stars right now, which is pretty significant for an operator. This is because we introduced it relatively long ago in 2019 and we were running it in production. Since then, ClickHouse Inc. has started developing their own operator, and in January 2026 they introduced what they call the official ClickHouse operator for Kubernetes. Of course, we were curious what it is about, and a lot of our users are also asking.

It is pretty hard for us to be unbiased, of course, but it is evident that ClickHouse Inc. is very early in their game with the operator.

The Altinity Operator is used in production by multiple companies and is operating thousands of clusters. I am not talking about Altinity.Cloud solely, but companies like eBay and OpenAI are global users of the operator, among many others. For the ClickHouse Inc. operator, it is not clear if it is used in production or not. It is certain that it is not the operator they use in their own cloud, which is quite strange, because for us it was very good that we operate with our own operator. That drives our development. We were very heavy users of it ourselves.

The Altinity Operator is certified for a number of Kubernetes platforms. By certified I mean we have production deployments on those, and these are not just the big three but also DigitalOcean, Linode, OpenShift, and some exotic ones like Oracle Cloud Kubernetes.

The Altinity Operator supports any ClickHouse distribution, any ClickHouse build. ClickHouse Inc., I believe, tends to use only official builds because they want to be 100% official.

I showed you a bit of configuration management, but that is just the tip of the iceberg. There are so many things you can do to customize your ClickHouse deployment. As far as we can see right now, the ClickHouse Inc. operator does not invest heavily in this area. They want to make it as simple as possible, fire and forget, which is usually not true for production.

We support any database engine. When we started, the only engine was Ordinary. Now there is Atomic, Replicated, and many others. Everything can be used with the Altinity Operator. ClickHouse Inc. forces users to use the Replicated database engine, which is great and convenient, but it is a limitation. It can only be used with Keeper.

Speaking of that, we support Keeper and are pretty agnostic about what you are using. ClickHouse Inc. has integrated Keeper support in their operator. By the way, Keeper is production-ready and has been for several years, but it is still slower than ZooKeeper on high-rate transactions. This is frankly ridiculous. We recently retested it and were very surprised to see that, and we are now preparing a pull request with some changes to ClickHouse in order to improve concurrency for Keeper.

Finally, backup and restore. We can do it with a sidecar, and by that I mean it is full backup management. You can do table-level backup and restore. You can use a lot of different media to store your backups. ClickHouse Inc.’s operator has backup on their roadmap for this year. We will see what they come up with.

The good thing is that both operators are Apache 2.0 licensed, so you can use either if you want to.

[59:15] – 2025–2026 Roadmap Highlights

Alexander: Finally, to wrap up our presentation, here are some major new features for the Altinity Operator this year. We are typically not very good at communicating our roadmap, so we want to use this opportunity.

First, we will add post-start and pre-stop hooks. These are needed to better support Project Antalia. For example, when swarm nodes are shut down, we want to do it gracefully, and that requires some SQL execution on the shutdown sequence. We plan to add special SQL hooks for that.

Also, we started to productize the Keeper support, and tighter integration of CHI and CHK is coming later this year.

An interesting development we are considering is how to add new replicas with smaller downtime and faster. It can potentially be done via persistent volume snapshots. The snapshot itself takes time to make, but after you have a snapshot, you can spin up a new replica from that snapshot and only need to catch up replication for the smaller amount of time between when the snapshot was taken and when you actually recreate the volume from it. Some of our users in the wild have tried this approach and found it very interesting.

Finally, we have some pressure from users to make the operator extensible using plugins. We can imagine the first plugin would be backup: you can plug backup in as a plugin and configure it using manifests. Another example is a resharding plugin, which would do some heavy lifting outside of the operator but still be controlled by it.

That is it from what we wanted to share with you.

[1:01:54] – Q&A

Robert: We do have time for questions. By the way, thank you everybody for listening. We realized as we got into this that there is just such a huge amount of stuff in here. We have been working on this pretty hard since 2019, so it is coming up on about seven and a half years since development started.

The mother ship is the Altinity Operator GitHub. That is the place to find information. If you have other questions, please contact us and join our Slack. You can ask us interactively. If you have specific issues or feature requests, please file issues, or better still do pull requests. We have had lots of community contributions on this.

Robert: We have a question from Paul Julian, who had to drop off. He asked: I have seen recently that ClickHouse Inc. is working on their own operator for ClickHouse on Kubernetes. What is the path moving forward?

We are 100% committed to the Altinity Operator, as you saw from this presentation. Our whole business depends on it, as do hundreds, if not thousands, of companies. Around 300 customers at Altinity right now, and a substantial chunk, at least 80%, are using the operator in some form or another. We are going to keep pushing as hard as we can on this. And as Sasha says, we will keep an eye on what is going on with the upstream ClickHouse operator. Having production systems running on the operator, like our own cloud, is a real forcing function to get the quality right. Rolling upgrade is a very hard problem, particularly at scale.

Robert: Sasha, there is a question in the Q&A about NVMe. They are using EKS and asking about autoscaling.

Alexander: This is a hard one. As you know, NVMe disks are local disks, not network disks, so you cannot easily scale compute, and this is one of the main reasons we do not use them in the cloud, even though we did experiments with NVMe disks five years ago when we started operator development. Having said that, we now have a use case for NVMe: it is a high-performance cache for Parquet data for Project Antalia. We will develop some support and expertise on that. The biggest problem is how to mount them properly into your Kubernetes pod, because normal Kubernetes provisioners do not deal well with local disks. They depend on the node rather than the cluster itself. We are still looking for the best way to support it in our cloud, and as we figure it out we will definitely share it with the community.

Robert: Another question: documentation about custom metrics. You can go to the ClickHouse Operator project on our GitHub. You can also just clone the code and turn Claude loose on it and ask it any question you want. It will read the code and give you pretty good answers. We are starting to use that ourselves to remind ourselves what the code does.

Robert: Here is a question: does a ClickHouse Helm chart support KEDA so that I can use KEDA for scale-up? The answer is no. We are very interested in KEDA but we do not use it currently. Alexander, do you have any expansion on that?

Alexander: Yes, we were thinking about this, but unlike small OLTP databases, ClickHouse is typically a heavy database. If you have a lot of data, you cannot easily use automatic scaling mechanisms to add more resources. So we were a bit skeptical. But things may change. As we started to move towards storing more data in object storage, we may start using KEDA for swarms. Swarms are stateless compute servers, ClickHouse servers that only query Parquet data or data from object storage. They do not have any state and can be scaled up and down pretty dynamically. So for this use case it may be a good solution.

Robert: And I think that is a really important point. We have said very little about Project Antalia in this webinar because we are talking about the operator, and there is enough to talk about here already. But the big gap is not in the operator; it is in ClickHouse itself. Open-source ClickHouse does not have built-in separation of storage and compute. That is exactly what we are putting into Project Antalia. Two major features: one is these swarm clusters, which allow you to scale up and scale down resources for querying Parquet data on object storage, including Iceberg tables. The second is hybrid tables, which allow you to do tiered storage out to data lakes. In both of these cases, you now have the ability to scale compute up and down. Normal ClickHouse, because you have to replicate the storage when you build new replicas, is incredibly slow. It is a heavyweight operation. There is almost no point in automating it. But once you have swarm clusters, as Alexander says, it is a completely different picture. And not just scaling up and down, but also using things like spot instances much more widely to reduce compute costs.

Alexander: One final word. The Altinity Operator is an open-source project and we have more than 100 contributors. If you have something to contribute, you are totally welcome. We accept contributions and we love them.

Robert: Thank you all so much for attending. We hope this was useful. Feel free to ping either of us directly on LinkedIn. Thank you so much for the questions and join our Slack. Try the operator out and let us know what you think.

FAQ Section

Q: What is the Altinity Kubernetes Operator for ClickHouse and how does it work?

A: The Altinity Kubernetes Operator for ClickHouse® is an open-source operator built on the Kubernetes operator pattern. It allows you to define and manage ClickHouse clusters using simple YAML manifests (custom resources called CHI for ClickHouseInstallation and CHK for ClickHouseKeeper Installation). The operator reads those manifests and continuously reconciles the actual Kubernetes state to match the desired state, handling everything from pod creation to storage provisioning, configuration management, and rolling upgrades.

Q: How do I safely prevent accidental data loss when deleting a ClickHouse cluster on Kubernetes?

A: Use reclaimPolicy: Retain in your volume claim template. With this setting, the persistent volume claims (PVCs) will survive even if you accidentally delete the entire ClickHouseInstallation. When you recreate the CHI resource, the operator will find the existing storage and reconnect to it. This is one of the most important safety settings for production deployments.

Q: Can I scale a ClickHouse cluster on Kubernetes without downtime?

A: Yes. The Altinity Operator supports zero-downtime rescaling for both replicas and shards. When adding a replica, the operator creates the new pod and storage, automatically propagates the schema, and waits for replication to catch up before adding the new node to the load balancer. Removing replicas also follows a graceful procedure that removes the node from the load balancer and cleans up ZooKeeper or Keeper metadata automatically.

Q: How do I back up ClickHouse when running on Kubernetes?

A: The recommended approach is to use Altinity Backup for ClickHouse® as a sidecar container in the same pod as your ClickHouse server. Because the sidecar runs in the same pod, it shares the storage namespace and can perform hard-link-based backups efficiently. Backups are written to object storage such as Amazon S3. This approach supports table-level backup and restore, incremental backups, and a variety of storage backends.

Q: How does the Altinity Operator handle ClickHouse version upgrades?

A: Version upgrades are performed as rolling restarts, with minimal impact on running queries. The operator first removes the node from the load balancer and from remote_servers (so distributed queries skip it), waits for running queries to finish, restarts or recreates the pod with the new image, and then adds the node back. On large clusters, multiple shards can be upgraded in parallel to reduce total upgrade time. The operator is version-agnostic and works with any ClickHouse build.

Q: What is the difference between the Altinity Kubernetes Operator and the ClickHouse Inc. operator?

A: The Altinity Operator has been in active development since 2019, is used in production by thousands of clusters including Altinity.Cloud, and supports a very broad range of configuration management features, Kubernetes platforms, ClickHouse distributions, and database engines. The ClickHouse Inc. operator was introduced in January 2026 and is at an early stage. Notably, ClickHouse Inc. does not currently use it to run their own cloud. Both operators are released under the Apache 2.0 license.

© 2026 Altinity, Inc. All rights reserved. Altinity®, Altinity.Cloud®, and Altinity Stable® are registered trademarks of Altinity, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc.; Altinity is not affiliated with or associated with ClickHouse, Inc. Kubernetes, MySQL, and PostgreSQL are trademarks and property of their respective owners.