DeepSeek, ClickHouse®, and the Altinity Kubernetes Operator: A Twisted Tale

Last month DeepSeek jumped out of a hidden datacenter and boiled markets and minds. But it also affected ClickHouse users. Apparently, DeepSeek is using ClickHouse, and curious people found that its ClickHouse was fully open to the Internet, exposing sensitive information. How could this happen?

By default, ClickHouse is launched with pretty relaxed security. In particular, the default user may be used with no password. This is a significant security vulnerability. We patched it long ago in ClickHouse deployments managed by the Altinity Kubernetes Operator for ClickHouse, by introducing restrictive network rules (read our ClickHouse security hardening guide for the details). However, deployments with Docker compose were not protected, and it played out badly for DeepSeek.

The ClickHouse development team rapidly reacted to the news and introduced this security patch that disabled network access for the default user. They also quickly backported it to the last 3 releases and 2 LTS releases. In total, 5 ClickHouse releases got this patch:

• 25.1.3.23
• 24.12.4.49
• 24.11.4.45
• 24.8.13.16
• 24.3.16.16

Unfortunately, it broke some Kubernetes deployments that used the clickhouse-server:latest image, or upgraded to one of the versions above.

This happened because in order to disable network access for the default user, ClickHouse engineers decided to overwrite a file with out-of-the box user definitions when starting a Docker container. It is perfectly fine for a ClickHouse Docker container that runs on its own. In Kubernetes, however, configuration is typically managed by the operator and mounted as a read-only file. As a result, the Docker startup script (entrypoint.sh) failed with an error and ClickHouse could not start.

The ClickHouse team quickly discovered the problem and submitted the patch to the patch. It has been already included in the releases listed above. This patch allows the user to rollback the change by setting an environment variable. 

Fortunately, no Altinity.Cloud users were affected. We are not using Docker entrypoint.sh in Altinity.Cloud at all to avoid the risks of changed ClickHouse startup behavior. However, other Kubernetes users noticed the problem when using the Altinity Kubernetes Operator for ClickHouse and escalated it to our GitHub project. It took us a day to test and release a new version of the operator – 0.24.4 – that included a workaround. 

This is a good story to repeat and re-learn some lessons.

First, protect your ClickHouse deployment properly. ClickHouse is already very popular, and it is easy to scan standard ports across the Internet and find unprotected ones. So ClickHouse should not be exposed to the Internet at all, or, if it is, it should be protected by IP whitelisting or other means. You should also follow our rules for hardening the default user.

Second, do not use ‘:latest’, ‘:24.12’ and other “open-ended” tags for container images. While it may seem convenient in order to get updates automatically, it is very risky, especially for production environments. You may not notice that the version has changed for several days or even weeks, until a pod is restarted for some reason and a new image brings the cluster down.

And finally, open source rocks! Thanks to users who quickly report such problems and allow us to release fixes ASAP. (And, of course, thanks to the ClickHouse core team for responding so quickly to the original problem.) This is the true power of collaboration between developers and community that makes our products better.