Altinity.Cloud Anywhere: An Open Cloud for ClickHouse

I will be there
And everywhere
Here, there and everywhere
The Beatles

Clouds are convenient. Cloud providers manage infrastructure and services, and let users focus on applications. Managed services raise a very important question: who owns the data? Cloud and service providers own users’ data, and while they usually deploy extensive security controls to make sure data is safe and protected, it is not always sufficient. Many companies simply need to own the data themselves. Whether for security compliance, privacy, cost, avoidance of lock-in, or any other reason, the requirement is the same.

Data ownership requirements have long obliged users to stick with self-managed or on-prem deployments. Users control everything, but it comes at the high cost of operating and managing infrastructure. 

Is it possible to combine cloud management convenience with on-prem levels of ownership and control? There is a new reason to say “yes.” Altinity.Cloud Anywhere introduces a new concept of an “open cloud” that allows us to plug other clouds or on-prem deployments into our management plane. The only requirement is Kubernetes. Now we can manage ClickHouse wherever Kubernetes runs, which is virtually anywhere. 

What is Altinity.Cloud Anywhere?

The problem is not just operations and data ownership. For the business, the real values are applications and products. A modern application requires many services to work together, like databases, streaming engines, and others. All those are available in public clouds, but using multiple cloud services introduces a lot of dependencies, and is also costly. This is the second reason why many companies decide to manage services on their own. What enables development of the stacks required by these services? Open Source!

Let’s take a real example, one of many similar ones we have encountered. A company develops an application that requires a Kafka broker, ML pipeline, web UI, databases, and others. Wiring everything together using public cloud services is cumbersome. It is also inconvenient for developers who need to start dev instances quickly. So the company decides to use Kubernetes with open source deployments. All wiring is based on service names or internal DNS records, which is easy and transparent for developers. Now they need to add ClickHouse to the stack – does it make sense to go for a managed cloud service in this architecture? Probably not. They will want ClickHouse in Kubernetes as well. However, without ClickHouse expertise, it takes a lot of time and effort to get ClickHouse properly configured, managed and monitored. This is where Altinity.Cloud Anywhere comes in.

Altinity.Cloud Anywhere manages ClickHouse clusters in user Kubernetes environments. Users own and control all their resources, and use Altinity.Cloud console in order to manage ClickHouse clusters inside their own Kubernetes. Altinity.Cloud deploys components inside user’s Kubernetes required for ClickHouse operation, providing management and monitoring tools. The Kubernetes cluster does not need to be dedicated for ClickHouse only – it can be used for other services, and applications. A picture is worth thousands of words, so let’s look into the architecture diagram below:

The key component is the Altinity Connector. Altinity Connector is a management channel between Altinity.Cloud and the managed environment. It is deployed inside the user’s Kubernetes, dials back to the management plane, and enables Altinity.Cloud to manage ClickHouse resources. In order to do that, it deploys the open source Altinity Kubernetes Operator for ClickHouse, monitoring components, a load balancer and other components. It also inspects environment capabilities, like storage classes and node types, and makes them available for Altinity.Cloud.

Once everything is configured, users log in to their Altinity.Cloud account, start ClickHouse clusters from there, and go to building apps leveraging the true cloud experience.

Deploying the Altinity Connector inside the Kubernetes cluster is a one-time effort that requires minimal configuration, including addition of a service account and ensuring network connectivity. The Kubernetes cluster also needs to be capable enough in order to fully utilize Altinity.Cloud functionality, like high availability, expanding the storage, etc. Such capabilities are available off the shelf in managed Kubernetes offerings, like AWS EKS or GCP GKE. Using them dramatically simplifies Kubernetes management.

Many companies are using managed Kubernetes already, so setting up the Altinity Connector is the only thing they need to do in order to get their environment connected to Altinity.Cloud Anywhere. For others, we can get to the next level of automation and take care of the Kubernetes deployment part as well. Users won’t even have to know Kubernetes is there.

Bootstrapping the Kubernetes cluster

Altinity Connector can be deployed inside the Kubernetes cluster, and it can also be deployed outside. For example, it can be run as a binary inside an EC2 instance, or even on a local machine. Why might one want to do this? In order to spin up a properly configured Kubernetes cluster in the user’s own cloud account and connect it to Altinity.Cloud Anywhere!

Once the connector is started on a VM with proper permissions in the user account, Altinity.Cloud can bootstrap the Kubernetes cluster with optimal configuration, register it in the management plane, deploy Altinity Connector, and hand it over to the user. This capability will be available to users shortly. 

Adoption

Altinity.Cloud Anywhere may do something even more unique. We call it “adoption”. It works with an existing ClickHouse installation in Kubernetes managed using the open source Altinity Kubernetes Operator for ClickHouse. The process starts with deploying the Altinity Connector. After that Altinity.Cloud can read existing ClickHouse cluster resource definitions and take over the management. 

Adoption currently requires some manual customization. But the benefits are obvious: ClickHouse users have installed thousands of ClickHouse clusters with the Altinity Operator already. This is a unique opportunity to get their clusters managed without a painful data migration process. We are systematically smoothing out the process to make adoption easy for users, regardless of their level of familiarity with Kubernetes or the operator. 

Vendor unlocking

Adoption demonstrates the unique openness of the Altinity.Cloud platform. Clusters that are managed by Altinity.Cloud can be connected and disconnected from the management plane and keep working! Unlike traditional cloud vendors this is super easy: If you want to disconnect from the cloud, it is sufficient just to “unplug” the Altinity connector and fly on your own. Certainly, users would lose all the neat features of managed ClickHouse if they disconnect, but analytic applications will keep running.

Vendor unlocking is possible thanks to a not-so-secret strength of Altinity.Cloud – the components it manages, including ClickHouse, the Altinity Operator, Prometheus monitoring, and others are 100% open source. Users can run entire analytic pipelines themselves if they wish, adding and removing management from vendors as needed. We believe such freedom offers enormous advantages to enterprises. 

The model goes even further. Altinity.Cloud can move ClickHouse clusters between environments using efficient backups. So one may start in GCP Kubernetes cluster and migrate to AWS EKS, or even to a self-managed user environment.

Anywhere and Everywhere

Altinity.Cloud is a universal platform for managing and operating ClickHouse clusters in cloud environments. It currently supports AWS and GCP as primary platforms. 

Altinity.Cloud Anywhere extends the management to user-owned environments, and can go well beyond the mainstream cloud providers. Kubernetes and open source components make it possible for us to have cloud management while still only not just data but apps and the operating environment too. 

Altinity.Cloud Anywhere provides a flexible and open model that adapts to user requirements and infrastructure.

Let users focus on their applications, and we will handle ClickHouse.

Please sign up for Altinity.Cloud or Altinity.Cloud Anywhere, and we will do the rest.

Anywhere and Everywhere!