Own your ClickHouse® data with Altinity.Cloud Anywhere

Recorded: Tuesday, January 17, 2023
Presenters: Robert Hodges & Alexander Zaitsev

Altinity CEO Robert Hodges introduces Altinity.Cloud Anywhere, the offering that extends Altinity.Cloud’s managed ClickHouse® service to any Kubernetes cluster the customer controls, whether that is Amazon EKS, Google GKE, or even a physical machine running Kubernetes in a home closet. The core idea is cloud convenience without cloud lock-in: Altinity manages the cluster, but the data lives entirely in the customer’s VPC.

The session is structured as a live demo followed by a step-by-step walkthrough. Robert first demonstrates the product working in real time: a rescale operation initiated in the Altinity.Cloud console propagates to an EKS cluster in Oregon, and a table created directly in a Minikube cluster running locally becomes immediately visible through the management plane. He then walks through the three setup steps in detail: preparing a Kubernetes cluster with the necessary IAM roles, CSI drivers, and autoscaler; connecting it to Altinity.Cloud using the Altinity Connector; and configuring node types, backup storage, and access controls.

Key architectural points covered include how the Altinity Connector works as a secure outbound-only HTTPS connection with no inbound ports required, how all analytics stack components including the Kubernetes Operator, Prometheus, and Grafana, are open source so the cluster continues to run normally if the management connection is dropped, and how the Altinity access control allows customers to set read-only or no-access restrictions on Altinity personnel at any time.

Here are the slides:

Key Moments (Timestamps)

Key moments generated with AI assistance.

• 0:02 – Introduction: Robert Hodges and Alexander Zaitsev
• 0:32 – Audience and prerequisites: app developers, Kubernetes users
• 1:22 – Altinity background: founded 2017, Altinity.Cloud, Kubernetes Operator
• 2:09 – What is Altinity.Cloud Anywhere?
• 2:20 – ClickHouse overview: real-time analytic database, SQL, open source
• 4:10 – What Altinity.Cloud does: fully automated managed ClickHouse
• 5:09 – Why the standard SaaS model doesn’t work for everyone: VPC, compliance, cost, lock-in
• 6:11 – Demo begins: Altinity.Cloud Anywhere on Amazon EKS in Oregon
• 7:03 – Live rescale operation: adding a node to the EKS cluster
• 10:50 – Second cluster demo: Minikube in Robert’s closet
• 11:26 – Creating a table locally and seeing it in Altinity.Cloud
• 13:10 – What this means: Cloud management of any Kubernetes, anywhere
• 14:03 – Back to slides: architecture overview
• 15:38 – Three setup steps: prepare Kubernetes, connect to Altinity.Cloud, go
• 15:55 – Which Kubernetes distributions are supported: EKS, GKE, Minikube, others in progress
• 16:38 – Step 1: Preparing an EKS cluster (eksctl, IAM, CSI driver, Karpenter)
• 19:50 – EKS cluster provisioning YAML
• 20:31 – Installing the EBS CSI driver and storage class fix
• 22:04 – Adding Karpenter autoscaling
• 22:44 – Testing the cluster before connecting
• 24:07 – Step 2: Connecting to Altinity.Cloud (altinity-cloud-connect)
• 26:12 – What the connector commands look like in practice
• 27:34 – Resource configuration: AWS button, storage classes, node pools
• 29:03 – Connection process and DNS provisioning timing
• 30:09 – Step 3: Working with clusters in Altinity.Cloud Anywhere
• 30:42 – The cluster experience is identical to standard Altinity.Cloud
• 32:17 – Differences: controlling which node types Altinity can use
• 33:53 – Backup configuration: S3 bucket, schedule, retention
• 36:05 – Creating and restoring backups manually
• 37:01 – Advanced topics: private connectivity, running other apps in the cluster, DR
• 39:02 – Architecture under the hood: ClickHouse, Operator, system services
• 40:08 – Vendor unlock-in: disconnect the management plane, clusters keep running
• 41:00 – Data stays in your VPC: what Altinity stores and doesn’t store
• 42:07 – Altinity access control: no-access, read-only, full-access
• 42:49 – Enterprise support model
• 44:03 – Support scope: Kubernetes issues included
• 44:53 – Pricing: based on compute only, no data charges
• 46:00 – Roadmap: Azure, DigitalOcean, Red Hat; automatic EKS provisioning coming
• 47:26 – Call to action: two-week free trial

Webinar Transcript

[0:02] — Introduction

Robert: Welcome, everybody. This is Robert Hodges, and I’m delighted to welcome you to our webinar on Altinity.Cloud Anywhere and how it allows you to own your own data but still have Cloud management. We’ll go into all the details in this talk. I’m joined today by Alexander Zaitsev, our CTO, who has been the driving force behind developing the software we’ll be showing you today.

Just a few introductions: here at Altinity we’re database geeks and Cloud geeks, as you’re going to see in a few minutes. Collectively we have centuries of experience in databases and applications, particularly analytics. About two-thirds of our 45 people are engineers. We’re pretty big on databases and the applications that run on top of them. We assume you’re an app developer looking to build real-time analytics on ClickHouse®. You may not have heard of ClickHouse yet, and we’ll give you a quick intro. And there’s a good chance you like Kubernetes, because we’re going to show you how to run ClickHouse fully managed on your own Kubernetes clusters.

[1:22] — About Altinity

Robert: Some background on Altinity: we’ve been around since 2017. We offer enterprise support and services for ClickHouse including Altinity.Cloud, which was the first public cloud offering of managed ClickHouse on both Amazon and GCP, started about two and a half years ago. We also do quite a bit of open-source work. If you use Kubernetes you may be familiar with our Kubernetes Operator for ClickHouse, which is used by literally thousands of clusters worldwide. We do a bunch of other open-source projects as well.

[2:09] — What Is Altinity.Cloud Anywhere?

Robert: Let’s start by answering the question: what is Altinity.Cloud Anywhere? It’s a managed service for ClickHouse, but what is ClickHouse? Just in case you haven’t heard of it, ClickHouse is a real-time analytic database. It’s outstandingly fast and allows people to deliver very quick answers on rapidly arriving data, often millions of rows per second, where people need to know within seconds of events occurring in the real world. For example, if you have a security problem, you can run analytics very quickly to discover the scope, where it emerged, how long it’s been going on.

How does it do that? It’s a SQL database, so it talks a variant of SQL. It runs on bare metal or cloud, extremely portable, runs anywhere Linux does, from a phone to a Docker container to rack equipment or VMs in the cloud. It has a shared-nothing architecture where nodes own patches of storage and are connected by a network. This model is evolving toward a more separated compute and storage architecture but for now this is basically how it works. Other things you’d expect from an analytic database: columnar storage, parallel and vectorized execution, the ability to scale to many petabytes. And it’s open source. Taken together, it’s become the most popular real-time analytic database on GitHub, with hundreds of unique contributors per year and thousands of users.

[4:10] — What Altinity.Cloud Does

Robert: Altinity.Cloud runs this ClickHouse data warehouse in the cloud for you. It completely automates everything you need to do to set up and run ClickHouse, from initial installation all the way to upgrades and rescaling, with tools for DBA support. The cloud is optimized for real-time analytics: rapidly arriving data, quick answers, consistent response. It supports all versions and features of ClickHouse. It runs in Amazon, GCP, and, this is the new part, your own Kubernetes clusters.

[5:09] — Why the Standard SaaS Model Doesn’t Work for Everyone

Robert: The standard SaaS model is what most of our customers use right now, but for many people it isn’t the right model because the data runs in our VPC and they don’t have control of it. That has a number of effects depending on your business.

You might want to run in your own VPC because you have security compliance requirements. You might want to use your own infrastructure because you can control costs better. You might want to keep things close to your on-prem data sources. And this is a really important one: many people want something that’s 100 percent open source with the data under their control so that they don’t have lock-in to a vendor that controls the data with opaque software they can’t run themselves. Altinity.Cloud Anywhere solves this problem completely.

[6:11] — Demo: Altinity.Cloud Anywhere on Amazon EKS

Robert: The simplest way to prove that is to show you rather than spend more time on slides. Let’s do a demo.

So we’ll go to an Altinity.Cloud Anywhere cluster. This is what the Altinity.Cloud interface looks like if you haven’t seen it before. Let me show you where this is running: it’s running in what’s called an environment. This is basically a dashboard for an Amazon EKS managed Kubernetes cluster running in US West 2, somewhere in Oregon near the Columbia River.

You can see Altinity.Cloud can see into this environment. We can see VMs and manipulate things in a couple of namespaces. We can also manage clusters in there. Using the standard Altinity.Cloud interface, we can go ahead and manage this cluster directly.

Let me also show you something interesting: this is a kubectl command, the standard administrative command for Kubernetes, doing a watch and showing us the pods running ClickHouse in this Kubernetes instance.

[7:03] — Live Rescale Operation

Robert: Let’s make a change to this system using Altinity.Cloud and show how it affects this EKS cluster. We’re going to do a rescale operation and add a node, basically adding another replica that can support an additional subset of our data.

I’ll go ahead and confirm this operation. It should be starting now. What we should see is a new pod coming up in the Kubernetes cluster directly, because I set this cluster up myself and can manage it directly. Currently this cluster is set to auto-scale, which means it’s actually spinning up a new VM to support the new node. This will take a couple of minutes, so while it’s doing that I’m going to quickly show you something else that illustrates the power of this model.

[10:50] — Second Demo: Minikube in Robert’s Closet

Robert: This particular cluster is running in Minikube in my closet about 10 feet away from me here in California. Let me go into this cluster’s explore interface and show the tables available: none right now.

I’m going to log into the machine in my closet and create a table. Here we go. I’m logged into the machine in my closet, it’s called logos2. Let me use kubectl to talk directly to the ClickHouse instance, go into the pod, start the ClickHouse client, and create a table called foo.

I’ve just created it completely locally, going from my laptop over my wireless network to my machine. Let me do SHOW TABLES and there it is: table foo. Now let’s go back to Altinity.Cloud and run SHOW TABLES there. And there you are: that table is now visible to Altinity.Cloud.

[13:10] — What This Demonstrates

Robert: This is an example of the power you have with this model. You have what amounts to installed software on Kubernetes, which means we can set it up to run in practically any environment because Kubernetes runs just about everywhere. At the same time we have full Cloud management: I can provision, extend, and upgrade this cluster from a central management plane. I have Cloud convenience but full control over the actual infrastructure and the data it’s running on.

Let me check on the original EKS cluster. We now have two nodes. Another node was allocated, both pods are fully running. So you have a complete Cloud experience for managing this, running on Kubernetes clusters, and it’s not just restricted to clusters in the cloud: it can manage a cluster running on-prem as well.

[14:03] — Architecture Overview

Robert: A little bit of background. We call these Kubernetes clusters “environments,” and that’s what an environment really is: a Kubernetes cluster underneath. In the traditional Altinity.Cloud we run them in our own VPCs and provision them soup to nuts. What’s going on with Altinity.Cloud Anywhere is we’re now using exactly the same model but with somebody else’s Kubernetes cluster.

To make this work we add one key piece of technology called the Altinity Connector. It’s a pod that runs in your Kubernetes cluster and sets up a secure outbound connection using HTTPS that comes to Altinity.Cloud, establishing a bidirectional management channel. We don’t move data: it’s relatively low bandwidth, but it gives us the ability to reach into these clusters securely and manage them.

[15:55] — Which Kubernetes Distributions Are Supported?

Robert: For running ClickHouse in your own Kubernetes environment, right now the most common choices among our users are EKS, the Amazon managed service, and GKE, Google Kubernetes Engine. Minikube we use for testing and demos only; I would not recommend running production on it. We are working on certifications for Azure, Red Hat, and DigitalOcean.

[16:38] — Step 1: Preparing the EKS Cluster

Robert: There are three steps to get started. The most complex is to prepare Kubernetes, but if you’re a Kubernetes person you won’t find it too bad. Then we register with Altinity.Cloud and we’re off to the races.

For EKS specifically, the Kubernetes requirements for Altinity.Cloud Anywhere are fairly simple. There are essentially three things to do: provision a cluster, set up Auto scaling, and make sure you can manage block storage for data.

We use eksctl, a really handy command-line utility for setting up EKS clusters. You start with a jump VM. The reason is that with Amazon you can build a role and assign it to that VM, which then gives it the permissions needed to set up EKS. After that you install the AWS CLI, eksctl, kubectl, and the AWS IAM Authenticator. All this can be set up in about 10 minutes.

To provision the cluster, eksctl takes a specification file. Here’s an example: M5 large instances, with the relevant node group parameters. You feed this to eksctl and it spins up an EKS cluster that’s immediately usable.

[20:31] — CSI Driver and Storage Class

Robert: As of Kubernetes 1.23, the CSI driver that handles EBS, Amazon’s cloud block storage, has to be installed separately. You need to create an IAM policy that allows the CSI driver to manipulate block storage, and then install the driver itself. eksctl makes this very simple: two commands, a couple of minutes to run.

One small gotcha: if you’re manipulating CSI drivers, you need to make sure your storage classes know you’re using EBS and this specific CSI driver. If you go with the default out-of-the-box EKS storage class it won’t find the provisioner correctly. Adding the correct provisioner name to your storage class cures that. Three commands total for this step.

[22:04] — Karpenter Autoscaling

Robert: For autoscaling I recommend Karpenter, which is a one-size-fits-all provisioner that will both allocate and deallocate VMs, and not just one VM type but many, so it’s very configurable. It has excellent documentation, installs via a Helm script, and requires defining what the provisioner looks like. Once you’ve got that you’re pretty much done with the Kubernetes setup.

One thing I highly recommend: test the cluster out before connecting. Bring up a test application that allocates a volume. What you want to see is that you can allocate a VM, that storage gets correctly allocated, and when you take the deployment away, that storage goes away too. At that point you’ve got a working Kubernetes cluster and can begin connecting it to Altinity.Cloud.

[24:07] — Step 2: Connecting to Altinity.Cloud

Robert: Connecting to Altinity.Cloud is quite simple. If you don’t already have an account, you can fill out a trial request for a free two-week trial. When you log in with a new account it will put you straight into a workflow to set up the connection.

The key piece is the Altinity Connector, also called altinity-cloud-connect. The steps are exactly shown in the UI. You install the software, run a command that authenticates you with Altinity.Cloud, and then the Altinity Connector generates a script that creates a service account and a couple of namespaces and other resources in your Kubernetes cluster. You just pipe that into kubectl and you’re set up.

Here’s what those commands look like in practice:

# Download and install the connector
# Authenticate with Altinity.Cloud
altinity-cloud-connect login
# Generate the setup script and apply it
altinity-cloud-connect kubernetes | kubectl apply -f -

If you’re paranoid, you can inspect the script before applying it so you or your security team can verify there’s nothing unexpected. Once you’ve done that, go back to Altinity.Cloud and press proceed.

[27:34] — Resource Configuration

Robert: The next page is resource configuration, where you do some basic setup to let Altinity.Cloud know what resources are available. You’ll flip the AWS button to let us know you’re on Amazon, which helps us scan resources correctly. We list block storage types automatically if we can see storage classes; you can add additional ones if needed. Then you set up initial node pools: something like “I have three zones, I can use M5 Large instances, up to ten per zone, for ClickHouse or ZooKeeper or system services.” You fill this in, press proceed, and the connection process begins.

A fair warning: you’ll see a connection progress bar that may seem to time out. Certain things in Kubernetes required for setup, like DNS provisioning and certificate provisioning on Amazon, take tens of minutes. The best thing is just to press finish and go away for a while. When you come back, if everything is set up correctly you’ll see a dashboard with full stats, allocated RAM and storage, and your current nodes.

[30:09] — Step 3: Working with Clusters

Robert: Once you’re in, the cluster experience is essentially identical to standard Altinity.Cloud. The cluster view, the dashboard, the ability to create or modify clusters, monitoring, connection details, configuration, DBA tools, alerts, and logs: it’s all the same complete Cloud experience, just running on infrastructure you own.

[32:17] — Differences: Controlling Node Types

Robert: There are some differences because it’s your infrastructure. Altinity.Cloud is flexible about rescaling and normally populates a wide range of node types automatically. When you’re running Anywhere, you control which node types we’re allowed to use. In my demo I only see one node type because that’s all I’ve configured. You need to tell Altinity.Cloud that more options are available. You do this through a “manage node types” button on the environment dashboard: see the existing node types, press “add node type,” enter the machine type name, and the memory and CPU fill in automatically. That’s it.

There’s no danger of Altinity.Cloud using resources you haven’t approved: if you don’t tell us there are nodes we can use, we won’t use them.

[33:53] — Backup Configuration

Robert: Backups are configured in the environment pane under Edit. You turn them on, set a schedule, and set retention. In my example I’ve set it to run daily at 5 AM and keep the most recent three backups. Then you give us information about where to store the backups: an S3 bucket with an access key, secret key, region, and bucket name. That’s all there is to it.

Backups are one of those painful things that you just want to get right and then fire and forget. Once it’s configured it just runs. Backups run automatically on schedule, but you can also create them manually any time from the dashboard. Restores are available from the same interface.

[37:01] — Advanced Topics

Robert: There are a few other things you can do with Anywhere environments that I’ll touch on briefly.

First, private connectivity: Altinity.Cloud supports VPC endpoints. You can enable them by contacting us for support; this feature isn’t yet available in the UI.

Second, running other applications in the same Kubernetes cluster: this is possible. We use two namespaces and cannot see or manipulate other namespaces. There are some configuration steps around taints and selectors to prevent resource allocation conflicts, which we can advise you on.

Third, disaster recovery: we’ve worked with at least one customer on a DR cluster. It involves some legwork but is absolutely possible, and we can advise on the best approach.

[39:02] — Architecture Under the Hood

Robert: Let me describe what’s running inside your Kubernetes cluster. There are two kinds of things: the ClickHouse clusters themselves plus ZooKeeper, which make up your analytics stack, and the Altinity system services that include the Altinity Connector you installed, an Edge Proxy that routes incoming traffic, Prometheus, and Grafana for monitoring.

The reason this matters: if you ever decide to disconnect from us or migrate away, everything related to your analytics stack other than the Edge Proxy is open source. You can basically disconnect us and your clusters continue to run. Moreover, if the connection fails for any reason, your apps are not affected at all. They do not depend on active communication with Altinity.Cloud Manager. If there’s a transient network failure, when it comes back you’ll just be able to see the clusters again. There’s no disruption to your application.

[40:08] — Vendor Unlock-In

Robert: Your data stays entirely in your own VPC, and we actually store very little information ourselves. We have some configuration information that you gave us at setup, but for most things we look inside ClickHouse itself or inside the Kubernetes cluster to find important information.

You can control what access we have. On any cluster dashboard there’s an “Altinity access” button where you can set no-access, which takes away the Altinity user entirely; read-only, where we can see system databases and so forth; or full access. If you pick full access this button turns red to indicate that Altinity personnel can reach your data. You can unplug us and have us go away, and your analytics will continue to work. This is what we call vendor unlock-in: the opposite of lock-in.

[42:49] — Enterprise Support

Robert: Enterprise support is part of every environment. That includes everything from schema design to integrations, bug fixes, emergency response, and random questions. We currently have two ways of reaching support: Slack and Zendesk. We’re considering collapsing those into one channel.

If you’re running Anywhere, we will of course help you with Kubernetes problems too. Anything relevant to running Altinity.Cloud Anywhere, whether it’s a Kubernetes issue or a ClickHouse issue, we’ll help diagnose and fix it. We manage a couple hundred clusters entirely based on Kubernetes, so we know this environment well.

A few tips to make support work better for you: talk to us before you have a problem. We can be very proactive because we have the management connection. For example, if you’re inserting data in blocks that are too small we can see that and give you a heads-up. If you have big upgrades planned or load profile changes coming up, tell us in advance so we can keep an eye out.

[44:53] — Pricing and Roadmap

Robert: You may be wondering about cost. Talk to us for specific numbers, but the key point is it’s based purely on compute. We do not charge for data or anything else you’re doing inside the Kubernetes cluster. If you’re running a small number of instances or have queries that run once a day, you pay for the resources necessary to run those queries and nothing else.

The Altinity Kubernetes Operator for ClickHouse can run ClickHouse anywhere that Kubernetes runs. EKS and GKE are the ones we’re most familiar with, but we’re actively adding Azure, DigitalOcean, and Red Hat OpenShift. If you’re not a Kubernetes person, we’re also developing the ability to fully provision EKS for you: all you have to do is give us an IAM role in a VPC and we’ll take care of everything, including Kubernetes itself.

[47:26] — Call to Action and Close

Robert: If you like what you heard, come and start a trial. It’s simple to sign up and it costs nothing for the first two weeks. It gives you access not just to the software but to the services as well, so you can get a taste of what support is like.

I want to thank you all for attending, and I apologize for the small hiccup at the start of the demo. We’ll go back and figure out what happened there.

Alexander, you’ve done something quite amazing here, and I’ve been working with it for a few years now. This is the first iteration that works as well as this does. I hope everybody will try it out and have that same experience.

Thank you all for attending. Have a great day.

FAQ

What is Altinity.Cloud Anywhere and how is it different from standard Altinity.Cloud?

Standard Altinity.Cloud runs ClickHouse clusters in Altinity’s own VPCs on Amazon or GCP. Altinity.Cloud Anywhere uses exactly the same management software and cloud interface but runs the clusters inside the customer’s own Kubernetes environment. The customer retains full ownership of all compute, storage, and networking resources, while Altinity provides the management, monitoring, and enterprise support through a secure outbound connector. Data never leaves the customer’s VPC.

How does the Altinity Connector work and what access does it give Altinity?

The Altinity Connector is a pod that runs in your Kubernetes cluster and establishes a secure HTTPS outbound connection to Altinity.Cloud. There are no inbound ports required. This connection establishes a bidirectional management channel that allows Altinity to provision and manage clusters, read monitoring data, and respond to support requests. It does not move data. If the connection drops, your ClickHouse clusters and all applications that query them continue to run normally; there is no dependency on the active management connection.

What are the required steps to set up Altinity.Cloud Anywhere on Amazon EKS?

There are three steps. First, prepare a Kubernetes cluster: provision an EKS cluster using eksctl, install the EBS CSI driver and correct storage class, and add an autoscaler such as Karpenter to handle automatic VM allocation and deallocation. Second, connect to Altinity.Cloud: install the altinity-cloud-connect utility, authenticate, generate a setup script that creates the required service accounts and namespaces, and apply it to the cluster with kubectl. Third, configure resources: specify which node types Altinity is allowed to use, set up an S3 bucket for backups, and configure a backup schedule.

What happens if I want to stop using Altinity and manage ClickHouse myself?

You can disconnect Altinity.Cloud at any time. Because all analytics stack components, including the Altinity Kubernetes Operator for ClickHouse, Prometheus, and Grafana, are open source, your ClickHouse clusters continue to run entirely on open-source software after the management connection is removed. Your data remains in your VPC and your applications continue to function without interruption. Altinity calls this “vendor unlock-in” rather than vendor lock-in.

Can Altinity.Cloud Anywhere run ClickHouse outside of AWS and GCP?

At the time of this webinar the supported distributions are Amazon EKS and Google GKE, with Minikube available for testing and development. The session also demonstrates running it on a physical machine in a home lab. Active work was underway to certify Azure AKS, DigitalOcean, and Red Hat OpenShift. Additionally, Altinity was developing the capability to fully provision EKS on behalf of the customer, so that the only requirement is an AWS IAM role and a VPC.

How is Altinity.Cloud Anywhere priced?

Pricing is based purely on compute resources used. There are no charges for data stored, data transferred within the cluster, or any other activity inside the Kubernetes cluster. If you run a small cluster or have infrequent query patterns, you pay only for the compute resources those patterns require.

© 2023 Altinity, Inc. All rights reserved. Altinity®, Altinity.Cloud®, and Altinity Stable® are registered trademarks of Altinity, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc. Altinity is not affiliated with or associated with ClickHouse, Inc. Kubernetes, MySQL, and PostgreSQL are trademarks and property of their respective owners.