25 Nov, 2019
The Altinity team just returned from sponsoring at KubeCon North America in San Diego November 19-21. I would like to thank all the people who visited our booth, especially current customers and ClickHouse users. It’s starting to be a real crowd!
Here’s a picture of my colleague Aurimas Jacikevicius manning our booth in the start-up pavilion. The cool-looking graph on the left is a Grafana dashboard showing airline data from ClickHouse. Altinity maintains the ClickHouse Grafana plugin. It’s great to be able to feature our recent work.
In the remainder of this article, I would like to share some impressions of the conference.
The prize goes to Sugu Sougoumarane, CTO of PlanetScale: “If you’re moving to Kubernetes, don’t leave your data behind.”
Sugu did a keynote presentation on Tuesday that featured this quote. He completely described how we feel about managing big data on Kubernetes. I really wish we had thought of it first.
Based on the number of times people asked us at the booth, it has to be: “Why data warehouse on Kubernetes?” “Because we’re at KubeCon” got a lot of laughs but there are two more serious reasons.
First, flexible coupling with applications. Applications are increasingly built around data, not the reverse. Kubernetes lets you create many data warehouses, each tuned to a specific use case. For instance, you can use the ClickHouse Operator to deploy a data warehouse per application. You can also have multiple copies of each data warehouse for prod, QA, and dev, with different classes of service. We sometimes describe this as data monoliths going cloud native.
Second, portability. Kubernetes runs just about anywhere. You can develop an application using minikube on a laptop, then deploy to KOPS running on Amazon with a few config file changes. As your usage scales or security needs change, you can move the application easily to an on-prem environment. It’s not just the application that is portable. Management, monitoring, and all your day 2 operations are portable as well. Users can deploy applications, including analytics, wherever it is most convenient.
We don’t just recommend these benefits to others. We use Kubernetes to build our own solutions for analyzing data and see the same benefits ourselves.
Being located in the start-up pavilion was really great. First, our booth was near the pool and ping-pong tables. Second, we had a ton of innovative start-ups as neighbors. Picking the best one was really tough. We also had to overcome some natural bias. We think Altinity has a great thing going with ClickHouse but in a spirit of modesty didn’t include ourselves. 🙂
With this in mind I saw two companies that particularly deserve a close look: Carbon Relay and Opsani. Both companies are focused on the problem of optimizing Kubernetes cost/performance, which is the bin-packing problem on steroids. Kubernetes, of course, enables you to pack a set of diverse applications onto a finite set of hardware resources in a portable way. But actually doing so in an optimal manner is extremely hard. We have seen this in our own work at Altinity.
Carbon Relay uses an ML-based approach that accumulates data from application tests, then computes optimal settings along the cost/performance curve. These include Kubernetes settings like memory size but also configuration settings for the app itself. Opsani does something similar for CI/CD pipelines. There were lots of other clever technologies at the show but these two companies are going after an economically important problem that affects the margins of thousands of SaaS businesses. They are asking the right questions. We’ll be looking very closely at the answers.
Follow-ups and More Blog Articles
KubeCon North America was great. We’re planning on sponsoring at KubeCon 2020 in Amsterdam and hope to see you there.
Meanwhile, we had a final inspiration during the conference. There is a huge number of services to manage logs in Kubernetes. It turns out that this is a favorite use case of ClickHouse users. We’ll be starting a series of blog articles devoted to building log management applications on ClickHouse. Stay tuned!