KubeCon Recap: Data on Kubernetes and Observability with ClickHouse®

KubeCon’s co-located events are my favorite part of the conference. This year’s Observability and Data on Kubernetes events were packed with practical talks with a focus on building the next generation of developer platforms on top of Kubernetes.

Observability Day & OpenTelemetry

Profiling, Auto-Instrumentation, and managing telemetry at scale were all major themes at this year’s Observability Day. OpenTelemetry is beginning to mature into it’s position as a universal standard. There were some great talks including: Éamon Ryan of Grafana on managing observability data volumes; “Distributed Tracing: All the Warning Signs Were Out There!!” by Vijay Samuel & Sandeep Raveesh from eBay; and from yours truly, “Where’s the Auto in Auto-Instrumentation?“

Data on Kubernetes Day

The co-located Data on Kubernetes event at KubeCon North America featured a half-day of talks focused on the intricacies of working with databases on Kubernetes. All of the talks are now available on YouTube and are worth checking out. These were my favorites:

Bridging the DevOps to Data Divide with a Common Cloud Native Stack – Elad Hirsch & Mey Beisaron

A great overview on how to run databases the cloud native way, including the CloudNativePG operator and the Atlas operator for schema management, with a golden path for monitoring.

Lightning Talk: Cloud Native PostgreSQL – Running PostgreSQL on Kubernetes – Peter Zaitsev

While the talk focuses on PostgreSQL, Peter shares some valuable history and lessons for anybody running or using a DBaaS in the modern landscape.
“At least by the star measure, CloudNativePG is kicking everybody’s ass.”

Lightning Talk: Kubernetes as Your DBA – Karen Jex, Crunchy Data

You wouldn’t think that a life-long DBA would love the ephemeral nature of Kubernetes, but Karen provides an excellent overview for how to work with Kubernetes (and not against it) for running highly available and performant databases.

Observability Hallway Track

I spent a lot of the conference exploring the latest and greatest innovations in observability. Many organizations are still dependent on metrics and non-centralized logs, but many are also seeing the value in independent observability platforms that remain available even when everything else is broken. As those organizations scale, ClickHouse becomes an attractive choice especially for logs and traces. (Which are really just a specific type of structured log, right?)

Last9

Last9 has already made a name for themselves with their impressive control plane and custom high-cardinality metrics pipeline. Now they have added logging capabilities as well as unsampled tracing. “How?” you might ask: with ClickHouse, of course. It’s the natural choice for anybody hoping to handle tracing at enterprise scale and without sampling.

SigNoz

SigNoz is a well known observability platform built on open standards. Of course, their logging and tracing is built on top of ClickHouse. Recently they’ve added a powerful interface for tracing through message queues like Kafka and AMQP using OpenTelemetry and the new OpAMP standard. This is an open protocol that allows OpenTelemetry Collectors (and other agents) to be configured directly from within your SigNoz Dashboard — completing the Observability feedback loop.

Dash0

Dash0 is an impressive new, OpenTelemetry-native observability platform that aims to always put relevant information right at your fingertips. They just launched v1.0 the week before KubeCon. Dash0 leverages ClickHouse to power on-the-fly querying across OpenTelemetry resources (entities).

HyperDX

Opinionated observability platforms can provide a lot of convenience, but if you’re looking for something more flexible, you may love HyperDX. It’s an open-source and full-featured telemetry explorer that allows for the correlation of metrics, traces, logs, and user sessions. It includes an SDK for popular application platforms and it is compatible with the most popular collection agents: Fluentd, OpenTelemetry, and Syslog. HyperDX’s powerful correlation and full-text features are, of course, powered by ClickHouse.

Quesma

Quesma offers a unique solution for organizations who are currently using the ELK stack for logging: An Elastic-compatible API for ClickHouse. This allows you to leverage the power and scalability of ClickHouse while combining the best of both Elastic and SQL-style queries.