Kubernetes Cluster Logging with ClickHouse® and OpenTelemetry

If you love ClickHouse®, Altinity can simplify your experience with our fully managed service in your cloud (BYOC) or our cloud and 24/7 expert support. Learn more.

In this post, we will put together a working demo with the OpenTelemetry Collector gathering all logs from a Kubernetes Cluster. Of course, we will need somewhere to put those logs — ClickHouse is great for this at any scale thanks to its powerful compression and deduplication capabilities.

The completed code for this tutorial is available on GitHub. To try the demo yourself you can simply clone the repository and run:

helm install cluster-logs-demo . --namespace=demo --create-namespace

The OpenTelemetry Collector is a powerful and flexible open source tool for gathering, processing, and forwarding telemetry. Read An Introduction to the OpenTelemetry Collector first if you’re unfamiliar.

What is ClickHouse?

ClickHouse is a massively-scalable SQL-compatible database that dominates in observability use cases through its ability to ingest millions of rows per second and to provide sub-second query times over petabytes of data. It uses the Apache 2.0 license.

Need help operating ClickHouse? Altinity® can provide expert support for open-source ClickHouse or run it for you in Altinity.Cloud.

The OpenTelemetry ClickHouse Exporter

The ClickHouse Exporter for the OpenTelemetry Collector can export all three signals — metrics, traces, and logs — to a ClickHouse cluster. As we saw in An Introduction to the OpenTelemetry Collector, this data can come from any of the available receivers.

To create some sample data we’ll configure an OpenTelemetry Collector daemonset with the filelog receiver, and we’ll use the ClickHouse exporter to send all of the logs to our ClickHouse cluster.

Getting a ClickHouse Instance

ClickHouse can run on anything from a Raspberry Pi to globe-spanning clusters. You can get started easily with Docker, or Helm/Kubernetes. You can also quickly spin up a production-optimized Kubernetes Cluster with ClickHouse on EKS using Altinity’s Terraform Module.

Configuring the ClickHouse Exporter

The ClickHouse Exporter is included in the contrib distribution of the Collector, so all we need to do is enable it and provide it with the connection settings for our ClickHouse cluster. In order to do this, we add the following to our Collector’s configuration.yaml (In the demo code this is provided via the helm values.yaml file):

exporters:
  clickhouse:
    endpoint: tcp://127.0.0.1:9000?dial_timeout=10s&compress=lz4
    database: otel
    async_insert: true
    create_schema: true

There are a few important options to note in the configuration snippet. 

• The additional performance of asynchronous inserts is typically desirable for observability data. Thus, we enable `async_insert`
• The `create_schema` option will allow the plugin to create the necessary tables in the target database if they do not exist, but it does require elevated permissions in ClickHouse and this option will prevent the entire Collector from starting up if the schema creation process fails for any reason. (With `create_schema` set to false the exporter will fail gracefully if ClickHouse is not available or the schema does not exist).

Important Notes for Replicated Clusters

Importantly, for any ClickHouse cluster with multiple replicas, we need to add a few more options for the schema creation:

cluster_name: my_cluster
table_engine:
  name: ReplicatedMergeTree

Configuring the OpenTelemetry Collector for Kubernetes Logging

Configuring the filelog receiver is quite straightforward thanks to a preset configuration in the OpenTelemetry Collector. Enabling this option automatically adds the filelog receiver to our receiver pipeline and adds it to our logs pipelines:

presets:
  logsCollection:
    enabled: true

Since we want this receiver to run on every node in the cluster, we’ll deploy this OpenTelemetry Collector as a `daemonset`. This will give us complete logging from our entire Kubernetes cluster.

Integrating ClickHouse with Grafana

Next, we’ll use Grafana to view and query our logs. You can read more about how to install and configure the Altinity Grafana Plugin for ClickHouse.

Need help with any of these steps? Join us in the Altinity DB Slack Community to get answers to your questions and talk with Altinity developers.

Viewing our Logs

Finally, we can create a logs panel to show some or all of the logs from our ClickHouse cluster. This will include a lot of noise, so we can use the OpenTelemetry Resource Attributes column to drill down into a specific entity of interest. 

For example, we can look for logs from a specific node in the cluster:

SELECT message FROM otel.otel_logs WHERE ResourceAttributes[‘k8s.node.name’] = … 

The Grafana panel above uses the OpenTelemetry `LogAttributes` column to show logs for a specific file:

SELECT Timestamp as "timestamp", Body as "body", SeverityText as "level", ResourceAttributes, ScopeName FROM "otel"."otel_logs" WHERE ( timestamp >= $__fromTime AND timestamp <= $__toTime ) AND ( LogAttributes['log.file.path'] IN (${file:sqlstring}) ) ORDER BY timestamp DESC LIMIT 2000

Next Steps

This is just one example of a pipeline that can be created using the OpenTelemetry connector. By combining the ClickHouse exporter (or other exporters) with the huge number of formats supported by the Collector, we can get our telemetry signals into one database no matter where they are coming from. 

In a future post we’ll dive deeper on working with logs as well as metrics and traces in ClickHouse.

Try It Yourself & Further Reading

If you’d like to try this logging example in your own cluster it is available as a Helm template on GitHub.

There’s also a great repository of docker compose recipes for ClickHouse if you’re just looking to do some local experimentation and development.

If you need to spin up a Kubernetes cluster to try it on, the Altinity EKS Terraform Module for ClickHouse is a painless way to deploy a cluster with ClickHouse included. Want to talk more about using ClickHouse for observability data? Join us in the AltinityDB Slack or schedule a free consultation.