Cloud Native Rejekts 2024 Talk: Building an Open Source Observability Stack from Raw Telemetry

This session will guide you through the process of building a complete observability stack from the ground up using raw telemetry data ingested directly into a datastore, combining a variety of open source signals including OpenTelemetry and Prometheus formats.

We’ll start by leveraging our existing telemetry — likely logs and metrics — for maximum effectiveness — because telemetry without action is just storage. We’ll delve into the foundational aspects of creating OLAP cubes, essential for efficient data analysis and real-time insights. We’ll transform and enrich telemetry data to make it actionable — and show how to optimize storage and query performance to handle large-scale data with ease.

We’ll use this data for creating insightful visualizations with tools like Perses. I’ll show how to create well-formed time-series data even when the underlying data has gaps or varying granularity, and we’ll add robust alerting.

Drawing from experience with proprietary and open source observability tools, we’ll then evolve our monitoring by filling in instrumentation gaps and adding application telemetry. We’ll use automations like eBPF to fully observe a massively distributed cloud database offering.

This talk will equip you with the knowledge to implement a scalable, secure, and efficient open source observability stack tailored to your unique needs. Whether you’re starting from scratch or looking to enhance your existing observability infrastructure, join me to discover practical strategies and innovative solutions that you can implement today.

Open-Source & CNCF Projects Mentioned

Kubernetes, OpenTelemetry, Prometheus, Grafana, Tempo, Pyroscope, Perses, Kafka, CockroachDB, ElasticSearch, ClickHouse, DuckDB, qryn, Coroot, OpenTofu, Helm

Benefits to the Ecosystem

OpenTelemetry has led to another Cambrian explosion in observability tooling, which is exciting but also confusing for practitioners. Most SRE and observability leads that I speak with are in a similar position: They have a lot of metrics and logs, they have heard about OpenTelemetry, and they know that they should do something about it but they are not sure where to start.

This perfectly mirrors the current state of monitoring when I joined Altinity, where we produce a hosted ClickHouse offering with a cloud management portal. We have an abundance of Prometheus-style metrics about our infrastructure, and we have application and infrastructure logs in a full-text database.

Drawing from my own experience as a product manager at an observability vendor and combining it with the Altinty team’s expertise in managing massive data sets, we are on our own  journey to better leverage OpenTelemetry and distributed tracing for a more proactive approach to reliability.

This talk will draw from that experience and share concrete and actionable lessons for practitioners. Key points of the talk will include leveraging existing telemetry and using advanced database features to normalize and optimize telemetry.

Given our unique experience with ClickHouse, this talk will also provide ideas and insights for project contributors on how we ingest, store, and query massive data sets.

Finally, we’ll highlight the burgeoning open source all-in-one observability stacks that are appearing such as Coroot and Qyrn.