Observability on ClickHouse®: What Breaks at Scale and How to Model Around It

Recorded: July 23 @ 09:00 am PDT
Presenters: Josh Lee (Altinity) and Prathamesh Sonpatki (Last9)

Every ClickHouse schema works fine at 10GB. Most fall apart somewhere between 1TB and 100TB, and it’s rarely for the reason you’d guess.

Last9 runs its entire observability platform (logs, metrics, and traces, all on one shared ClickHouse layer) at real production scale: 480M log lines a minute, 20M+ unique label combinations per metric, sub-second aggregations over unsampled data. Getting there meant hitting the failure modes everyone eventually hits: high-cardinality columns that break naive indexing strategies, retention and tiering schemes that either blow up storage costs or blow up query latency, and control-plane decisions about what to filter before data ever reaches ClickHouse.

Watch this webinar, and let Altinity and Last9 guide you through the actual schema and architecture decisions behind a unified telemetry platform at scale: ORDER BY and cardinality tradeoffs, when to filter/reshape data before ingestion vs. after, tiered retention design (hot/warm/cold), and where “it’s fine in a demo” modeling choices turn into production incidents.

This session brings together Josh Lee, Open Source Advocate @ Altinity, and Prathamesh Sonpatki, Field CTO and Observability for AI-Native Teams | Developer Evangelist at Last9.

Explore more:
▪ Handling high cardinality in ClickHouse at scale (Last9)
▪ How Last9 and Altinity work together on ClickHouse observability
▪ Schedule a demo with Last9
▪ Useful Observability Queries eBook

Key Moments (Timestamps)

Key moments generated with AI assistance.

• 0:01 – Welcome and why observability data keeps growing
• 1:26 – Designing an observability product in 2026
• 2:50 – Speaker introductions and agenda
• 4:27 – OpenTelemetry, ClickHouse, and the cost tradeoff
• 8:38 – The six stages, from instrumentation to agents
• 14:14 – Shared responsibility and why ClickHouse fits
• 18:52 – Exporters, gateway collectors, and batching
• 21:59 – Sampling versus cutting noise at the gateway
• 23:24 – On-the-ground challenges for a vendor
• 30:03 – Multi-tenancy tradeoffs and single-tenant storage
• 36:07 – The Last9 telemetry data platform architecture
• 42:01 – Merges, materialized views, batch size, and buffers
• 49:07 – Why bring your own cloud
• 51:25 – Live demo: control plane and AI assistant

Webinar Transcript

0:01 – Welcome and the Growth of Observability Data

Josh: All right, hello everybody, and welcome to our July Altinity webinar. I am joined today by Prathamesh from Last9. Last9, if you haven’t heard of it, is a pretty awesome observability platform that we’re going to talk about a bit more today. The overarching topic, of course, is observability on ClickHouse. You’ve probably seen this. Even on Reddit, it’s gotten to the point where this is the butt end of a joke, like, oh, it’s another observability platform using OpenTelemetry and ClickHouse. You’ve all been hearing a lot about this, and we’re going to talk about how you can actually do this for real at production scale, not as a toy demo, which Last9 has a lot of experience running, using ClickHouse backed by Altinity.Cloud.

So the problem really is driven by the size of the data. This is the root of the observability challenge today, and it’s being driven by multiple compounding factors. First, we have more people than ever before deploying workloads. Those people are deploying more workloads than ever before, and those workloads are generating more telemetry than ever before. So even all of these things, which are themselves potentially nonlinear, compound into very nonlinear growth in the size of observability data sets, both for individual organizations and especially for observability vendors trying to deal with multiple organizations. So what does that mean for you?

Prathamesh: Observability in 2026, and how a vendor should design an observability product, is something I’ll talk about today based on our experience with Last9. Typically, when you talk about observability products, you talk about logs, metrics, and traces, but as Josh rightly said, it is all collapsing, because at the end of the day it is all data and you don’t know who is generating it. It has become very easy to generate telemetry data with OpenTelemetry, all the auto-instrumentations, and even eBPF. It has advanced to a stage where generating telemetry is no longer a problem. Containing that telemetry is a bigger problem that we are all running into.

User experience expectations have not reduced at all. People want the experience they’re used to, that they are already using. They don’t want to change their workflows, which also makes it very difficult to build an observability product. At the same time, agents are also here. It’s no longer just humans who are going to access the data; agents are also going to access the data. So you also have to worry about how the agent experience is going to be, not just the developer experience. These are some of the observations and tradeoffs we have to consider when building an observability product in 2026, and we’ll of course talk through them today.

2:50 – Speaker Introductions and Agenda

Josh: Quick note from our sponsors. As I mentioned, I’m Josh. I work at Altinity. We do hosting and support for ClickHouse, and as you’re going to hear, we also do it in your own cloud. We offer our enterprise support. We are contributors to ClickHouse, the open source library, but importantly, we are not affiliated with ClickHouse Incorporated, the company.

Prathamesh: My name is Prathamesh. I am from Last9. We are an observability platform not just for humans but also for agents. Developers, AI agents, all can use Last9. We use ClickHouse a lot, and we also use Altinity. We’ll talk about some of the challenges and how we use all of this today.

Josh: Quick agenda. We’re going to talk about what it looks like to architect an observability platform that can actually handle the challenges we’ve been mentioning. We’re going to talk about why ClickHouse is an awesome solution for dealing with those challenges. And then, as awesome as ClickHouse is, it’s not a magic bullet, so we’ll cover how Last9 does it at scale and deals with some of the challenges that are still present. Then we’re going to talk about why you can run both of these offerings together in your own environment. Why would you want to do that? Why is that important? Then we finally have the best part: the actual demos. We’re going to click around and see how this all works together, and then close off with some fun but horrifying tales from production. So, architecting an observability platform. Prathamesh, do you want to tell us about this?

4:27 – Architecting an Observability Platform in 2026

Prathamesh: Definitely. Typically, when you’re working on an observability platform in 2026, you will always hear two names. One is OpenTelemetry, and the second is ClickHouse, because you’ll see a lot of vendors and a lot of people building observability solutions on these two tools, because they have made it very easy to get started. OpenTelemetry has made it easy to get started from an instrumentation perspective, and ClickHouse has made it easy for us to dump any and all data into it, so we can use it for different use cases: logs, metrics, traces, and even profiles, as some people have attempted. So it has become easy to get started with these two tools.

But observability is not just the storage layer or the instrumentation layer. Observability also includes a lot of other things that are important when architecting a platform. What has not changed is that there are three things that are always in tension: cost, performance, and the tradeoffs between them. Those are very important when architecting the platform. Scale is another challenge. We have seen people pumping massive amounts of data into platforms like ClickHouse, and it becomes very important to design a system around ClickHouse, or around OpenTelemetry, that works seamlessly, rather than just focusing on the storage layer. Because you also have to worry about how people are going to experience the data, how they’re going to consume it, in which formats and protocols, how the data is going to be stored, for how long, and what the access patterns are, who is going to access it. So you have to consider all of these things when architecting an observability platform today.

Josh: And, as with anything, I think it comes down to tradeoffs. Go ahead.

Prathamesh: Cost is almost always the important tradeoff. You don’t want to spend too much on your observability platform. Budgets are not increasing. In fact, a lot of times these days, the AI budget is eating everything. The token cost is so high that people are optimizing everything else around it, and it is the biggest contributor to overall engineering cost these days. So cost is definitely one of the very important aspects, and any potential buyer considers the cost of an observability platform, not just usage but also migration experience and so on.

Performance remains key. With all the data and all the agents and humans accessing it, it becomes paramount that the platform is designed to be performant for different use cases, different access patterns, and the UX as well. UX as well as AX, the agent experience, becomes an important factor. So I like to consider it almost like an area of influence, a circle where all three of these things are pulling the equilibrium point toward each other, and you’re designing a system where you want to maintain the equilibrium at the center. You don’t want to go too high on cost, too low on performance, or too poor on user experience. You have to figure out a cost-effective but performant setup with a good user experience for the users. That remains a key design principle for all the choices you have to make to design an observability product. So this is like a triangle that you always have to think about when you’re designing and architecting an observability platform.

8:38 – The Six Stages, From Instrumentation to Agents

Josh: All right. So what are the components?

Prathamesh: Typically, when we talk about observability, we only talk about the storage layer or the query layer. But it actually starts with instrumentation. Instrumentation is where all the data gets collected and generated. One of the challenges OpenTelemetry has solved is the standard for instrumentation. But we also have a lot of instrumentation outside of OpenTelemetry. As a case in point, most of the time people have traditional logging; they don’t use OTel logging, because changing logging is an extremely hard job for any engineering team, also because we are used to our logging patterns. But at the same time, now you have to manage traditional logging alongside traces coming from OTel and metrics coming from Prometheus or something like that.

The other challenges around instrumentation are also about SDKs. The OTel SDK keeps getting updated. It’s not expected that everybody will always be on the latest versions of collectors and SDKs. So it becomes extremely important to design a platform that handles all of these cases, so the user experience is not impacted.

Ingestion is critical because of the nature of instrumentation, where data has different shape, volume, variety, and velocity. It becomes very important to shape the data at the ingestion layer. What I mean by shaping is, for example, if you have logs, you can extract relevant standard attributes at ingestion time, which can act like caching. When you actually query the data, the query doesn’t have to go over raw data, because you can extract and remap certain attributes. We’ll see some examples of that today.

Storage again becomes critical: how you architect the storage, what your hot and cold storage patterns are, how you retain data for longer durations. Are you creating materialized views, rollups, or precomputed metrics to manage access to the data depending on the use case? The query layer is what we already know: different patterns of access. Do you want to access data using SQL or LogQL? It all becomes important, because the expectation of users is also, hey, we don’t want to change our workloads. I’m used to Grafana, let me use Grafana. It doesn’t matter what the backend is. I’m used to my dashboards, let me see the same dashboards. Or if I’m doing a migration, I want to see the exact same names of the dashboards and alerts I’m used to in my existing platform. Alerting again becomes very critical.

What changed in the last six months, or even slightly before that, is how agents are dominating everything and have become a central theme for every observability vendor, because everybody wants to build their own AI solutions. In my experience, the trend has been that people want to build their own agents and sub-agents, or different types of organizational agents, to access the telemetry data and combine it with other data sources, which means I have to make data available for agents too. So observability is not just storage with a query layer on top to access the data. It is a combination of all these six things, and because of that, it becomes very important to design a system that works together rather than in isolation.

If you are instrumenting too much data, then you have to build an ingestion layer that can handle that much data, that can handle spikes, that can handle sparse traffic if it comes. You have to design the storage layer connected to that ingestion layer, so data can be correlated and stored in an optimized way. You have to build a query layer on top of that storage layer, so the query utilizes all the optimization you’ve done at the ingestion and storage layers. So it’s the entire system working together as a team; that is what observability is. It is not one part in isolation. That also drives your design decisions, because the tradeoffs you make at the ingestion layer can help you reduce cost at the query layer. For example, if you’re building aggregations or rollups at the ingestion layer, that helps reduce query cost, because you don’t have to heavily invest in the compute required for very high-load queries. Any optimization you maintain at the ingestion layer for handling data spikes, or creating buffered views so the pipeline ingests the data, helps you handle spikes that can go from, let’s say, 100K to 300 million, which we’ve actually seen in production, where one customer jumped from 80K to 350 million in a couple of minutes, and the system had to scale for that traffic to handle ingestion, storage, and everything.

So observability is not one part in isolation; it is a system that works together, and these are the six stages. Agentic access is almost like the sixth; typically you can combine it with the query layer, it’s just that instead of humans, the agents are querying it. But five stages are very critical: instrumentation, ingestion, storage, query, and alerting.

14:14 – Shared Responsibility and Why ClickHouse Fits

Josh: And you had those first two in green, which was interesting, because they’re kind of… OpenTelemetry enters the chat.

Prathamesh: Yes, and it changes everything.

Josh: We’re both huge fans of OpenTelemetry, of course. It does change everything for both the consumers of observability and the vendors, because it changes the responsibility model. We have this new responsibility model where we’re sharing the responsibility for telemetry quality and the generation of the telemetry. And because that generation is shared, the responsibility for the volume is now shared, and the volume is continuing to grow nonlinearly, as we mentioned. So everybody has to deal with that on both sides of this. Oh, ClickHouse enters the chat. So ClickHouse kind of solves the data problem. You can dump anything and everything at ClickHouse. How?

Prathamesh: That’s a great question. I think a lot of our audience is already aware of ClickHouse and has used it, so I’ll not go into too much technical detail about what ClickHouse is or how its architecture is different. But one of the key advantages of ClickHouse, especially for observability use cases, matches the data model we have today, where you can generate data from any source, and that data can have any shape: log, metric, trace, profile, it doesn’t matter. You can dump all of that into something like ClickHouse without worrying about how to ingest it, because of course there are collectors and ingestors already available. So it allows me to literally dump anything and everything into ClickHouse storage.

SQL is a very standard, uniform way of accessing data, which a lot of us are already used to from other tools. And because it is SQL, I can also build a query layer on top of it where I can control the user experience. I can give people flexible ways of accessing that data in other query languages they’re used to as well. So there’s a standard that allows me to build solutions on top of it from a single unified data source. And of course, it has proven at scale for very large volumes, so I don’t have to worry about, hey, my labels are increasing. I want to track each and every customer of mine so I can understand the API performance for every customer; I can just create a field for it and design the schema in a way that it can handle that kind of cardinality, or whatever label dimensions I have. That makes it extremely viable and useful to use ClickHouse for observability use cases.

Of course, there are some challenges to using ClickHouse for observability use cases that you have to circumvent or build solutions around. If you’re used to an Elasticsearch-style full-text search experience, you’ll find some challenges, although there is now full-text search available in recent ClickHouse versions as well. Users migrating from an existing ELK stack will ask how you’re doing full-text search, or how they should search on the body. That’s where the system you build around ClickHouse helps you navigate, because you can build some indexes and use existing ClickHouse features that are not really full-text search but allow you to do fast searches on raw strings in a text body. So while that challenge is there, ClickHouse does allow me to dump a huge amount of data in a predictable way, which makes it very viable and useful for observability use cases.

18:24 – Nuts and Bolts: Exporters, Gateways, and Sampling

Josh: Especially that point about cardinality. To me, cardinality is one of those things where, if you don’t have an engineering solution for it, it takes up a disproportionate amount of your mental load whenever you’re doing anything else. You don’t want to be spending that much time thinking about it if you can have an engineering solution that solves it for you. So, nuts and bolts.

Prathamesh: Together, ClickHouse and OpenTelemetry are meant for each other, because there are open source exporters in the OpenTelemetry world which are available for ClickHouse. You can just plug it into your existing OTel collector and send data to ClickHouse very easily. There is a Grafana data source available for ClickHouse if you want to access the SQL data directly. There are default schemas out there too. Even if you just download the default schema and start using it, you can build a quick observability solution within half an hour if needed, at least as a starting point. Of course, scaling comes into play after that. It also allows you to do isolation in terms of retry traffic, which becomes very important when you’re handling scale, because you can decide the resources for each layer depending on how much traffic you’re expecting. So OpenTelemetry and ClickHouse have good support for each other, which makes it easy to use something like OTel to send data to ClickHouse.

I should also call out that a lot of the data we see, at Last9 as well, is not OTel. While OTel is mainstream and has auto-instrumentation for multiple components, even serverless ones, there is a huge amount of data that is not OTel. But you can dump that into the same pipeline by designing solutions around it. You can get data from, say, Cloudflare logs; OTel has a receiver for that. If you want to ingest logs from your Kubernetes clusters or Docker containers, there are receivers for getting those file logs and dumping them into OpenTelemetry. People have built adapters for transforming one line format into another, where, let’s say, you have Fluent Bit that can be converted into OTel as a plugin and then ingested in OTel format. So a lot of data is not OTel, and you can still dump it into ClickHouse, which again proves the earlier point about consolidating all the data together and then making sense of it, querying it to understand what your system is doing. But OTel and ClickHouse have good support for each other, which makes it extremely easy to unify different telemetry data together.

Josh: A common pattern I’m seeing here, and this is important if you’re using this exporter, is to have a collector deployment running somewhere as a gateway to the data store. If you have DaemonSet collectors deployed on your cluster, don’t have each one of those connect to the ClickHouse cluster, because, as we’re going to talk about, batching is really important when you’re inserting into ClickHouse. So you want a centralized point to control that batching.

Prathamesh: Another interesting point about the gateway collector is sampling decisions. We always hear that you should do sampling to reduce the data, but if your collectors are not connected in a way that they have all the trace data to make the sampling decision, it is almost always going to do the wrong kind of sampling, where you’ll miss out on data. So our recommendation to a lot of people is that instead of sampling, you can cut down the noise by removing unwanted spans and internal spans, because there’s a lot of noise that these Java and .NET SDKs create for internal spans. That plays an important role too. I’ll cover that when I talk about the architecture of Last9, but I wanted to mention it, because that’s where treating it as an entire system comes into play, rather than just treating OTel and ClickHouse together and expecting them to do the magic and everything gets solved.

Josh: Sampling, just like cardinality, is another one of those things you don’t want to spend a disproportionate amount of effort on if you don’t have to.

23:24 – On-the-Ground Challenges for a Vendor

Josh: So let’s talk about Last9 architecture and how you specifically address some of these challenges.

Prathamesh: I’ll talk about how Last9 tries to address a lot of these challenges, and we’ll also see what people have today. There are on-the-ground challenges we see as an observability vendor. Typically, as I said, people have some OTel, but they also have a lot of non-OTel data by default, and it is very difficult to migrate it on day zero. Of course, there are efforts and initiatives to convert it into standard data, but it takes an immense amount of effort and time for developers and engineers to stop all the other work they’re doing and allocate a lot of time to this. So instrumentation like eBPF becomes very important for a lot of these things. Lack of structured data at instrumentation time also results in slow queries and performance impact, or correlation challenges when you query the data. I’ll talk about how we address that as well.

A lot of times people come from multiple tools: logs are in ELK, metrics are in Prometheus, for APM they’re using some vendor. So there are already data silos, which is an opportunity for a unified solution, because when they see the data and can jump from logs to traces, then see their metrics and quickly set an alert, that becomes a great user experience for them, because they’re not used to that kind of experience. So unification helps, because you’re able to show the power of all of this put together, where you can correlate and jump between the data points.

There are always some business-specific needs. It is not just that you do auto-instrumentation and everything is done. You want to have some SLOs. There are business metrics you want to capture. Logs have a lot of business context, and it’s very hard to do metric instrumentation or OTel instrumentation for that. Logs have a lot of relevant context, and you need a way to extract that information for business alerts and other use cases. Managing alerts is again a big challenge. There have been a lot of efforts with agents and models to solve that, but it still remains a key challenge, because setting thresholds and managing them is just a nightmare. Nobody wants to keep doing that all the time.

All of this results in a lot of data, but it’s harder to get insights from it. This is amplified now with AI, because even if you have AI and a lot of data and means to query that raw data, converting that raw data into aggregations and then deriving insights also means you have to spend a lot of tokens, and those tokens can be very expensive, which we’ve seen in the wild, people spending huge amounts of tokens on all this telemetry data. So while people want to build AI solutions or agents for telemetry and observability use cases, it almost always needs an agent cache, a thing that sits on the raw telemetry data but gives the agent structured, correlated information they can use for actual investigations. So that becomes important for using all of this data with AI as well.

Legacy code is a very common challenge. It’s very difficult to migrate those systems, whatever stack they’re in. They can use older SDKs, so you almost always have to build a system that compensates for the versions across different releases of OTel, and makes sure the system manages all these naming conventions, because OTel keeps changing naming conventions. Earlier it was http.status.code; now it is http.status_code. And now all the default dashboards you’ve built are broken, because people updated to the new SDK and the keys changed. So you almost always have to keep track of all these instrumentation libraries. One interesting point about why this was not a problem before OTel is that earlier vendors used to have their own agents for instrumentation, like a Datadog agent or a New Relic agent, so the experience was managed end to end. But now, if I’m building something on top of OTel, I have to build an experience that compensates and solves for different types of SDK versions, because I can get data from any source. It doesn’t matter where I get the data from; it is OTel standard. Anybody can build their own SDK and their own layer and send me the data. So it forces me to rethink how I want to build a platform to solve for these challenges. These are some of the extremely key challenges we see on the ground, which makes it very interesting to solve these problems.

Josh: Yes, interesting is the right word.

Prathamesh: Whenever people want to try out an observability product like Last9, unification is very appealing because of the existing data silos and the promise of seeing all data together, even though it’s not always needed. Just the single source of truth in terms of unification makes it very important. Unified telemetry also allows us to build centralized control over the telemetry, because everything is getting ingested into one place. I can connect the data together, build correlations at ingest time, and use that later for the query, dashboards, and all the other experiences built on top of it, which also helps reduce raw data or noise and give key signals to customers. So these are some of the things customers need. Even though we have OTel and ClickHouse, we still have to solve these on-the-ground challenges, and that’s what we’re trying to solve for.

30:03 – Multi-Tenancy Tradeoffs and Single-Tenant Storage

Josh: These challenges, you deal with all of this as a single organization, and then as an observability vendor you have to deal with this for multiple organizations, which presents a bunch of new challenges. Like you mentioned, the one customer who had that spike from, what was it, 80,000 to 300 million or so.

Prathamesh: Yeah, that’s one customer, and that can impact your other customers, in addition to being a challenge for that one customer’s environment.

Josh: So multi-tenancy definitely creates some challenges. But we’ve also been talking a lot about a single unified source and centralization and the benefits of that. So there are definitely some tradeoffs here. The central one is cost versus quality of service. You can basically not do multi-tenancy, which is the choice we’ve made within Altinity. So whether you’re on our managed SaaS or our BYOC, every ClickHouse cluster is its own Kubernetes cluster. We made that decision because, at the database layer, we decided as a company that the second worst thing that could happen is that we lose data. But the worst thing that could happen is that we show someone someone else’s data. So we don’t want to do that. We also want to guarantee quality of service, because the types of production workloads our customers run on their ClickHouse clusters need that kind of control. You’re one of our customers; you need that quality of service, so you can pass it on to your customers.

But that’s not always the right tradeoff. Sometimes, if you want to really quickly cold-start an environment, multi-tenancy can create a lot of challenges. It’s also expensive. Creating small developer and sandbox environments gets really expensive when you don’t have architecture support for multi-tenancy.

Prathamesh: One of the other important aspects of multi-tenancy is the architecture in terms of compute and storage, because what you need to think about is what layer of your stack or platform you want to be multi-tenant. There is almost always a gateway or something that can be common, a very lightweight gateway that can fan out to the actual individual deployments per tenant. But it also means there’s ease of use in terms of maintenance and entry point to your architecture, but beyond that point you can distribute the system so that every tenant gets isolated compute and isolated storage.

It also allows you to build. For example, what we did was create a proxy API layer for accessing the data, so nobody could actually connect to ClickHouse directly. That was one of the decisions we took early. There were reasons, pros and cons, and tradeoffs. One reason for doing this was to manage the experience in terms of the output of the data. We were able to optimize the queries so we could give a better user experience because of that query API. We were able to cache a few things, manage the attributes we query and how we query, and circumvent some of the full-text search problems we discussed earlier. That allowed us to build that layer on top, and it also allowed us to support multiple query backends; LogQL was possible, even a proxy SQL layer was possible at times.

So while multi-tenancy is an important aspect for any team serving multiple customers, there are ways of architecting it so that it doesn’t really remain multi-tenant. You can leverage the usefulness of single-tenant architecture, and that’s where our experience with Altinity came in handy, because we were able to use the single-tenant architecture for the actual storage layer with Altinity, which solved a lot of problems for us in terms of the quality of service and cost that you mentioned. It’s always interesting how you want to look at the multi-tenancy problem, because it not only affects the user experience, it also affects how you onboard customers and how you deploy. For example, if I have hundreds of deployments for hundreds of customers, then I have to design a system for deployment itself. Any new change I want to roll out, let’s say I added a new query function optimization in my stack, rolling it out to 100 customers automatically without any glitch is itself a problem statement your engineering team has to address from the ground up.

So the tradeoffs, or the impact of these decisions, can be widespread. They can impact how the rollouts happen, how fast your releases reach customers, and how fast you can upgrade your software. Sometimes they can even impact whether you can touch the software or not, because sometimes you can’t even look at or touch a piece of software. So multi-tenancy plays an important role. What we’ve realized over time is that there are components we want to have single-tenancy for, and there are components we’re okay having a multi-tenant compute or intelligence layer for. With Altinity, we were able to move the ClickHouse layer to a single-tenancy setup and serve the customers better.

Josh: That’s a great way to navigate all of those challenges: building tools, managing the pipeline that produces the outputs instead of managing the output. Okay, speaking of pipelines.

36:07 – The Last9 Telemetry Data Platform Architecture

Prathamesh: This is where all of these things we spoke about come together to solve the challenges we discussed. This is the Last9 telemetry data platform high-level architecture and how it works. I’ll cover some parts of it in the demo as well. Essentially, the telemetry data platform has a data platform where you can dump any data, but it also has a control plane, an intelligence layer, where you can decide what to keep and what to drop. You can shape the data. You can enrich and transform the data before it gets stored, so that when you query it, it is already correlated and already has accelerated attributes for you to query. It allows you to transform the telemetry as well. If you have a legacy service that only has logs, but for alerting, dashboards, or other use cases you want to convert those logs to metrics, then that also becomes solvable through these control plane use cases, from programmatic access as well, either via the AI experience or via Terraform and IaC. So you can control it in different ways. Then there is a data platform layer that stores all the data, and a query engine on top of it that allows you to query that data effectively.

We have gone with a decision to build a query layer on top of SQL, so people don’t have to directly query the raw data. There are pros and cons. The pros are that we can control the user experience better and give capabilities rather than raw data for a lot of use cases. For example, querying the Apdex score of a particular service rather than querying latency and then calculating it on the fly. These are some of the tradeoffs that help you navigate the user experience. Of course, people can query the raw data as well when they want. But you can build actions on top of it that help everybody, not just humans but also agents, understand the system.

It also allows us to create data. I would not say just materialized views, but we also use some other tricks, like creating fast metrics from certain logs and traces, so there can be different consumers of it from a logs and traces perspective. I’ll also talk about some pitfalls of materialized views that we’ve seen at scale, because one of the biggest problems in designing such a system is that all of this must work at scale. We have customers who send 300 million or 480 million telemetry data points per minute, and 50TB of data on a business-as-usual day. But during the IPL, which is one of the biggest sporting events in the world, cricket being a very popular sport in India, we have seen people sending us 120TB of data per day, and expecting that data to be queryable immediately after, for debugging incidents and finding transactions. So that also requires designing this in a way that works at scale. We have some experience and challenges with ClickHouse that we’ll also talk about. But essentially, this is the architecture where the data platform has a control plane that allows you to shape the data, so you can standardize, extract, and enrich it, drop everything you don’t want, keep only what you want, and then query and access that data in multiple ways.

This is also one of the very important design principles we have, because one of the principles in building a product is about accepting where users are coming from. This is very important, because not all users are used to your way of thinking. They can come from different backgrounds and different existing experiences. They may not have the liberty or leverage to do a lot of the things you’re asking them to do. So it’s almost like building an observability product for where you are as a customer, rather than, hey, come to me and I’ll show you the light in terms of how things should be done. That is one of the design principles we have, which also means it should be available for multiple data patterns. If somebody wants to use Grafana, LogQL, PromQL, or TraceQL with it, so be it. We’ve built a query layer on top that allows it, though it will have some limitations compared to the canonical implementation of, say, LogQL. But that’s the tradeoff, going back to that cost, performance, and user experience balance; your user experience is not hampered, for the cost and performance you’re expecting. So observability for where you are is a very important design principle for us as a product, and it shapes a lot of the decisions we’ve made. This is one of them: flexibility of access. If people want to access data in different ways, it should be possible.

Josh: I couldn’t agree more. And I know PromQL especially is such a hot topic right now. I can’t wait for that to land fully supported in ClickHouse Core. Okay, let me know when you want me to hit the button.

42:01 – Merges, Materialized Views, Batch Size, and Buffers

Prathamesh: These are some of the challenges we’ve seen, especially at scale, and these are also challenges you have to architect the system for, not just use ClickHouse as-is. When there are massive spikes, the merges increase, especially if you’re using materialized views, because you can have merges for those materialized views and inserts happening concurrently with the main table. At that time, we’ve seen ingestion get lagged or impacted, because there are multiple queries running at the same time, which has caused problems for the health of the ClickHouse cluster. Some of these challenges are only encountered at scale; they may not show up at a normal scale.

This is one of the problems we’ve seen: materialized views, while they are great for solving some rollups and challenges around aggregating data or keeping a faster view of the raw data, can also cause problems if not designed properly. In the early design days, we designed a few materialized views that caused problems in terms of ingestion. What we came up with was a slightly different approach: we converted some of those materialized view use cases into metrics themselves, because we have the ETL pipeline for transforming the data as it is ingested. We started using some of that ETL pipeline for our internal use cases as well, for designing faster access to the raw data, and we made the decision to use metrics for it. That proved interesting, because it allowed us to use that data in multiple ways, since it was also accessible via PromQL. It opened up multiple access levels for that data, which was earlier not possible. So while we do use materialized views and they’re great, and there are some use cases we solve with them, we’ve seen them result in bottlenecks at large scale, for which we’ve shifted to faster metrics, which has been good for performance so far.

Josh: That makes a lot of sense.

Prathamesh: Batch size is a very common problem if you’re using ClickHouse. A lot of people in the audience today may have come across this. There are people who send one log line; this is an interesting production discussion I had with a couple of people recently. There was a person trying to insert one log line and expecting it to be available immediately on the platform. Yes, it was available immediately, that’s how it works when I’m running it on my laptop. But of course, you have batch settings in your ETL pipeline, and the advice is that you should use an asynchronous insert and do batches correctly, because doing one insert in ClickHouse is not good. The advice is to use asynchronous inserts and do batches correctly, so you can insert in an effective way without causing too many queries on the ClickHouse cluster.

So batch is one of those interesting settings you have to manage. This is also interesting because batch starts at the OTel collector layer, because whenever you have the ClickHouse exporter, you have to define the batch there, but it has an impact on your storage layer. And if you’re using the same thing for query, effectively, if you have a replica and use that for query, it has end-to-end impact. If you mess up the batch size, it can derail the entire pipeline. You can’t keep it too small, but you have to manage it so it becomes dynamically tunable to the traffic levels you’re seeing. So we developed an adaptive algorithm internally that tunes it depending on the traffic, based on the levels of ingestion. This is again a very common problem that a lot of people may have seen, and there are good blog posts and recommendations. I think the Altinity blog also has some good posts about how to manage it. But this is again an interesting problem we’ve seen.

Josh: I see we were getting some questions in chat. We’re running a little slow, so I don’t have time for questions, but we do have an office hours tomorrow. So folks, if you have questions, just join us for the office hours tomorrow. Prathamesh, maybe you can join us and answer any follow-up questions.

Prathamesh: For production workloads, adding a buffer like Redpanda or Kafka seems a very common architectural choice, but now you have two problems, because you have to manage the inflow and outflow from the buffer. That’s the biggest problem, because if your ingestion is spiky and you’re seeing massive variations in traffic, now you’ve introduced a middleman that’s going to manage the outflow to the ClickHouse cluster, or the inflow from the OTel collector layer or whatever gateway architecture you have. Now you have two problems to solve. While it has its own pros and cons, what we’ve seen is that it requires a certain scale to enable this. There can be cheaper alternatives to adding something like this to your stack, to avoid it without really needing it. So we do use a buffer in places, but we also have a dual architecture where we can avoid it dynamically.

Josh: Classic engineering tradeoff.

Prathamesh: Storage scaling is again time-consuming. It doesn’t happen instantaneously, if we expect it to, which also means the pipeline parts become important. It has to absorb the pressure. If there are spikes or weird traffic patterns, even if you have to scale at runtime, it is not going to be fast. A lot of times, for databases, you have to vertically scale, which takes time, which means everything before that has to be tuned so you can save that game, otherwise there is data loss. So that is something we designed a few things around, to solve that.

Josh: We talked about this one. Every time, it gives me shudders.

49:07 – Why Bring Your Own Cloud

Josh: Very quickly now, why bring your own cloud? For observability especially, I think it just makes sense. First of all, there’s no vendor lock-in. Observability and telemetry promise no vendor lock-in. I’ve given multiple talks about why that’s partially true, but not as true as the marketers would want you to believe. So having actually true open source storage combined with OpenTelemetry actually fulfills that promise, end to end, of no vendor lock-in.

The data in your own environment is really important for a number of reasons. One, just having the data close to where you need it and close to where it’s being generated can save money, because you’re not paying for cloud egress costs. So that’s a great benefit. But also, of course, you might have sovereignty requirements, or a reason that you want to keep this data in your own environment and never let anyone else have access to it, which you can certainly facilitate using Altinity Managed BYOC. And because it’s a managed service, you get all of the benefits of a managed service. You own all of your data, but you don’t have the responsibility for keeping the system up, healthy, and functional; we take responsibility for that. And lastly, the costs are predictable, because you only pay for the cluster you provision. So you don’t ever get charged with a surprise bill for a spike in your telemetry, because you’re paying for the ClickHouse cluster compute, not for the actual number of events you’re sending. It’s the same billing model for Last9. So if you combine the two tools, you have low, predictable costs, especially compared to some legacy vendors.

And then this is really cool: ClickHouse is not just an observability database. It’s a whole data warehouse or lakehouse now, with some of the features that have been added in core and also through Altinity Project Antalya. So it’s not just your observability database; you can use it as your whole lakehouse. And if you already have a lakehouse, if you already have ClickHouse, you’re already paying for it and managing it. Why not also use it for your observability data? That’s why I’m really excited that there is an end-to-end managed BYOC solution for observability with ClickHouse.

Josh: Do we have time? We have six minutes left. Do you want to do a two-minute demo?

51:25 – Live Demo: Control Plane, Correlation, and AI Assistant

Prathamesh: Yeah, I’ll cover a couple of examples that we discussed. Okay, I can share my screen. What I want to cover is a couple of examples of the control plane aspect we discussed. I’ll not waste a lot of time, but this is a log line that has a trace ID in it. You can see that trace ID here. Now, the instrumentation is done in a way that this is not an attribute, so the correlation is difficult. It is buried somewhere in the log line, and it doesn’t help me navigate to the trace. If it has a trace ID, it means it has an associated trace as well, but I can’t go to that, and when I go to the related traces, I will not see that individual trace, I’ll see everything. So I don’t have the ability to jump.

Now, what Last9 allows you to do is, at query time, you can define regex or JSON extraction rules so you can extract it. Now, if I click on view details here, I see the trace ID and the span ID, because it is extracted at query time, and I’m also able to jump through the trace. So the correlation problem is addressed even though my logging and tracing are disconnected from each other, and I’m able to connect it together at query time. But on top of that, what you can do in Last9, and this is the Last9 control plane, is define a remapping rule. This is a rule I just defined some time back, where the same regex is now added as a rule in the ETL pipeline. So while the data is getting ingested, these trace IDs and span IDs will be extracted as accelerated attributes. So now when I go and query the data, it’s the same service, but if I look at, say, the last 10 minutes, I’ll have the trace ID attribute directly here, and I don’t have to do the regex parsing at query time, because we’re able to do that at ingest time itself.

It also means that whoever is accessing this data already knows these are the standard attributes available; just use that to do the correlation. This is the agent cache example I was taking earlier, because this also helps agents query the data predictably, do the correlation, and solve it at ingest time rather than reinventing it at query time. So this is one example; there are a few others. One interesting thing this also helps with is, this is one of our AI assistant examples, but I can also generate a lot of these remap rules using AI, because it can access the data and understand the data patterns. In this case, for this AI agent service, it understood that the NC characters are being printed in the logs, and then it gave me an example of how I should update my OTel collector if I have one running. But even if I’m not using an OTel collector, I can add the same rule on the control plane itself, and it also allows you to quickly add the rule from the assistant itself. Or if you’re using MCP, then MCP also has that capability, so it connects things end to end. For example, an AI agent investigates that this time it used more tokens because the data was not standardized, but here is a recommendation to add this rule, and it will become standardized the next time you want to query it. So that closes the loop on the telemetry. A code change is not needed; you really don’t have to change your workflow. This is one example of what we discussed earlier about managing a few things at ingest time that can help you.

Josh: That’s really cool, because most vendors are like, trust me, when it comes to correlation, whereas you’re giving the customer levers to control it themselves, which is really cool. I think I’ve taken control back.

Okay, so we had a quick demo of Altinity BYOC, where your ClickHouse just works. We have tons of content online if you want to go on YouTube and find a demo. We do have an Altinity.Cloud manager that makes everything really nice and easy, giving you access to Grafana dashboards, backups, and things like that. But mostly, if you’re using it with Last9, your ClickHouse just works, so you don’t have to think about it. We don’t have time for tales from production, so maybe we’ll do a follow-up session, because these were some really fun but terrifying stories. We’ll do an observability Twilight Zone episode or something.

So here’s a QR code if you thought this was really cool and you want to go check out Last9. I did mention that we have office hours tomorrow where we’ll answer follow-up questions. I see several people asked questions in the chat and Q&A, so we’ll try to capture those and answer them in the office hours. Even if you’re not present, that recording will go up on YouTube, so hopefully you’ll get an answer to your question one way or another. And if all else fails, come join us in Slack and send me a DM, like, Josh, you never answered my question, come on man, and I will fix that. So that’s all. Prathamesh, thank you so much. It’s been an absolute pleasure doing this with you. Thank you.

Prathamesh: Thank you.

FAQ

Why do ClickHouse schemas that work at 10GB break at larger scale? A schema that performs well on a small dataset often runs into trouble somewhere between roughly 1TB and 100TB, usually for reasons that are not obvious. High-cardinality columns break naive indexing strategies, retention and tiering schemes can inflate either storage cost or query latency, and merges increase sharply during ingestion spikes. The fix is to design the whole system around ClickHouse rather than relying on it as-is.

Why use ClickHouse for observability data? ClickHouse lets you ingest logs, metrics, traces, and even profiles from almost any source into a single store, using SQL as a standard access layer. It has been proven at very large volumes, so it can absorb high-cardinality label dimensions when the schema is designed for it. There are open source OpenTelemetry exporters and default schemas that make it quick to stand up an initial observability stack.

What is the benefit of bring your own cloud (BYOC) for observability? BYOC keeps telemetry in your own environment, which avoids cloud egress costs and supports data sovereignty requirements. Because it is a managed service, you own the data while the provider keeps the system healthy. Costs are predictable because you pay for the provisioned ClickHouse cluster compute rather than per event, which contrasts with some legacy usage-based vendors.

How does Last9 handle correlation between logs and traces? Last9 uses a control plane that can extract fields such as trace IDs and span IDs using regex or JSON rules. This can be done at query time, or promoted into the ETL pipeline so the attributes are extracted at ingest time as accelerated attributes. That gives both humans and AI agents predictable, correlated data without changing application code.

When should you use asynchronous inserts instead of batching yourself? Inserting one row at a time is hard on a ClickHouse cluster, so data should arrive in batches. Asynchronous inserts let ClickHouse handle server-side batching, which helps when many clients send small inserts and client-side batching is not feasible. Batch size still needs to be tuned to traffic, because a batch that is too small can strain the pipeline end to end.

© 2026 Altinity, Inc. All rights reserved. Altinity®, Altinity.Cloud®, and Altinity Stable® are registered trademarks of Altinity, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc. Altinity is not affiliated with or associated with ClickHouse, Inc. Kubernetes, MySQL, and PostgreSQL are trademarks and property of their respective owners.