--- url: 'https://altinity.com/webinarspage/supercharging-observability-at-opsverse-using-clickhouse-real-time-analytics-joint-webinar-with-opsverse-and-altinity' title: 'Supercharging Observability at OpsVerse using ClickHouse® Real-Time Analytics | Joint-Webinar with OpsVerse and Altinity' author: name: Ryhan Sunny url: 'https://altinity.com/author/rsunny/' date: '2023-03-28T13:41:50-07:00' modified: '2026-06-03T07:23:39-07:00' type: post summary: 'OpsVerse replaced Cassandra with ClickHouse for Jaeger traces, built a full APM module with materialized views and Grafana — no changes to Jaeger''s code.' categories: - Webinars tags: - Altinity.Cloud - ClickHouse - joint-webinar - Monitoring - Open Source image: 'https://altinity.com/wp-content/uploads/2023/02/webinar_opsverse_01.png' published: true --- # Supercharging Observability at OpsVerse using ClickHouse® Real-Time Analytics | Joint-Webinar with OpsVerse and Altinity **Recorded**: March 28 @ 9:00 am PT **Presenters**: Arul Jegadish (Co-Founder @OpsVerse) and Robert Hodges (CEO @Altinity). In this joint webinar, OpsVerse co-founder Arul Jegadish and Altinity CEO Robert Hodges walk through how OpsVerse replaced Cassandra with ClickHouse as the trace storage backend for their full-stack observability platform and, built an entirely new APM module on top of the data they already had. OpsVerse runs a DevOps tools platform that provides observability, CI/CD, developer portals, and product analytics as a service, deployable on any cloud or as private SaaS in a customer’s own account. The observability stack brings together metrics, logs, traces, and profiling under one platform, and the key challenge was that Jaeger’s default Cassandra backend offered no analytical capabilities and was operationally expensive at scale. After switching to ClickHouse, Arul’s team gained rich analytical access to the same trace data, used materialized columns and materialized views to transform raw Jaeger JSON spans into queryable columns without touching Jaeger’s code, and built an APM dashboard in Grafana showing request latency, error rates, and SQL query behavior per application. Robert covers the ClickHouse internals that make this work: columnar storage with codec-based compression (DoubleDelta plus LZ4 for time-series integer columns, ZSTD for JSON), the Null table engine plus the materialized view pattern for on-the-fly JSON transformation, join support inside materialized views for denormalization, and the Altinity Kubernetes Operator for ClickHouse for production deployment. The Q&A explores the Cassandra migration, Kubernetes cross-cloud portability across EKS, GKE, and AKS, and the use of Crossplane to manage non-Kubernetes cloud resources. ##### **Here are the slides:** [Slide Deck – Supercharging Observability at OpsVerse using ClickHouse Real-Time Analytics | Joint-Webinar with OpsVerse and Altinity](https://altinity.com/wp-content/uploads/2023/03/Supercharging-Observability-at-OpsVerse-using-ClickHouse-Real-Time-Analytics-2023-03-28.pdf)[Download](https://altinity.com/wp-content/uploads/2023/03/Supercharging-Observability-at-OpsVerse-using-ClickHouse-Real-Time-Analytics-2023-03-28.pdf) ## **Key Moments (Timestamps)** Key moments generated with AI assistance. - 0:00 – Introduction: Robert Hodges and Arul Jegadish - 1:03 – Speaker introductions: Robert Hodges (Altinity CEO) and Arul Jegadish (OpsVerse) - 2:19 – OpsVerse overview: DevOps tools platform, cloud-agnostic, private SaaS - 2:49 – Altinity overview: enterprise ClickHouse provider, Altinity.Cloud, Kubernetes Operator - 3:57 – OpsVerse full-stack observability stack: metrics, logs, traces, profiling - 5:08 – The Cassandra problem: no analytical capabilities, high operational overhead - 6:37 – After switching to ClickHouse: APM module, Grafana integration, cost savings - 8:07 – Kubernetes platform architecture: Terraform, Argo CD, Crossplane, Altinity Operator - 10:47 – ClickHouse introduction: real-time analytic database, SQL, open source - 12:04 – ClickHouse architecture: shared-nothing, columnar storage, vectorized execution - 15:00 – Table definition walkthrough: codecs, Alias columns, partitioning, ORDER BY - 19:21 – Loading JSON data into ClickHouse: JSONEachRow format - 20:28 – Materialized columns: auto-populating columns from JSON strings - 23:06 – Null table engine + materialized view pattern for on-the-fly transformation - 26:12 – Joining on other tables inside a materialized view for denormalization - 27:40 – Cloud-native ClickHouse: what Kubernetes is and why it matters for databases - 29:12 – The operator pattern: custom resource definitions and how the operator works - 30:52 – Example ClickHouseInstallation CRD YAML - 32:03 – OpsVerse demo begins: observability stack in the OpsVerse console - 38:12 – Arul’s APM implementation: how Jaeger writes to ClickHouse, Argo CD deployment - 40:08 – Adding materialized columns to Jaeger’s base table - 42:28 – Materialized view to pre-join Jaeger tables for APM queries - 43:55 – Example query: finding all spans with HTTP errors - 44:18 – Grafana APM dashboard demo - 47:50 – Wrap-up and key takeaways - 49:12 – Q&A: Cassandra-to-ClickHouse data migration - 51:10 – Q&A: Kubernetes cross-cloud portability (EKS, GKE, AKS) - 53:05 – Q&A: data on Kubernetes adoption and running production databases in Kubernetes --- ## **Webinar Transcript** ### **[0:00] — Introduction and Housekeeping** **Robert:** Welcome to our talk on supercharging observability at OpsVerse using ClickHouse real-time analytics. My name is Robert Hodges and it’s my pleasure today to be presenting with Arul Jegadish. Just a couple of housekeeping things: you don’t need to scribble notes because we will be recording this webinar. I will send you a link to the recording as well as the slides after the webinar is complete. We do have time for questions. You can enter them into the Q&A box in the Zoom control bar, or put them in the chat. If they’re relevant to what we’re talking about at a particular moment we may answer them right then; otherwise we’ll have time at the end. ### **[1:03] — Speaker Introductions** **Robert:** My name is Robert Hodges. I’ve been working on databases for over 40 years, most recently on ClickHouse. My day job is I’m CEO of Altinity. Arul, would you like to introduce yourself? **Arul:** Hello everyone. I am one of the co-founders of OpsVerse. I have more than 20 years of experience in the DevOps and developer tools domain. I’m a great believer in open source and using open standards and open-source-based tools for DevOps. That’s something we embedded into the founding story of OpsVerse: with OpsVerse, we make it easy for anyone out there to consume DevOps tools. And Robert, thank you a lot for inviting me to this webinar. I’m a great fan of the Altinity webinar series. I’m really excited to be here today to share some of our experiences using ClickHouse at OpsVerse. ### **[2:19] — OpsVerse Overview** **Arul:** OpsVerse provides a DevOps tools platform. With OpsVerse you can get OSS-based DevOps tools running in minutes on any cloud, including within your own cloud account. We’re talking about tools like observability tools, CI/CD tools, developer portals, and product analytics tools. You’ll be able to get these tools up and running wherever you want, within minutes. ### **[2:49] — Altinity Overview** **Robert:** At Altinity, we’re an enterprise provider for ClickHouse. We’ve been in the market for over five years. Some of the folks on our team were among the earliest users of ClickHouse in North America. We run a cloud platform for ClickHouse called Altinity.Cloud, which was the first ClickHouse platform on Amazon as well as GCP, and we basically help people build real-time analytic applications in the cloud, on Kubernetes, and on-prem. We’re really big fans of open source. ClickHouse itself is open source, and there’s a whole ecosystem of tools around it that enables people to build very high-performance analytics without worrying about lock-in. ### **[3:57] — OpsVerse Full-Stack Observability Stack** **Arul:** One of the tools we run on our platform is a full-stack [observability](https://altinity.com/observability-and-logging/) solution. This is a solution where we bring metrics, logs, traces, and profiling under one platform, under one umbrella. You can come to our website, click, and run it on any region, any cloud. You can run it as SaaS or as private SaaS, meaning the entire stack gets fully managed within your own cloud account or your own network. One of the sub-components of this stack is the distributed traces component, where we use OpenTelemetry to instrument and collect traces, and then use Jaeger in the backend to process and store them. To begin with, we were using Cassandra as the backend store for Jaeger’s data, and then we switched to ClickHouse. That’s going to be the main topic here: why we searched around, and what benefits we got when we switched. ### **[5:08] — The Cassandra Problem** **Arul:** Before we started using ClickHouse, some common problems we had were that Jaeger was using Cassandra as the data store. We didn’t really have any analytical capabilities on top of that data. Jaeger would dump the data into Cassandra and we could make use of the Jaeger UI and Jaeger API, and that was pretty much it. We were not able to get anything else out of that data. Plus, running and managing Cassandra beyond a basic scale was also a little problematic, both with respect to operational overhead and cloud resource cost. At that point we started looking at other options and found ClickHouse. ### **[6:37] — After Switching to ClickHouse** **Arul:** After we switched, we got Jaeger to dump all the data it received into ClickHouse. Out of the box we gained a lot of analytical capabilities. We were able to extract a lot of insights from the Jaeger data. We built a new APM module on top of ClickHouse, and this itself was a great takeaway: if you are using OpenTelemetry to instrument your applications and generate spans and traces, just by storing those traces in ClickHouse you can actually gain a lot of insights. We connected the ClickHouse database to Grafana through a ClickHouse data source and were able to build out a full project APM module with that. We were also now able to generate alerts on top of the tracing data. And as a bonus, our infrastructure cost went down because ClickHouse has really good compression, and the overall management overhead also went down. ### **[8:07] — OpsVerse Kubernetes Platform Architecture** **Arul:** Today in this webinar we’ll be talking about Kubernetes a lot. Here’s a quick overview of how OpsVerse runs our platform. At the top is basically all the awesome DevOps tools we run and orchestrate for our customers. At the bottom we have a very thin layer of Terraform, and we use Terraform to create Kubernetes instances on any of the public cloud services: EKS, GKE, or AKS. We do the same thing within customer cloud accounts for our private SaaS. Once we create Kubernetes, everything is Kubernetes-native. We use Argo CD a lot to manage the delivery of different resources to these Kubernetes clusters running across different public cloud infrastructure. We make use of Kubernetes custom resources and operators, and that’s where we use the Altinity Operator to run ClickHouse. We also use Crossplane to create any additional cloud resources that live outside Kubernetes. For example, if we need to provision an S3 bucket or a GCS bucket, or make Route 53 entries, we use Crossplane as the bridge between the Kubernetes API and the outside world. ClickHouse fits in through the Altinity Operator, and the second part of the webinar goes into some of the implementation details. ### **[10:47] — ClickHouse Introduction** **Robert:** Let me give a quick intro to ClickHouse and then talk about running it on Kubernetes. ClickHouse is a real-time analytic database. It’s particularly well-suited to support applications that have large amounts of data rapidly arriving and that data is being presented to end users, often with very high performance requirements: the ability to bring up an operational dashboard and have it appear in a second, or to query the database and answer a question within 10 milliseconds so you can add it to a page as it’s rendering. ClickHouse understands SQL using a dialect similar to MySQL. Like MySQL it runs pretty much everywhere. It’s open source and uses the Apache 2.0 license, which means you can use it virtually any way you want without encumbrances. ### **[12:04] — ClickHouse Architecture** **Robert:** On the analytic capabilities: ClickHouse was originally designed around a shared-nothing architecture, where nodes have attached patches of storage and connect over a network. This is a straightforward architecture that is very flexible about the types of storage it can use. We store the data in columns. This means that when we do a SELECT we only have to read the data for the columns we’re interested in. Columnar format also enables enormous compression: we can get rates greater than 90 percent or even 99 percent in some cases. ClickHouse does what’s called vectorized query execution, treating columns as arrays that get loaded onto the processor. We can use single-instruction multiple-data (SIMD) instructions and are well aligned with CPU cache boundaries, so we get very efficient utilization of cores. We can also run queries in parallel across many nodes. ClickHouse clusters can scale to tens of petabytes. One of our customers I work with regularly runs about 187 nodes in their main cluster. As a result of these capabilities, ClickHouse has become very popular as a real-time analytics engine, particularly for SaaS businesses like OpsVerse: reading from event streams, reading from object storage, and exposing data to consumers in a variety of ways. ### **[15:00] — Table Definition Walkthrough** **Robert:** Let me show you a typical table definition that illustrates some visible features of columnar storage with high compression. This is test data for temperature readings on sensors. The sensor ID column has a CODEC defined: DoubleDelta. This means we don’t store the actual sensor ID values, but instead we store the change of the change as we walk down the column. After sorting by ascending sensor ID, this makes the amount of data required for this column almost vanishingly small. Then LZ4 compression is applied on top. You can learn more about [how codecs work](https://kb.altinity.com/altinity-kb-schema-design/codecs/) in ClickHouse from the Altinity Knowledge Base. We can also create Alias columns, which appear to be columns in queries but are just computed from available data. The date column is a transformation on the time column to produce a date, stored for free. Down at the bottom are partitioning and ordering. Partitioning by month creates bins of data, which are management units. For example, to keep 12 months of data you can just drop the oldest partition at the end of the 13th month. Ordering sorts the data within each part, so related data ends up in contiguous stretches of storage: for example, all readings from a particular sensor, sorted by time. This helps compression work better, helps codecs work better, and results in very fast scans. ### **[19:21] — Loading JSON Data** **Robert:** It’s very common to read data in from JSON. ClickHouse supports about 50 to 60 different input formats, from CSV to Parquet and ORC. The format called JSONEachRow means one JSON document per line. You just say INSERT INTO readings and specify the columns, add FORMAT JSONEachRow, and pipe the JSON data into the ClickHouse client. It parses the JSON and assigns it to the appropriate columns. Incredibly simple to load data. ### **[20:28] — Materialized Columns** **Robert:** One of the things we might want to do is keep the source JSON and then materialize some of those values out into proper table columns. This is a very common way of handling data, especially when the JSON documents may contain dozens or hundreds of items. We keep the raw JSON in ClickHouse in a column and then materialize certain values out. In this particular test dataset, because of compression the amount of information stored per row is extremely low: across a trillion-row dataset, we’re storing 1.34 bytes per row on average. The raw JSON document, which is pretty voluminous, only takes 4 bytes because there’s so much repetition and ZSTD compression makes it very small. Here’s an example: the table definition still has the original columns, but now there’s a JSON string column at the bottom. The sensor ID column now has a MATERIALIZED clause pointing to a JSONValue function that pulls the value out of the JSON string. When we load data, we just insert the raw JSON string and ClickHouse automatically fills in the remaining columns. The distinction between MATERIALIZED and DEFAULT is that materialized columns won’t appear in SELECT *. If you want them to appear in SELECT *, use DEFAULT. Otherwise MATERIALIZED is the right choice. And loading is dead simple: you just insert the JSON column as tab-separated values, pipe it into ClickHouse, and the table gets populated. ### **[23:06] — Null Table Engine and Materialized Views** **Robert:** A really important feature in ClickHouse is [materialized views](https://altinity.com/blog/clickhouse-materialized-views-illuminated-part-1). They take data from a source table and every time you load a block of data with an INSERT they fire a trigger automatically. This trigger runs a query over that data and puts the results into another table. We commonly use it to pre-aggregate data, but the fundamental idea is transforming input data as it arrives. Here’s the pattern: the source table uses a special engine called Null. This means we’re not actually storing any data there. The table simply exists as a place to receive input. What we’re going to do is transform that input and stick it into the real table. The real table has all the usual columns and a JSON string column. The materialized view creates a trigger: every time something is put into that input Null table, it runs a SELECT that pulls the data out of the JSON and puts the results automatically into the target table. This is a very powerful, commonly used transformation when loading data. You can almost think of this as constructing a mini pipeline within ClickHouse itself. ### **[26:12] — Joins Inside Materialized Views** **Robert:** There’s one other cool feature worth describing: as you’re loading data through the materialized view trigger, you can actually join on other tables as well. We’re not just transforming only those input columns; we can pull in data from other tables. Here’s a simple example: I have a sensor type lookup and I’d like to pull the name and description of the sensor as strings into the target table. I modify the materialized view to include a standard JOIN syntax in the SELECT query. This allows pulling in additional dimension data, so every record in the target table already has the supplementary information attached. One important behavior to understand: because this is an INSERT trigger, it only fires when you add data to the left-side table. Changing the other tables in the join won’t add data to the target table. ### **[27:40] — Cloud-Native ClickHouse: Kubernetes and the Operator Pattern** **Robert:** Because this is the environment OpsVerse is using, let me talk a bit about cloud-native ClickHouse. Kubernetes is an orchestrator for container-based apps. It has a resource model describing distributed systems: pods map to containers, volume claims map to physical storage like Amazon EBS. The problem is that this resource model is complex. For a database like ClickHouse with several nodes, it could take dozens of Kubernetes resources to represent it properly: distributing across availability zones, setting up shards and replicas, talking to ZooKeeper, and so on. Kubernetes solves this problem through operators. [The Altinity Kubernetes Operator for ClickHouse](https://altinity.com/kubernetes-operator/) defines a new kind of resource: a ClickHouseInstallation. You fill out a YAML file just like any other Kubernetes resource, apply it with kubectl, and the Kubernetes API hands it to the Altinity Operator. The operator reads the resource definition, compares it to the current state of resources in Kubernetes, and adjusts reality: if a pod doesn’t exist it creates it, if storage needs to be a different size it arranges to change it. The result is that by reading a relatively simple file describing your ClickHouse cluster, you get a full deployment in Kubernetes in some namespace. And when you need to make changes, you simply update the resource definition and resubmit it to Kubernetes. The operator handles extending storage, adding or removing replicas, and changing the ClickHouse version. Here’s a flavor of what the CRD YAML looks like: number of shards, number of replicas, where ZooKeeper is, and so on. With this operator, setting up complex clusters inside Kubernetes and then managing them is relatively straightforward. ### **[32:03] — OpsVerse Demo: Observability Stack** **Arul:** Thanks, Robert. So I’ll walk through our observability stack implementation and then show a quick demo. Just a quick recap: the OpsVerse zero stack brings metrics, logs, traces, and profiling data together. We’re talking specifically about how we handle OpenTelemetry-based traces using ClickHouse. We have an ingestion pipeline where data comes in, we transform it in a way that’s easier to query later, and we get this data primarily in JSON format. We use materialized columns and materialized views to transform it. Here’s the full data pipeline: you instrument your applications with the OpenTelemetry SDK, which makes your applications emit spans and traces. Those are collected using the OpenTelemetry Collector. The Collector sends those traces to Jaeger in the backend. Jaeger is run as part of the OpsVerse observability cloud service, and Jaeger dumps the data into ClickHouse. Users access the data through Jaeger and Grafana, either as dashboards or through ad hoc queries. ### **[34:32] — OpsVerse ClickHouse Implementation Details** **Arul:** With respect to our implementation, we needed to bring a ClickHouse installation into our Kubernetes environments. We picked the Altinity Operator. To create a production-ready ClickHouse cluster, all we have to do is define that custom resource, which is picked up by the Altinity Operator. The operator does the rest of the magic: provisioning the required pods, provisioning storage, setting up the config map, setting up the load balancer, and setting up Ingress rules. You get a production-ready ClickHouse installation. We push this custom resource file to different Kubernetes clusters and different environments through Argo CD. Now that we have ClickHouse, Jaeger needs to connect to it and write data. We use Jaeger’s plugin for ClickHouse to make that happen. That’s how we moved from supporting Cassandra to ClickHouse. And as I mentioned, we use materialized columns and materialized views to transform the data and load it into new tables. From there we use Grafana for both visualization and alerting. ### **[36:57] — Transforming Jaeger Trace Data with Materialized Columns and Views** **Arul:** When Jaeger dumps data to ClickHouse, it creates a simple table with a timestamp, a trace ID, and a model column, which is a JSON representation of the span with all the information collected through the OpenTelemetry pipeline. What we do is add materialized columns to this table. As new data is inserted, these materialized columns get populated automatically, pulling different fields out of the JSON string and creating them as first-class columns. Now we can write different queries on top of these columns. The other thing we do is deal with the fact that Jaeger splits its information across multiple tables. When we query, we have to join them. To avoid running that join every time you run a query, we create a materialized view that pre-transforms the data into a final, already-joined table. From there, we can create all our dashboards very easily. This is the base for our APM module. Then here’s an example: if you want to find all spans with HTTP errors, you can execute a simple query directly on that pre-joined table. We had to do some data transformation to get to this point, but we were able to do all of that within ClickHouse itself. We didn’t have to bring in another ETL system to process all this data. Everything was done within ClickHouse. And we didn’t have to touch Jaeger’s code at all. We just reconfigured Jaeger to dump all that data to ClickHouse and we were able to do all the data processing and schema building within ClickHouse itself. ### **[39:00] — Demo: Grafana APM Dashboard** **Arul:** Let me show a very quick demo. This is the OpsVerse admin console. When you sign up for a new account, this is what you see. You can select the tool you need and launch it. I’ll focus on the observability stack because that’s where ClickHouse comes into the picture. You go into the Observability product, click Launch, select where you want to run, give it a name, and within minutes you will have a full-fledged full-stack observability solution. I already have an observability stack up and running. You can see the different observability sub-modules launched here. I’m going to switch to Grafana. We have out-of-the-box dashboards, and you can also explore the different telemetry signals that come in. I’m going to switch to the APM module. This is the final result of everything we’ve seen so far: data comes in from your applications through the OpenTelemetry pipeline, sits in Jaeger and ClickHouse, where we transform the data, and then we’re able to build APM-level insights here by connecting Grafana to the ClickHouse data source through [Grafana dashboards on ClickHouse](https://altinity.com/blog/2019-12-28-creating-beautiful-grafana-dashboards-on-clickhouse-a-tutorial). Here you get all types of insights into your application: how it’s performing, how many requests are coming in, what the request latency is, what the error rate is, and you also get information about the SQL queries your applications are running. All of that just by instrumenting your application with OpenTelemetry. These are canned dashboards, but as an end user you can also select the ClickHouse data source, create your own ad hoc queries, and go into alerting to get alerted when, for example, your application latency is bad. ### **[42:00] — Key Takeaways and Wrap-Up** **Robert:** To quickly summarize. ClickHouse is a real-time analytic database designed to answer questions quickly on rapidly arriving data, often millions of rows per second, running into tens of petabytes, and able to answer those questions within a second. The dashboards Arul just showed illustrate that perfectly. ClickHouse runs very well on Kubernetes. Our Kubernetes Operator has been around for years. We run our own cloud service on it, we have hundreds of ClickHouse clusters out there, and as far as we can tell there are thousands or even tens of thousands of clusters worldwide using it. You can try it out on Minikube in about 30 minutes end to end from scratch on a laptop. For production environments most people use managed Kubernetes services like Amazon EKS. **Arul:** And from our perspective, by switching to ClickHouse we were able to build out a full APM solution on top of the OpenTelemetry data. If you are already using OpenTelemetry in some form, you can actually just dump that data to ClickHouse and your developers will be able to derive a lot of insights from it. If you’re not yet using OpenTelemetry, that’s something you should definitely consider. And if you’re looking for a full-stack observability solution, there are services like the OpsVerse DevOps tools platform. ### **[49:12] — Q&A: Cassandra-to-ClickHouse Migration** **Robert:** Arul, can you talk about the migration from Cassandra to ClickHouse? Did you have to move data? **Arul:** Yes, we did migrate the data. In this case we wrote a custom migration script to migrate the data from Cassandra to ClickHouse. This is all time-series data, so it was pretty easy for us to do it in batches. The data is not very complex: we really had to bring in essentially one table from Cassandra into ClickHouse, and then the ClickHouse pipeline took care of massaging the data. ### **[51:10] — Q&A: Kubernetes Cross-Cloud Portability** **Robert:** I think you showed Azure as one of the environments you’re supporting for Kubernetes. Did I see that correctly? **Arul:** That’s correct. The whole platform is cloud-agnostic. The whole platform can run wherever you have a Kubernetes cluster. We already have the OpsVerse tools platform running on all three major clouds, including within customers’ own cloud accounts. **Robert:** Did you notice any differences between the Kubernetes implementations on AWS, GCP, and Azure? **Arul:** Not really, and I think that’s the key point here. There is some difference when you bring up and install Kubernetes itself, but once you have Kubernetes up and running the differences are pretty minimal. If you truly develop your applications and tools to be Kubernetes-native, you can think about bringing Cloud-agnostic features to your application and making your applications multi-cloud. **Robert:** That is very consistent with our experience. When we built Altinity.Cloud we started on Amazon in October 2020, and about nine months later we decided to bring GCP on. It took us eight weeks end to end to port from Amazon to GCP, and it was mostly deciding how to get the clusters set up within Kubernetes itself. The places where we do see differences are things like external connectivity, like what kind of load balancer you set up to get traffic off the internet. But by and large, within Kubernetes itself things just run. The portability story is really good. ### **[53:05] — Q&A: Data on Kubernetes** **Arul:** I also want to highlight our usage of Crossplane. Once you’re in the Kubernetes world, if you need to bring in any other non-Kubernetes cloud resources, Crossplane is a great way to connect your Kubernetes to the rest of the world. For example, today when a user comes to our admin console and launches an observability stack, we need to provision S3 or GCS buckets and make Route 53 entries. In those cases we can still operate within the Kubernetes context. Making a Route 53 entry is just another custom resource we push to the Kubernetes cluster, and the Crossplane connector takes care of making that connection to the AWS resource. **Robert:** Did you and your team have background in Kubernetes already, or was it something you started when you founded OpsVerse? **Arul:** We already had some experience running production-grade applications on Kubernetes, but specifically running databases and data on Kubernetes was something we started only at OpsVerse. In general in the industry, adoption of data on Kubernetes is still low. There is this assumption that you can’t run big databases on Kubernetes. But we did it at OpsVerse. We run ClickHouse, PostgreSQL, and a few other data stores on Kubernetes at scale, multi-cloud, across different cloud environments. **Robert:** Since you mentioned data on Kubernetes, I want to put in a quick plug for the organization called DoK, or Data on Kubernetes. It’s a worldwide community devoted to promoting people running databases on Kubernetes. We’re also starting a new Meetup for data on Kubernetes in the Bay Area, and our first Meetup will be in May. Arul, this is something where the folks at DoK would definitely be interested in your experiences. **Arul:** Definitely, perhaps we can come and do a talk there. **Robert:** I don’t see any further questions, so I think we’ll call it a day. Arul, thank you again, and thank you everybody for attending. We’ll send out the links to the slides and the video within a day or two. **Arul:** Thank you, thanks everyone, thanks, Robert. ## **FAQ** **Why did OpsVerse switch from Cassandra to ClickHouse as the Jaeger trace storage backend?** Cassandra offers no analytical capabilities beyond what Jaeger’s own UI provides. Once data is in Cassandra, you cannot run custom queries, build custom dashboards, or generate alerts on the trace data without a separate analytics layer. Cassandra was also operationally expensive at scale. Switching to ClickHouse gave OpsVerse full SQL access to the trace data, enabled building a custom APM module and Grafana dashboards on top, and reduced both infrastructure costs and management overhead through ClickHouse’s built-in compression. **How does OpsVerse transform Jaeger’s raw JSON trace data in ClickHouse without modifying Jaeger?** OpsVerse adds materialized columns to Jaeger’s base ClickHouse table. When Jaeger inserts a row, ClickHouse automatically evaluates the materialized column expressions and populates derived columns by extracting fields from the JSON payload using built-in JSON functions. For more complex transformations that require joining across multiple Jaeger tables, OpsVerse creates a materialized view that pre-joins the tables and writes a clean, queryable result into a new target table. All of this processing happens inside ClickHouse with no changes to Jaeger’s code. **What is the Null table engine plus materialized view pattern and how is it used for JSON loading?** The Null table engine creates a table that accepts INSERTs but does not store data. It acts as a trigger point. A materialized view attached to a Null table fires every time a block of data is inserted, running a SQL query over that incoming block and routing the results to a real target table. This pattern decouples the insert format from the storage format: raw JSON can land in the Null table and a materialized view can transform it, apply codecs, join with dimension tables, and write highly optimized rows to the target table. The same pattern is used for real-time aggregation, schema normalization, and PII masking. **How does the Altinity Kubernetes Operator for ClickHouse simplify production deployments?** The operator introduces a ClickHouseInstallation custom resource that lets you describe a ClickHouse cluster in a YAML file, similar to any other Kubernetes resource. When you apply this file with kubectl, Kubernetes hands it to the Altinity Operator, which creates all the necessary pods, persistent volume claims, services, config maps, and ZooKeeper configuration automatically. When you need to change the cluster, such as adding replicas, extending storage, or upgrading the ClickHouse version, you update the YAML and resubmit. The operator reconciles the desired state with the actual state of the cluster. **Is ClickHouse portable across different Kubernetes providers such as EKS, GKE, and AKS?** Yes, and OpsVerse’s experience confirms this. Once Kubernetes is running, the differences between cloud providers are minimal: most challenges arise when initially provisioning the Kubernetes cluster itself and when configuring external networking. Within Kubernetes, ClickHouseInstallation CRDs, operator behavior, and Argo CD deployments work consistently across all three major clouds. OpsVerse also uses Crossplane to manage non-Kubernetes cloud resources such as S3 buckets, GCS buckets, and Route 53 entries through the Kubernetes API, maintaining a fully Kubernetes-native workflow regardless of which cloud is underneath. **What is the full observability pipeline OpsVerse built with ClickHouse?** Applications are instrumented with the OpenTelemetry SDK, which emits spans and traces. The OpenTelemetry Collector receives those traces and forwards them to the Jaeger backend. Jaeger uses the jaeger-clickhouse storage plugin to write all span data to ClickHouse. OpsVerse then adds materialized columns to extract key fields from the JSON span model and creates a materialized view that pre-joins Jaeger’s multiple tables into a single queryable APM table. Grafana connects to ClickHouse via the ClickHouse data source plugin and queries this pre-joined table to power APM dashboards showing request counts, latency percentiles, error rates, and SQL query behavior per application, as well as generating alerts. --- © 2023 Altinity, Inc. All rights reserved. Altinity®, Altinity.Cloud®, and Altinity Stable® are registered trademarks of Altinity, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc. Altinity is not affiliated with or associated with ClickHouse, Inc. Kubernetes, MySQL, and PostgreSQL are trademarks and property of their respective owners.