Real-time Gets Real: The Shift to Fresh Analytics Data

Recorded: May 11 @ 10:00 am PT
Presenters: Jorge Sancha & Robert Hodges

In this joint fireside-chat webinar, Altinity CEO Robert Hodges and Tinybird CEO Jorge Sanchez trace how real-time analytics moved from an expensive Wall Street luxury to something any developer can build on, and explore what that shift actually unlocks for businesses.

Robert opens with the origin story of ClickHouse® at LifeStreet, an ad platform that was generating 300 billion events per day and needed to compute the value of individual ad impressions, including all the losing bids, fast enough to act on them. The discovery that ClickHouse could answer these questions reliably in 10 milliseconds or less was the founding insight for Altinity. Jorge mirrors it with Tinybird’s origin at Carto, where customer datasets were growing by an order of magnitude each year and the only way to keep up was to stop building hand-tuned ETL pipelines and give developers a direct path from data to API.

The conversation then ranges across four themes: what real-time means beyond just speed (affecting the present user experience rather than analyzing the past), use cases that wouldn’t be conceivable any other way (intent-driven subscription prompts, real-time inventory accuracy, security event correlation), the open-source stack that made this accessible (Hadoop, Kafka, Kubernetes, ClickHouse, visualization toolkits), and what makes ClickHouse uniquely suited to this category of problems. Both demos run live: Jorge builds a serverless analytics API from schema generation to parameterized endpoint in Tinybird, and Robert demos Altinity.Cloud managing a trillion-row cluster on Amazon EKS alongside a two-node cluster on a physical Intel NUC in his home closet.

Here are the slides:

Key Moments (Timestamps)

Key moments generated with AI assistance.

• 0:00 – Introduction: Cameron Archer, Tinybird content lead, opens the webinar
• 0:53 – Robert Hodges and Jorge Sanchez introduced
• 1:13 – Robert and Jorge on the format: conversation rather than presentation
• 2:44 – Robert’s origin story: Alexander Zaitsev, LifeStreet ad platform, 300B events/day
• 5:38 – The problem: valuing ad impressions including all the losing bids
• 6:41 – Jorge’s origin story: Carto, location intelligence, data growing 10x per year
• 9:06 – The cathedral of infrastructure problem: data lakes, pipelines, warehouses, caches
• 12:29 – The two constants: data will never stop growing, people want results faster
• 13:22 – Real use case 1: intent-driven subscription pop-up, 10ms ClickHouse query
• 15:21 – Jorge: real time shifts analytics from understanding the past to affecting the present
• 17:10 – Real use case 2: hotel booking personalization with real-time user vectors
• 18:53 – Booking.com scarcity signals: “3 other people are looking at this room”
• 19:44 – Real use case 3: security events and event management
• 21:00 – Lambda architecture going away: fast and batch layers merging into one database
• 22:31 – Real use case 4: smart inventory and real-time stock accuracy for retailers
• 25:14 – The compounding effect of faster decisions
• 27:10 – Developer experience: query latency as a development multiplier
• 28:44 – Historical context: Wall Street in the 90s — fast analytics existed but was unaffordable
• 29:08 – Open source as the key democratizing force: Hadoop, and what followed
• 32:47 – Event streams and Kafka as a facilitator for real-time data capture
• 35:00 – Kubernetes as a new private cloud: portability, cost control, BYOC
• 37:13 – Serverless: removing infrastructure friction for developers and companies
• 39:52 – Visualization as the other key element: Grafana, Superset, JavaScript toolkits
• 41:35 – Low-latency column stores and the SQL revival in analytics
• 42:53 – What makes ClickHouse great: Jorge’s perspective
• 44:57 – What makes ClickHouse great: Robert’s perspective — Aggregates off source data
• 47:11 – Tinybird demo begins: serverless analytics API over ClickHouse
• 59:08 – Altinity.Cloud demo: trillion-row cluster on Amazon EKS
• 1:02:00 – Live query on trillion-row dataset: sub-second response
• 1:03:05 – Materialized views demo: 23ms response
• 1:03:55 – Altinity.Cloud manages a cluster on a physical Intel NUC in Robert’s closet
• 1:06:15 – Vendor unlock-in: disconnecting Altinity.Cloud leaves ClickHouse running
• 1:07:52 – Wrap-up and contact information

Webinar Transcript

[0:00] — Introduction

Cameron: Welcome everybody. My name is Cameron Archer, I’m the content lead here at Tinybird. I’m just running things here so that Jorge and Robert can focus on doing their thing. We are recording this webinar so we’ll send out that recording after we’re done. Feel free to ask questions throughout, or wait until the end. If they aren’t relevant to what we’re discussing, Jorge and Robert will answer them after. With that, I’ll hand it over to Jorge and Robert. We’ve got Robert Hodges, CEO of Altinity, and Jorge Sanchez, CEO and co-founder of Tinybird. Jorge, I’m making you the host, so you’re in charge.

[1:13] — Format and Introductions

Jorge: Great. Thank you, Cameron. Welcome everybody. I’ll be sharing my screen in a second. We have some backup slides for Robert and me to use as background.

How are you, Robert?

Robert: I’m doing great, Jorge. Thank you so much for having me as a guest.

Jorge: Of course. I think this is not very common, you know, two companies that are doing very similar things, to a certain degree competitive, talking together and trying to teach about some of the things we find in the field. We’re going to talk about real time, which is what we’re covering today. So thank you for joining. Should we get started?

Robert: Let’s do it.

[2:44] — Robert’s Origin Story: LifeStreet and ClickHouse®

Robert: So the way that I got into analytic databases was that I joined Altinity. One of my best friends, Alexander Zaitsev, is CTO there, and back in 2015 or 2016 he told me: hey, there’s this pretty cool database I think you ought to have a look at, it’s called ClickHouse®.

The way this really got kicked off was that Alexander was running analytics for an ad platform in the Bay Area. They were doing what’s called real-time ad bidding, and this is perhaps not super well known how data-intensive it is. They had built a platform generating up to 300 billion events a day coming off their ad platform.

One of the questions they wanted to answer from this flood of data was a really basic one: how much is an ad impression worth? If you’re running a platform and you’re running ads, you’re always trying to figure out whether you’re getting your money’s worth. It turns out this is a surprisingly difficult question to answer because there are literally hundreds of properties of one of these impressions and the bids for them. Where’s the person from, what platform they’re on, what creative is showing, where is this going to be shown. You need serious number crunching to figure out the value of that impression so you know what to bid.

And one thing that makes this problem even more complicated: when you’re valuing the impression it’s not enough to know the winning bid. You have to see all the losing bids as well. It’s like when you’re bargaining for a car and you say I’ll pay you $30,000 for it, and the salesperson says done, and you think: well, I obviously paid too much, or they would have had to go back and talk to their manager. That was the problem they were solving.

They were originally working with Vertica, which was helping them solve this problem when their data was at a lower level. But they ran into scaling problems with Vertica, mostly related to the fact that it didn’t use hardware super efficiently. Alexander brought in ClickHouse, which had recently been open sourced in 2016. They applied it to this problem, and loading 300 billion events a day, they were successfully able to solve it. The realization was: this database is really something special. The fact that it can solve this problem and enable analysts to deal with this torrent of data ought to be the foundation of a company that supports it. That’s how Altinity got started.

[6:41] — Jorge’s Origin Story: Carto and the Data Growth Problem

Jorge: Super interesting. One day you keep finding new magnitudes of data. You think you’re working with amazing amounts, and then you find another customer doing things you couldn’t imagine.

I’ll tell you a little about Tinybird. We came to Tinybird through Carto, where all the Tinybird founders previously worked. Carto was doing location intelligence, technology based on Postgres at the time, and we could see that our customers were bringing in an order of magnitude more data on average every year. Maybe a customer would start with 10 million records, which for a Postgres database is not nothing, and the next year it would be 100 million, and the following year a billion. Postgres wasn’t really designed to scale like that, or not without a lot of fine-tuning on a case-by-case basis.

What we were finding is that we spent a lot of time helping our customers pre-aggregate data and think closely about exactly what data they needed to upload into Carto. Instead of saying here’s data, let me start querying and building over it, we had to help them build complex ETLs and massage the data first. We were all developers and engineers, and we hated that.

Data engineer is almost a new role, based on the amount of data being generated. We started thinking: how could we work with a thousand times the data? How could we make sure the platform will scale and people can throw data at it and start working with it?

[9:06] — The Cathedral of Infrastructure Problem

Jorge: Carto went in a different direction, but we started going to other companies and seeing the same things. Whenever there’s large amounts of data and you want to do something with it while the data is still fresh, people throw cathedrals of infrastructure at this problem. They have something to capture the data, then they move it into a data lake, then there’s an Airflow pipeline getting that data and putting it into a data warehouse, then there’s dbt or something preparing that data, and finally if you want to do something low latency you have to move it to a DynamoDB or MongoDB-type database and still build an application. It just feels like a lot of things to do for something that is conceptually pretty simple: you have data, you want to write queries, and you want to do something with the result.

That’s how we started thinking about Tinybird. We can do better. We can make this easier. And it should be easier, because otherwise as data grows, how are companies going to be efficient at building new things over it?

[12:29] — The Two Constants

Jorge: We could see two things that were never going to change. One is that data was not going to stop growing. The other is that people were never going to want to wait longer for results. If anything, people want to move faster. Those are constants. That’s how we came to Tinybird and the problem we try to solve.

Robert: And it’s interesting. I think of I.T. as going through waves of what I call cathedral building. You know, Romanesque, Gothic, late Gothic, these different waves of architecture in the Middle Ages. And the current era of cathedral building in I.T. is building real-time analytic systems. As you say, the Carto experience brought you to this new set of problems. For us it opened up a whole world of applications and problems that could now be solved with real-time data.

[13:22] — Real Use Case 1: Intent Detection at 10 Milliseconds

Robert: Can I take the first one?

Jorge: Yeah, go ahead.

Robert: One of the joys of this is seeing the creative ways that people use real-time data to drive their business, create new opportunities, and in some cases create entirely new businesses. One of my first big ahas was an application designed to manage when you pop the question. You know: hey, you’ve been browsing this site for a while, would you like to sign up for a subscription? And maybe turn your ad blocker off too?

The way they managed it was that as people were traversing the site, their browser was sending requests and the server was capturing logs from the web server. They were tracking which pages people had visited, and basically each time a page rendered they would go back to ClickHouse, run a query asking: how many pages has this person visited, what’s their intent? And within about 10 milliseconds, return an answer: is it time to pop the question?

This is something that with traditional BI technology, which tends to be slow and unpredictable in performance, would be inconceivable. With a database like ClickHouse, because you can reliably get answers in 10 milliseconds or less if you know what the question is in advance, you can now solve this problem. It was a mind-blowing aha moment that then opened up even more use cases.

[15:21] — Real Time Shifts Analytics From Past to Present

Jorge: That’s an amazing insight. One thing I’ve learned over the last couple of years when it comes to real time is how analytics has traditionally been used to understand the past: what’s been going on in my website, what have people been doing, what are they interested in. Whereas with use cases like these and with real-time technology like ClickHouse, you can affect the present. You can affect the user experience. You don’t have to wait to see what I’ve learned about all these users, make a change to my website, and then wait again to see if it’s improving. You can actually affect the user experience in real time based on what the user is doing right now, what other users like them are doing right now, and what is their likely intent. You can give them a discount or intervene. That’s been a great insight: what real time can actually do. Some companies say we don’t need real time for this, but if you haven’t realized that you can improve and gather information and react to those insights in real time, you don’t know what you’re missing.

[17:10] — Real Use Case 2: Hotel Booking Personalization

Jorge: We have a few personalization use cases similar to what you’re describing. We’re capturing events for users browsing a hotel booking website. One of our customers keeps a vector for each individual visitor. They don’t know who this visitor is; they just have a footprint. But they know the top destinations visited, how many times, how many queries they’ve made. That vector is being kept up to date in real time, and every time the user takes a step on the website, the system looks at this vector and sends it to a machine learning model to get a response: what’s the chance that if I offer them a discount they’re going to convert?

Think about how you’d do that without technology like ClickHouse. It’s hard.

And then use cases like what we’ve seen at Booking.com: “three other people are looking at this room.” There’s a bit of a dark pattern there, which both our businesses help enable by the way, creating scarcity. But actually that’s not easy. If you look at doing it without a columnar database like ClickHouse or technology like ours, it’s very hard. You have to build something specific for that data pipeline, that capture process, that storage format. Whereas now you can do it in a much more straightforward way.

[19:44] — Real Use Case 3: Security Event Management

Robert: There’s another class of use cases that ties in here. You mentioned history and being able to look back, and also being able to work in real time. One of those is security and event management. One of the key things about security events is you want to see them pretty quickly. Somebody’s breaking into your systems; you need to know immediately, or as close to it as possible. But at the same time, once you’ve gotten that real-time notification, you need to go back and look at the history of what was going on. Hey, somebody is suddenly making DNS calls to a site that’s a known malware source. Where did this happen? What’s the prevalence? Can we drill down to the application making this call? These are things that require going back into history, then doing slicing and dicing queries.

This is one of the really interesting things about real-time data technology. In the old Lambda architecture, which Nathan Marz described back in the Hadoop era, you’d have a fast feed reading straight off the event stream and a batch layer doing complex number crunching, with that batch data showing up 24 hours later. Because there was no database that could do both well. What we see now is people doing both of those in the same database. That’s been a key driver to reduce cost and reduce complexity. These vast pipelines that did separate fast and batch processing just go away.

[22:31] — Real Use Case 4: Smart Inventory

Jorge: When we say real time creates opportunities, it has to do with not just reducing complexity and reacting faster, but the compounding effect. If you can make decisions faster, you don’t have to wait for things to crunch; you know immediately. The compounded effect of all those decisions over time is huge. Whether reacting to an opportunity during a Black Friday marketing campaign or reacting to problems when suddenly sales aren’t coming in and there’s an upstream issue you’ve missed.

One interesting case is smart inventory for big retailers with warehouses all over the world constantly emitting events: one less of this item in this location, and so on. Getting a snapshot of how inventory looks across the world at a given time for purchasing decisions is not an easy problem. The sheer volume of real-time events makes it very difficult to maintain an accurate snapshot. And this has an impact on sales too: you’ve probably bought something online and gotten an email saying sorry, it’s out of stock. They sold it to you because they didn’t know it was out of stock. They made an educated guess based on a batch calculation. We’ve solved this for a couple of customers and it has a huge compounding impact: not just money in the moment but better user experience and more people coming back over time.

Robert: And it all comes down to making better decisions faster. Now enabling people to order the shoes and know they’re actually going to arrive. For me as a vendor, to give somebody an offer and know that there’s a reasonable chance it’s the best one I can show them. There’s a whole world of these use cases and we see them every day.

[27:10] — Developer Experience: Query Latency as a Development Multiplier

Jorge: There’s another aspect to this that has to do with developer experience. If you’re working with a technology like Spark, which is great for certain things, but if you run a query and it takes 30 minutes every time, the experience of building something is horrible. Just having queries that return immediately while you’re developing has a huge compounding effect in terms of building faster, having happier developers, and shipping more code.

Robert: That’s a really important point. At LifeStreet, the company where Alexander originally used ClickHouse, the analysts had a word for that kind of 30-minute query: the “not worth trying” disease. You knew that if you queued something up, you’d wait so long that if you got it wrong you couldn’t repeat it. You’d get basically 16 queries in an eight-hour day at that rate. Combating the not-worth-trying disease was something they were doing with this new real-time technology.

[28:44] — The History: Expensive Before It Was Accessible

Robert: I want to give some historical context. The problems we’re describing were solvable in earlier times with appropriate technology, but it was extremely expensive. I worked at Sybase in the 90s and we dominated Lower Manhattan and Wall Street because people were building trading systems and needed to go fast. If you’ve ever worked in finance, you know they will pay any amount to win at trading.

What has happened is that this technology, which was proprietary and available only to a few, has really opened up over the last decade. The single biggest thing was open source. The data warehouse technology, beginning with Hadoop, was open source. All of a sudden any developer, given a large enough machine, could run it. That began a cascade of projects focused on analytics, accessible to all developers. And they were generally driven by people solving problems they actually had. Hadoop is a classic example: it was solving an analytic problem at Yahoo. Doug Cutting needed to crunch data at scale, came up with an implementation of MapReduce, built it. Open source has had a huge influence.

Jorge: Absolutely. And you can sort of see the same thing happening in AI right now. Some of the open source models are almost as good as the closed ones already. It’s a testament to the trend.

Robert: And it leads to a crowdsourcing effect on the software itself. The best example is you and I. As you said at the beginning, in another universe we’d be arch enemies, and yet there’s this underlying software we both share and have a combined interest in making better. Thank you by the way for the bug fix on the string aggregate problem last fall, that was a really great one.

Jorge: I’ll take all the credit and I’ve done nothing specifically myself to solve it, just kidding. I’ll pass it along to the team. Yeah, that’s super important: we can agree on where this technology should go and discuss it, and give back to the community. Both our companies are built on top of open-source technology, not just ClickHouse but everything else. So it’s really great that we get to contribute back by fixing problems and moving things forward.

[32:47] — Event Streams and Kafka

Robert: Event streams is a really important one. Event-driven architectures set the stage for real time because the first problem you have is how do you capture data in a safe way that you can read at different speeds for different purposes. Kafka has been a huge multiplier for real time. Back in the Big Data era, companies felt incredibly anxious about having all this data and not knowing where it was or how to capture it. Kafka gave the market a way to say: with this, you can capture the data even if you don’t do anything with it yet. You now know you have a way to do it and you can exploit it in many different ways.

Jorge: Kafka and projects like it made capturing huge amounts of data a convenient, almost standard approach. Before that, everyone was doing their own thing. It’s not that everybody uses Kafka right now, but Kafka has made that a really convenient way to capture huge amounts of data that you can then consume as soon as you’re ready.

Robert: What’s interesting about Kafka was that it developed completely independently of data stores. Message buses already existed in enterprises for decades, but they were much more closed models with assumptions that producers and consumers knew who they were. With Kafka, producers just throw data into this big pipe and consumers read it out. There can be as many consumers as you want. They can replay. Kafka is that central log that holds the stream, often for months of data. That one was a huge step forward.

[35:00] — Kubernetes as a New Private Cloud

Robert: Kubernetes has been an important one. I used to work at VMware, and VMware gave people the ability to utilize hardware efficiently at a time when packing applications onto hardware and using all the cores was hard. It solved that problem and gave people a stable platform that ran everywhere. Kubernetes does the same thing. You can take an application that runs on one of the machines on my desktop, and if it runs there it will also run in Kubernetes on Amazon or GCP. It has a tremendous impact on portability, and it means we can build services that run essentially the same way in multiple locations and use the underlying hardware very efficiently.

This has been key to controlling costs in two ways: one is utilization, which is always a concern when running applications. The second is that a lot of times costs are affected by where you run. Having the ability to say hey, I could run in the cloud but actually I need to run this one in my data center because I’m going to really work with the hardware, being able to have that choice is something that Kubernetes enables.

[37:13] — Serverless: Removing Infrastructure Friction

Jorge: Serverless has been an interesting one for us. There are always going to be businesses that want full control over every knob and configuration to extract maximum value and control over what they’re building. But we’ve seen huge interest from companies that just want to build quickly and get to a solution they can trust will scale, without necessarily understanding everything underneath. Initially we were seeing it a lot with small startups that just want to go fast, but we’re seeing it more and more with bigger companies who are realizing: if I don’t have to invest in infrastructure and DevOps people, can I invest that money in developers so we can ship more? We’ve seen that play out in a few companies and it’s been pretty exciting.

[39:52] — Visualization as the Other Key Element

Robert: I’ll highlight visualization as maybe the last one before we get to our favorite low-latency column store. When I started this job over four years ago I expected to hear a lot about AI and integration with machine learning. It was crickets. Maybe one percent of our customers have ever asked about it directly. But what every single user of ClickHouse and of our systems cares about is visualization. Along with event streams, that’s the other key element of these successful systems: being able to create UIs that allow people to slice and dice and run queries, count things in different ways, display data in different ways, and use the amazing human ability to recognize patterns in data.

There are tools like Grafana, Superset, and a host of other BI tools. And then on top of that there are really great JavaScript-based toolkits that allow people to build custom graphics that are amazingly responsive. That’s a really fundamental part of some of the more successful systems we’ve seen.

Jorge: Agreed absolutely. And I think people underestimate just how good we humans are at recognizing patterns and being able to dive into data when it’s presented well.

[42:53] — What Makes ClickHouse Great: Jorge’s View

Jorge: Low-latency column stores. What’s interesting is there was this big NoSQL movement because querying over large amounts of documents was really fast, and SQL was pronounced dead a few times. Then interestingly, column stores and the ability to query structured data, do joins over huge amounts of data, and have aggregation capabilities turned out to be the way to go for analytics. The separation of storage and compute is also still an evolving area, still some assembly required.

When we started tinkering with ClickHouse we were super excited because we thought: these guys are crazy. In a good way. Every function was designed to be incredibly efficient at what it does. We fell in love with it really quickly, even when all the documentation was in Russian. We love the ability it has to scale both horizontally and vertically, running queries over huge amounts of data or running thousands of queries over huge amounts of data. The combination of simply writing SQL and getting those results is amazing. Tinybird started with the thought: this is a Formula One car, how can we enable any driver to drive it without needing the mechanics in place the day of the race and having trained for 20 years? How can we enable any SQL developer to take advantage of this amazing database?

[44:57] — What Makes ClickHouse Great: Robert’s View

Robert: That drag racer image is one that most of us who use ClickHouse have in mind. It goes straight, doesn’t do curves so well, and needs to be handled with care to be used effectively. When I look at how our users use ClickHouse, it has properties that if you can think of a way to make a database go fast, there’s a pretty good chance ClickHouse uses it somewhere. But what users get out of it is this ability to do aggregates directly off source data. You’re reading in trillions of rows and running your queries straight off the sources without having to prepare them first.

When I first started I didn’t understand what that meant. Aggregates: counts, sums, histograms. The point is these are the operations that allow humans to get insight into these huge volumes of data. We almost never look at individual rows. We’re doing averages, number of unique visitors per hour, things that allow us to reason about enormous data sets. ClickHouse does this exceedingly well. In fact it was the original problem definition when ClickHouse was developed: it was built to solve web analytics, to do essentially the same thing that Google Analytics does, take source data, compute any aggregate the user wants on the fly, and hand back a result. And it is amazingly good at that.

[47:11] — Tinybird Demo: Serverless Analytics API

Jorge: We’ve got about 10 minutes left. Do you want to show how we help users?

Robert: Absolutely, go ahead.

Jorge: Tinybird essentially helps any SQL developer or data engineer to take advantage of ClickHouse and build APIs over large amounts of data without any infrastructure management. It’s a serverless product. When you log in you create a workspace with just a name, no machine size to specify, nothing like that.

There are two main concepts in Tinybird: data sources and pipes. Data sources are the source of data, which can be very different: you can send data through our events API, which is an HTTP streaming endpoint that accepts NDJSON; you can upload files; you can connect to Kafka, Confluent, BigQuery, Snowflake. More connectors are coming.

For this demo I’m going to use a feature we created recently using AI: you can generate a schema and start generating sample data. I’ll ask it to generate a schema for analytics events that track users across a website. It gives me: user ID, page URL, browser type, page title, event type, referral URL. I’ll confirm and ingest, and it starts generating data and sending it to this data source, which is a ClickHouse table underneath.

Now I can start querying. We do this with something called pipes. I’ll create a pipe called simple stats, add a node called filter data, select the columns I’m interested in: event timestamp, page URL, referral URL, device type, browser type. I’ll filter by device type equals desktop. Then I’ll add another node that groups and counts hits per browser type from that filtered result, ordered descending. This returns in 0.51 milliseconds. Under a millisecond on a live data source.

Once you have a pipe, you create an API endpoint by selecting the output node. Immediately you have this information available as JSON, NDJSON, CSV, or Parquet. You don’t have to build that API. You can paste the URL somewhere and integrate it into your product right away. You can also add parameters: instead of hardcoding desktop, I can make device type a template parameter with a default of desktop. Now the API is automatically documented and I can query for mobile instead by passing it as a query parameter.

For teams that want to treat this as code, I can pull everything via the CLI: the data sources serialize to a file with the schema and ClickHouse parameters, the pipes serialize to another file, and you can use Git and build tests as part of a proper data application workflow.

[59:08] — Altinity.Cloud Demo: Trillion Rows, Anywhere on Kubernetes

Robert: Great, let me show a contrasting view. We took a really different approach. Tinybird built a PaaS with impressive application development features. Our approach has always been the database as the core.

What we built is a platform designed to let you automate and control any aspect of a running ClickHouse cluster and run it anywhere. Here’s an environment running about a trillion rows of data on Amazon EKS. You can upgrade the server, rescale it, pause it, run backups. All the operations you’d normally have to script are in this API.

We have very detailed views of what’s going on inside the cluster. Metrics are all built in. We are the maintainer of the community Grafana plugin for ClickHouse, so these metrics are always available without any setup. On top of that we have enterprise support baked in. If you have a question like how do I integrate with Kafka on Amazon MSK, or how do I do schema design to minimize the amount of data I’m reading, we have people who know how to do that.

Let me run a quick query. Here’s some temperature sensor data on this trillion-row dataset. What was the day that a certain sensor recorded its maximum temperature? We get that back in under a second. That’s an example of real-time response.

People can also build the true real-time data using materialized views, getting answers back in the time it takes a page to render. Let me show an example. There it is: 23 milliseconds. If you call it a couple of times you get enough cache heating that you’ll be down to around 10 milliseconds or less. Really fast.

Here’s something interesting. This is an Intel NUC in a rack in my closet at home. We can manage a cluster on that too. Let me find it. Here it is: Robert’s NUC. We can automatically turn compute off when it’s not being used and restart at any time. Let me resume this two-node cluster. To prove this is not somewhere in the cloud, here’s a window where I’m SSHed into that Intel NUC, and if we watch for a minute you’ll start to see it spawning up Kubernetes pods to bring the cluster back up.

The point is that we can run and manage ClickHouse practically anywhere. The only requirement is that Kubernetes can run there. Our goal is to enable people to have the choice of running where it makes the most sense for the business and to have complete control over the data.

One other interesting property: you can actually unplug us. If you take us away, your analytics will continue to work. You’re not tied in. We call this vendor unlock-in. We call it that because you can run ClickHouse in your own Kubernetes with our management and disconnect us at any time and continue running on your own. It’s a cool illustration of how even though Jorge and I share this open-source core, we’ve taken really different takes on the market. And I actually think one of the reasons we get on is that this market is big enough for both of us, and we actually make the ecosystem richer by taking different approaches and giving people choices.

Jorge: Absolutely. And yeah, I think we’re about at time.

[1:07:52] — Wrap-Up

Robert: Thank you so much, Jorge. This has been a real pleasure. This is such an interesting topic and I think there’s this joy, beyond making money, at seeing the creative ways that people use this technology.

Jorge: I have the exact same feeling.

Robert: There seems to be a question from Diego. Ah, he’s telling us we did a great job.

Jorge: Thank you Diego.

Cameron: Thanks everybody. Robert and Jorge did a great job. We’ll send out the recording very soon, so keep an eye out for that.

Robert: Thank you Cameron. Have a great day everyone.

Jorge: Thank you, bye-bye.

FAQ

What is the “cathedral of infrastructure” problem and how does real-time technology solve it?

Traditional analytics architectures require data to pass through many separate systems before it can be queried: capture layer, data lake, Airflow pipeline, data warehouse, transformation layer, and finally a low-latency store for applications. Each of these requires separate expertise, infrastructure management, and operational overhead. Real-time analytic databases like ClickHouse collapse much of this stack by combining the fast event-feed path and the deep historical batch path in a single system, dramatically reducing both complexity and cost.

What does it mean to run ClickHouse queries in 10 milliseconds, and why does it matter?

Ten milliseconds is fast enough to embed a ClickHouse query inside a live user interaction: page renders, API calls, real-time pricing decisions, or intent signals. This enables a new category of application where the analytics layer actively affects the user experience in the moment rather than informing decisions made hours or days later. Traditional BI tools cannot hit these latencies reliably, which is why applications like real-time subscription intent detection or live inventory signals were simply not feasible before columnar real-time databases became widely available.

How do materialized views help achieve truly real-time response times?

Materialized views in ClickHouse run a query on every arriving block of data as it is inserted and store the pre-computed results in a separate table. When the application queries that results table rather than the raw data, it bypasses the full scan entirely and returns pre-aggregated values. Response times drop from hundreds of milliseconds on a raw query to tens of milliseconds or less on the materialized result, even over trillion-row datasets. This makes materialized views a key tool for building dashboards and APIs that must respond within the time it takes a page to render.

What is Altinity.Cloud Anywhere and how does it differ from a standard managed ClickHouse service?

Altinity.Cloud Anywhere allows Altinity to manage ClickHouse clusters inside the customer’s own Kubernetes environment rather than inside Altinity’s infrastructure. The customer retains full ownership and control of all data, compute, and networking, while Altinity provides the management plane, monitoring, and enterprise support. The cluster can run on any Kubernetes environment: Amazon EKS, Google GKE, Azure AKS, Hetzner, or even a physical machine running Kubernetes on-premises. Critically, the customer can disconnect Altinity.Cloud at any time and continue running their ClickHouse clusters independently using open-source components.

Why is the open-source ecosystem considered essential to the rise of real-time analytics?

Before open source became dominant in analytics, fast real-time systems required expensive proprietary software accessible only to large financial institutions or a handful of technology leaders. Hadoop was the first project to make large-scale data processing accessible to any developer with a large enough machine. The subsequent wave, including Kafka for event streaming, Kubernetes for portable container orchestration, and ClickHouse for columnar analytics, has made it possible for small companies with a handful of engineers to work with billions of events per day. Open source also creates a compounding crowdsourcing effect on software quality, enabling companies that might otherwise compete to collaborate on the shared infrastructure layer.

© 2023 Altinity, Inc. All rights reserved. Altinity®, Altinity.Cloud®, and Altinity Stable® are registered trademarks of Altinity, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc. Altinity is not affiliated with or associated with ClickHouse, Inc. Kubernetes, MySQL, and PostgreSQL are trademarks and property of their respective owners.