Iceberg, ClickHouse®, and AI ft. Shivji kumar Jha

Introduction

In this episode, recorded in person in Bangalore, Josh is joined by Shivji Kumar Jha, Staff Engineer at Nutanix, where he heads the distributed data organization. Shiv has spent more than a decade inside databases and streaming systems, first as a contributor to MySQL replication, later working with Kafka and Apache Pulsar, and now building on ClickHouse® and Apache Iceberg. He is an Apache Pulsar committer and has contributed to the ClickHouse server itself, which makes him unusually well placed to talk about what these systems look like from the inside.

The conversation starts with consolidation. Shiv describes a project where combining products would have meant running eight different databases, and how a challenge from his lead to reduce that number led him to the conclusion that Postgres and ClickHouse together can cover most workloads. From there it moves to why Apache Iceberg matters, framed less as a technical preference than as a business one: nobody enjoys database migrations, and an open format means owning your own data and choosing your query engine later.

The second half is about AI, and Shiv makes a specific argument. Model Context Protocol servers are not enough. Agents query databases in a fundamentally different way than humans do, generating enormous variety at high volume with plenty of bad SQL mixed in, and that means the optimizer, the protocol layer, and even the disk layer all have to change. He also covers what AI coding tools mean for open source maintainers, why context graphs may unlock the reasoning behind decisions rather than just the decisions themselves, and what engineers should be experimenting with now.

Episode Highlights

• [00:01:18] What changed in the data ecosystem: from making sense of data being hard to AI making it easy
• [00:02:19] Eight databases down to two: the challenge that led to Postgres plus ClickHouse
• [00:04:27] The belief he abandoned: engineering is passion, but the business has to make money
• [00:05:39] Where ClickHouse shines: simplicity, and being able to run it on a laptop
• [00:06:51] The most common mistake: expecting every database to be transactional
• [00:08:59] Why ClickHouse’s architecture made Iceberg support a natural fit
• [00:11:08] Why Iceberg matters: nobody has ever told him they love database migrations
• [00:13:55] Whether Postgres and ClickHouse can converge on Iceberg, and why he would wait and watch
• [00:15:23] Agentic speculation: why databases were never designed for agent users
• [00:19:10] Throwing data away before you read it, and getting smarter about it
• [00:23:39] Extend the database rather than adding one, and reduce your dependencies
• [00:25:45] What you give up with a managed database, and why open source shows you what is happening
• [00:29:58] Whether AI coding tools give open source an even bigger advantage
• [00:32:36] The maintainer squeeze: lowered barriers, generated PRs, and an SDLC built for humans
• [00:35:01] Three-year predictions: bidirectional optimizers, natural language protocols, and context graphs
• [00:41:28] What engineers should be experimenting with: evals, gateways, cost, and reliability

Episode Transcript

Josh Lee [00:00:10]

Hello and welcome to a very special episode of Unevenly Distributed. I am here in person with Shiv from Nutanix, and we’re going to talk about Iceberg, ClickHouse® and AI. All three topics that I’m very excited about, and Shiv is an expert in. Shiv, just introduce yourself.

Shivji Kumar Jha [00:00:28]

Yeah. Hey Josh, wonderful to be here. I’ve been working for over a decade around databases in different capacity. First as a contributor of MySQL, then I worked around the Kafkas and Pulsars of the world, and now I play around ClickHouse, which is where we meet, and it is so good to now talk to you. Both of us in open source. And I think we can’t skip the conversation without mentioning AI. So it should be fun.

Josh Lee [00:00:54]

Looking forward to it. Yeah, absolutely true. Okay. So you mentioned you’ve been doing this for about 10 years, and in that time you’ve worked with a lot of different databases like MySQL, Pulsar, Kafka, and now you’re working with ClickHouse and Iceberg. So what has changed recently in the data ecosystem that made this stack of ClickHouse and Iceberg compelling to you?

Shivji Kumar Jha [00:01:18]

I think a decade ago we didn’t produce so much data, and when we had so much data we didn’t know how to make sense of it. We could, but it was hard. With AI now you can make sense of data so easily, and with the proliferation of social media and whatnot, you have so much data today.

So the data platform has also evolved to make sure it can support these things. To me, if you look at what MySQL did a decade ago, or what Postgres did a decade ago, to what Iceberg does today, it tells you a story of how far we’ve come.

Josh Lee [00:02:02]

Yeah. Okay. So you’ve written a lot about and you’ve talked about unified data platforms. So in the context of all what you just said, what does that look like and what does that mean in practice? Or is it just marketing hype?

Shivji Kumar Jha [00:02:19]

It’s something that I truly believe in. I think what engineers over time do is they pick the best tools for the job, which sounds amazing. I was in this place where we were looking at some products and we said we will combine these products and make one product, and when we looked at what databases we were using, this combined product would end up with eight different databases. And it’s not fun.

Josh Lee [00:02:45]

No.

Shivji Kumar Jha [00:02:52]

So one of my leads had a very good idea and he challenged me to reduce it to the least number of databases that I could to make all of these applications work. We spent some time, came back saying, you know what, Postgres and ClickHouse can together do most of the things that we are doing here.

Josh Lee [00:03:06]

Mhm.

Shivji Kumar Jha [00:03:13]

And then we set off on that journey and we are validated. Now we feel like Postgres and ClickHouse together can do most of the things, not everything. We have a few other things that we’ve had to use apart from these, mainly graph databases.

Graph. The GraphQL is, by the way, coming to Postgres shortly, hopefully in Postgres 19 or something.

Josh Lee [00:03:38]

Okay, I didn’t know that, but that’s exciting.

Shivji Kumar Jha [00:03:40]

Right. Then there’s something called Apache AGE, which is a graphing solution around Postgres. There’s something coming around there as well. So over time Postgres and ClickHouse are probably going to eat the database world in a way.

But yeah, that’s where I think you don’t want to manage too many databases. You want to make as few databases as you can and concentrate on your business.

Josh Lee [00:04:07]

Yeah, that’s absolutely true. I mean, even just for a little PoC I was building, I was tempted to bring in Redis just because I needed a few dictionaries, but you can make that work in ClickHouse or Postgres if you have one of those available to you.

Shivji Kumar Jha [00:04:25]

Yeah, absolutely.

Josh Lee [00:04:27]

Okay, so what has changed, right? What’s something that’s changed in your thinking? What’s something you believed maybe five years ago that you don’t believe anymore, that has led you to some of these decisions?

Shivji Kumar Jha [00:04:38]

I used to believe that I was an engineer very passionate about building things. Last five years has taught me that at the end of the day it’s about your business making money. So you have to be very strategic about what the business really needs, and make technology that is very, very simple and easy to operate.

Josh Lee [00:05:04]

Yeah, absolutely. I think people sometimes forget about that, right? That it’s about making money. I hear people complain when I need to work on legacy code bases. And someone else said, I forget who, legacy code is just another way of saying the code that makes our money today.

Shivji Kumar Jha [00:05:21]

Yes.

Josh Lee [00:05:22]

So that’s okay. Very good. Very good. You mentioned RDBMS and OLAP working together, but where does the, of course I’m interested in ClickHouse, I’m biased, where does OLAP, where does ClickHouse really shine? And where does it maybe struggle?

Shivji Kumar Jha [00:05:39]

I think I love ClickHouse for its simplicity. I have used other systems in the past where OLAP would have five different components.

Josh Lee [00:05:45]

Mhm.

Shivji Kumar Jha [00:05:50]

In fact, more. And the claim was that it is modular and it is easy to operate because you can scale everything independently from each other. Hey, I can’t run it on my laptop and experiment. I need an AWS account to run it.

Josh Lee [00:06:06]

Yeah.

Shivji Kumar Jha [00:06:07]

So ClickHouse makes me do that very easily. I can just download it and set it off. Most people come from an RDBMS background, from their academic side, and it’s very easy to get used to ClickHouse if you come from there.

If you are just looking at experimenting, it’s a different matter that ClickHouse is a very different database than a Postgres or MySQL, and they make that clear. But getting used to ClickHouse when you open it first is very, very easy.

Josh Lee [00:06:38]

I love to hear that. That being said, I don’t hear it from everyone. So what are some of the mistakes that you see people making when they’re getting started?

Shivji Kumar Jha [00:06:51]

Oh wow. This is an interesting topic that I love talking about. Maybe slightly controversial, but

Josh Lee [00:06:58]

Let’s talk about it anyway. I love talking about controversial things.

Shivji Kumar Jha [00:07:03]

That’s why we are here. Yeah. So I think a couple of generations of people have been taught databases via RDBMS, be it Oracle, MySQL, Postgres or whatever, and they have come to believe that transactions should work everywhere. They don’t explicitly say it, but deep within they think if it is not transactional, if the update doesn’t happen in the sync path instantly, then it’s not a database.

So that’s a very interesting thing that I’ve been working with, trying to make people understand that, but it’s not easy.

Josh Lee [00:07:50]

I can see why. You say that comes from a lot of experience. Do you have any tricks? How do you get those people to think about it differently, and what’s the correct way to be thinking about it?

Shivji Kumar Jha [00:07:56]

I think I start from an explanation of row versus column storage, and then I let the conversation flow. That’s my usual trick. If we spend an hour or two, it comes through quite well.

Josh Lee [00:08:14]

An hour or two. Okay. Maybe that’s too long. Maybe that’s actually pretty good. I guess it depends on the context.

Shivji Kumar Jha [00:08:19]

Opinionated, Josh.

Josh Lee [00:08:26]

I certainly agree. I cannot argue with that. Okay. Well, on that note, let’s talk about Iceberg and ClickHouse again. Something I’m starting to develop some opinions about myself. But you’ve got a lot of experience here. So you’ve combined ClickHouse and Postgres to support Iceberg. How was the experience? Was it harder than you thought it was going to be?

Shivji Kumar Jha [00:08:59]

It was a fun project, actually. I was looking at how ClickHouse has evolved as a database to support Iceberg format in addition to their own format, and I was also looking at how the Postgres kind of setup has done that.

And the idea basically was that I study at least two databases to see some sort of patterns. What I came to understand is ClickHouse is actually very well written architecturally to do this. I actually have done a full-blown talk on how you look at ClickHouse, how it is designed, and how it is such a natural fit if you had to extend it to something like an Iceberg. In that sense, if you open up the ClickHouse source code, it’s very structured.

I would take an example. For instance, MySQL has this thing where they say you can put up any storage engine and it should work. The problem with the MySQL world, in my opinion, has been that only one or two of those storage engines were developed by MySQL developers. Everything else didn’t work well for a production setting.

The unique thing I have found about ClickHouse is that most of these engines are developed by the ClickHouse developers themselves, and it works wonderfully well with ClickHouse. So you have a full buffet of storage engines for your use case. Just use it. It’s such a nice interface, and it just works for the use case you want it to work for.

And then there are all the other interfaces in ClickHouse that have been defined wonderfully well. If you look at the file formats that ClickHouse can read from, that’s again an interface which has so many implementations. And in the same sense, I think given their design, it’s been easier for them to support Iceberg.

Josh Lee [00:11:06]

Okay. And why Iceberg? What’s exciting there?

Shivji Kumar Jha [00:11:08]

Oh, this is a topic that’s close to both our hearts actually. We both love open source, and I could say then that we hate being stuck with a vendor.

Josh Lee [00:11:21]

Yes. Unless it’s the vendor I’m working for right now.

Shivji Kumar Jha [00:11:28]

Right. You’re very broken source that way. Yes. But I think on a more serious note, it’s terrible for a business if you are stuck in some ecosystems and they can take you for a ride.

In that sense, Iceberg is a huge promise to me. All my life I have seen so many databases. Everyone has a proprietary format where your data is locked in. And I am yet to meet an engineer who tells me they love database migrations. I’ve yet to meet anyone who tells me that.

So I think I love the idea of Iceberg, that you have an open format and you can use a lot of different kinds of engines to query data sitting in Iceberg format. That’s happening as we speak. Everyone is integrated with Iceberg, and it’s coming along quite well.

So my hope is that in future we will really own our data, and then we can have a choice of query engine, depending on, you know, there’s a team that comes, they love a certain query engine, they can bring that, and they can query that on the data that I have. So the openness, the interoperability are two things that I am really excited about. And then there are so many vendors working around Iceberg today, so my hope is that it’s going to result, into something that will be really good.

Josh Lee [00:12:57]

Yeah, I agree completely, and I especially love the interoperability. This is something I’ve spoken about a lot with OpenTelemetry, but I think the same thing applies to Iceberg, right? It’s not just about switching vendors. That’s important, yes. But it’s also that everybody gets to use all of these tools together, all the time, every day.

Shivji Kumar Jha [00:13:13]

Absolutely. So I’m an engineer. I love working with ClickHouse. So I use ClickHouse to play around data that is in Iceberg format, and then I have a data engineering team that loves Spark. So you bring your Spark cluster, you hook it here, and you use it, right?

Yeah. So we both love that protocol that we are married to, but our data is in a central place. Duplicating data is the hardest problem there. You don’t want to do it.

Josh Lee [00:13:42]

Absolutely. And do you see this also being something that can bring Postgres and ClickHouse together, like if you’re using them combined with Iceberg and maybe sharing the same data?

Shivji Kumar Jha [00:13:55]

I think it’ll take a bit of time. The two ecosystems, the way I see it, are both integrating with Iceberg as of today. I think both ClickHouse integration with Iceberg and Postgres integration with Iceberg are very early in their journey as it comes to being deployed in production for long periods of time. But my hope is that hopefully we can get there.

But the amount of difference between OLTP and OLAP is actually staggering. So I would wait and watch. I’m not very sure we can converge OLAP and OLTP, but if you can converge everything and you get all that ETL out of the window, it would lead to a lot of simple architectures.

Josh Lee [00:14:36]

Yeah, data platform Nirvana, I think. Okay. So let’s talk about AI. You mentioned it in the intro, right? We have to talk about AI. It’s the dominant topic these days. And for good reason, right? I think even six months ago, to me, it was a little bit of a toy.

Shivji Kumar Jha [00:14:56]

Yes.

Josh Lee [00:14:58]

And my thinking has changed a lot just in that time. So I still am excited about the data layer as a very, right, like to me I see the data layer as a way that we can assert control and sort of have this stable base on which to build these less stable things, let’s call them.

So what does that look like to you, and what does the data layer need to provide in order to support that?

Shivji Kumar Jha [00:15:23]

So I am sharing similar views as you, where I genuinely think AI is great in extracting meaning out of all that data that we had been keeping in the hope that someday we would derive information from it and make a big business.

Josh Lee [00:15:42]

Right.

Shivji Kumar Jha [00:15:48]

So we finally have something that can understand unstructured data very, very well. So that’s a great thing about AI, and then the developer productivity and whatnot. So AI is definitely amazing.

I have a strong opinion around the data stack there. I think the data stacks have to evolve quite a bit to support AI really well, and in a lot of different ways. I don’t think making MCPs are going to be the only thing that will help you get value out of the data stack and AI together. You’ll actually have to go deep, and I can take a few examples.

I recently read a white paper that was published by, I think, Five Volt, and they talked about agentic speculation. The way agents would work in the future is an agent in an app would go and try to fetch some information sitting in a database.

It’s very easy for agents to make a swarm of agents, and everyone in a coordinated way or in a parallel way goes about asking the database for information, or asking for data and turning that into information. But the database was not designed for the agent user. The database was either designed for a human user firing SQL queries. So you have a variety of SQL queries, but they come in very infrequent patterns. So humans are slow, systems are faster, but the queries are constant because you have tested applications. You only have so many queries and the database knows that these are the queries. The database can actually cache it.

Agents are a different breed. So the variety of queries that they will produce along these swarms of agents will be huge. So you have a lot of variety, and some of those SQLs will be bad SQL. So we have to protect our database against it. But agents are going to very quickly come with a lot of different kinds of SQLs, and some of them will be bad. So you’ll have to detect them midway and you’ll have to handle them.

So the data layer has to be updated now to support agents well, and it not only is on the query layer where you are talking about MCP. It comes down to the planning layer, or the optimizer layer in the database. It can also come back to the disk layer.

So a lot has to happen. You look at all of these things that are happening around the Postgres ecosystem where people want to fork the database.

Shivji Kumar Jha [00:18:38]

If the storage provided a wonderful way to fork the database, that would make the life of database developers easier. But I certainly believe that in every layer in the database stack, we have to do a bunch of things to make databases help AI derive value from the data that we have collected over time.

Storage is cheap. You can keep a lot of data, but computing on a humongous amount of data is still hard. One of the things that I think ClickHouse does really well is figuring out how to throw away all the data, not even see them when you are firing a query, right? Yeah. Why? It’s sparse index.

And we have to be smarter about it. And then as the database itself, we have to understand the characteristics of the agents very, very well and code them inside the database.

I think as an industry with SQL, what we have done is we’ve made SQL users a bit dumb. We’ve given this interface to them and they don’t have to go inside and look at everything. It just works. And now we can’t go back. We have to sort of make something similar for the AI world, where there’s a wonderful abstraction that just works. But it will take some time coming, I think.

Josh Lee [00:20:04]

Yeah, absolutely. And we started this conversation, you mentioned MCP, and everybody has an MCP server now, we have one too. We’re not unique in this regard. But I have a little bit of fatigue from these AI developments. By the time I finally came around to using MCPs, everyone was already saying, no, it’s skills now, don’t use MCP. And I’m like, come on. There’s too much money invested in the AI companies.

Shivji Kumar Jha [00:20:28]

Yes. Okay. And these are VC backed companies. You know how it works.

Josh Lee [00:20:36]

Yes. Yes, I do. So we have to be careful of the new things, but we also need to adopt them eventually, because they’re not all just smoke and mirrors. For the people who are super resistant, right? If people are trying to think about this in all of the old-fashioned ways for AI workloads, what breaks?

Shivji Kumar Jha [00:20:50]

I think if you go back to the concept of agentic speculation and zoom in a bit on speculation, the database developers never did speculation, right? If you had, as a human, to go and fire an SQL query on a production database, I’ve seen enterprise companies making this process where this will be reviewed by some humans, then this SQL has to be put on a Jira, this has to be tested in a sandbox instance and all of these. And agents are going to cook up SQLs on the fly. I am like you. I still don’t trust the text to SQL that are out there.

So I think this speculation that the agents will come with, and we are talking about one agent now, we are going to have multi-agent architecture. It’s already there. I see some five design patterns about multi-agent orchestration. These agents are going to call tools.

Josh Lee [00:21:45]

Yes.

Shivji Kumar Jha [00:21:47]

Right. And they are going to call tools in very weird ways. The tools are going to break, and the agent has to work very closely with the database. The database actually has to understand the persona of the AI agent really well.

The optimizer layer has to come only from understanding statistics and caches to understanding how AI agents work, and together work hand in hand as the agent keeps evolving and learning what it really needs to get from the database.

So in the past it was one way, where you go and ask the database and it gives you the result. In future, in my opinion, it is going to be both ways, where you give me an SQL query and I’m going to run it as a database, but then you suddenly come back and say, hey wait, that’s not the SQL I wanted, it looks weird. So I’ll give you another SQL, and maybe then I say that, Josh, sounds like a terrible SQL to run. So then you give me a third one.

So that’s something that has to come in the database world. But I am pretty confident in saying that a lot of things are going to break. But I think for the speed at which the software ecosystem is changing, it’s not wise to predict things today.

Josh Lee [00:23:11]

I couldn’t agree more, but I am looking forward to how all of this plays out. Okay, cool. So earlier we talked a little bit about simplifying your stack down to as few tools as possible, which is always attractive, not always feasible. Is ClickHouse and Iceberg, and maybe we’ll throw in Postgres, is that enough for what we’re trying to build, or do we still need more in our stack?

Shivji Kumar Jha [00:23:39]

I think it is enough. The motto that I follow is that I use open source because the source code is open. So if I have a little use case that’s not satisfied by a combination of Postgres, ClickHouse and Iceberg, then I will go and make it work and I’ll contribute. I’ve been using it for free for long, and I better go and contribute a bit to it.

Josh Lee [00:24:07]

Right. We appreciate that.

Shivji Kumar Jha [00:24:13]

I would rather go, and with AI coding tools it’s much easier now. So you would go and extend the database to do something than make one more database as part of your stack.

I am a big believer in having as few dependencies as you can, because dependencies are mostly like this black box that have huge implications on your architecture and eventually your product. God forbid something breaks really badly and it’s because of something that happens in the black box. Good luck on debugging.

So I am a firm believer in reduce the number of dependencies in your stack, and that also allows you to go deeper on those black boxes of your dependencies and understand how they actually work. So that’s, I think, a very good design pattern. If you have some time to make these databases work, you should use as minimum a number of databases that you can.

Josh Lee [00:25:18]

Okay. I mean, I love that you use the black box analogy, right? Because you’re someone who has a lot of experience, and something that I’ve spoken about a lot is how that experience gives us the ability to know when it’s time to open the black box and, like you said, contribute back, fix something, make it work for the use case.

For people who don’t have your level of experience, if they want to chase these open tools, what trade-offs are they missing? What are they ignoring?

Shivji Kumar Jha [00:25:45]

So if you have used a DBaaS, a cloud-based database, you have actually offloaded a lot of your problems to the vendor, which is really good in some settings. You are a small startup, you want to concentrate on only your business, and you don’t want to work on problems that you don’t want to make money out of. So it’s a good strategy.

But I think as you become a bigger org, I love using open source because it tells me exactly what is happening. As a systems engineer, it’s important for me to know that. So I don’t think, unless you have a very good feel of, let’s take an example, ClickHouse, you can make the best architecture that you can for your application on top of ClickHouse.

And what I have figured out mentoring a lot of young engineers is people are somewhat scared of making open contributions. They feel like it’s hard, it will not be accepted. It’s not. I think my faith in humanity in fact has grown from the business world. When I look at people from so many different cultures work together in a sane way, if you asked me before all of the open source world if they would come together to make such beautiful products in open source, I would have said no. People are monkeys and they would have done crazy things.

But the open source world actually handholds you really well. You go to pretty much any good open source project, you will have these tags like good first issues. There are certainly some in ClickHouse. That’s where I made my first contribution, to the ClickHouse server. So you can go there, you can take a stab. Now you have all these AI coding tools that will, if nothing else, help you understand code easily.

When I wrote my first patch for ClickHouse server, it had been a few years since I had touched C++. And lambdas, I’m a bit older for lambdas, right? When I wrote code all day long I didn’t use lambda a lot, it was not so popular. But the ClickHouse codebase uses a lot of lambda, so for the lambda kind of code I made AI write the syntax for me, which is so wonderful.

And I genuinely think, to be a good engineer using these databases, if you are using a hosted DB from a vendor, you should still go and look at the open source version of the same DB to get a sense of how that operates internally. What will this architecture lend to, to make it easier on the application side? What will it make harder for you to do in your application? To have that sense means you are designing your applications really, really well. Otherwise you may end up complicating, and writing software is the easy part.

Have you had an engineer tell you, Josh, that the previous engineer who wrote the code that they are working with was not a good engineer? They wrote terrible code.

Josh Lee [00:29:06]

I’ve never heard that before in my entire career. No engineer would ever say that about another engineer.

Shivji Kumar Jha [00:29:12]

Right. So I think you should understand these systems, these databases, really, really well internally, and then I think you would make much simpler applications.

Josh Lee [00:29:30]

Yes, I agree with all of that, and I especially love what you said about open source, right? Like we could all use a little more faith in humanity, I think. So that’s awesome.

I have two follow-up questions. I’ll ask the simpler one first. You mentioned how open source gives you this ability to open the black box and to understand everything inside. Do you think AI now, do you think open source has even more of an advantage now that people have this ability to do the analysis with AI on code that they’re maybe not familiar with?

Shivji Kumar Jha [00:29:58]

Absolutely. So the first contribution that I have done to a database in the post AI era, I used GitHub Copilot a lot. So I have this thing. I go and open a new open source project, and I just go to GitHub these days and I have this funny GitHub Copilot icon. You click on that and you just say, explain this file to me. You can be so lazy.

And I love Bill Gates for one thing. He said once that I would love to have engineers who are lazy. They would make good solutions.

Josh Lee [00:30:41]

Yes.

Shivji Kumar Jha [00:30:43]

Right. And I am certainly lazy that way. So these AI tools actually help you understand code very, very easily. They help you write code very quickly. They help you understand the syntax, or write a complicated syntax easily.

You know what? Today you can write a patch in a programming language that you’re not as familiar with. If you work with a couple of different kinds of programming languages, a third one is not so hard to pick up. And if these AI tools can help you write those, you test it well, it’s much easier than it used to be.

Josh Lee [00:31:22]

Yeah.

Shivji Kumar Jha [00:31:24]

Like I open up an open source project today and I go and tell it, go to the src folder and explain to me directory by directory what they do.

So one wonderful feature that you see in the Postgres codebase is that in every directory of the source code they have a README file that tells you what the files in that directory do. I love that, and I always thought that something like that should be there in every open source database. Now you don’t need that, because AI can do that for you.

Josh Lee [00:31:54]

Yes. Okay. Yeah. But maybe that README should still be there, so that the AI even can more quickly navigate to the right file. Maybe it’s just easier for open source maintainers to generate that and verify that and save.

Shivji Kumar Jha [00:32:05]

Yes. Yeah. I see such an effort in ClickHouse. I don’t know if it’s still happening, but I see such an effort that they wanted to do.

Josh Lee [00:32:12]

So that would be, yeah. Let’s make things easier for everyone, so we can be lazy and our AI can be lazy.

Okay. So my second follow-up question then is, with these AI tools, do you think that it has lowered the barrier for junior engineers to make open source contributions?

Shivji Kumar Jha [00:32:30]

Absolutely.

Josh Lee [00:32:31]

Okay.

Shivji Kumar Jha [00:32:33]

Shortest answer. 100%.

Josh Lee [00:32:34]

Okay. Do you think that creates any problems for the open source maintainers?

Shivji Kumar Jha [00:32:36]

It does. Lots of them. You look at GitHub uptime today, it’s not very pretty, and that’s because people are starting to generate code and put too many of these PRs on GitHub now. That’s my sense of it.

Open source maintainers, I’m sure, are fretting over reviews now. I have been in a situation where my developer generated a huge code base for a critical database feature that we wrote internally, and they didn’t know exactly what their code did file by file. And then I was like, if you do this and make me review this, maybe I should be called the developer of it. I’m spending more time on that, right?

So for the open source maintainers, I think this is slightly harder time. I envision a time when, so Josh, if you think deeply about it, GitHub, CI/CD, testing, apart from dev, all of the other parts of the SDLC process were made assuming that humans will write code. All of these eventually have to evolve to AI writing code. Till that happens, it’s not going to be a great time for open source maintainers, I think.

So I know that a few companies are already investing a lot in making reviews really good using AI.

Josh Lee [00:34:16]

Mhm.

Shivji Kumar Jha [00:34:18]

But it still has to come a long way. The testing has to be better. So right now I think it’s a bit of a harder time if you are a maintainer of an open source project.

Josh Lee [00:34:29]

That’s a great insight though. I had not considered before that connection between the tool changing

Shivji Kumar Jha [00:34:35]

Absolutely.

Josh Lee [00:34:36]

to make things easier for the AI, to also help the maintainers of open source projects.

Shivji Kumar Jha [00:34:38]

Absolutely.

Josh Lee [00:34:44]

Yeah, I like that a lot. Okay, cool. All right. I think we could probably talk about this for another hour if we wanted to, but let’s move on to another topic. Let’s move into the future. We talked a little bit about the past and how we got to where we are. What do you think the data stack looks like three years from now?

Shivji Kumar Jha [00:35:01]

Wow. The speed at which Anthropic is releasing models, it’s hard. And the way they have orchestrated this whole thing about the Mythos model, it’s even more scary as to what is coming along. So three years is a long time, but let’s do the prediction.

Josh Lee [00:35:25]

Let’s do it. Yeah, I’m putting you on the spot.

Shivji Kumar Jha [00:35:30]

So you asked me about three years of data platforms.

Josh Lee [00:35:36]

Yeah.

Shivji Kumar Jha [00:35:37]

Okay. So I think the one hypothesis I made was about the database optimizer changing quite a bit to support this bidirectional path. The query layer, I think MCP is not enough. It has to evolve a bit more. The protocol layer using natural language to query could come along really well. And hey, why do you want an MCP server that is separate from your database codebase? It could go and sit inside the database codebase. So that could happen on the protocol layer. The optimizer, we talked about it. And then storage engines and all those things, even the disk has to change.

If you come back on the top, I think graph databases are going to be a bit more meaningful. I don’t know if you’ve somehow managed to miss that whole discussion about context graphs. Did you manage to somehow miss it, or did you

Josh Lee [00:36:38]

I don’t know which conversation you’re talking about, but I’m seeing the value of these graphs.

Shivji Kumar Jha [00:36:44]

Correct. Yeah. Correct. So context graphs are basically built on this thesis that for all our previous decades that we’ve been doing applications, we always stored in our database the decisions. So if you think about a Postgres table, every row is an action that happened, a decision that was taken. But the conversation starting and then a row being written in Postgres, there’s a journey, right? How did you arrive at the decision that you made?

So all of this how is what the context graph is supposed to capture. And this is going to unlock a lot of value, because think of it, when you transition an engineer from one company to another, one team to another, what you do as part of KT, right, the knowledge transfer, you do a few things because you know that for however much documentation you have, however much code you have, you have the backup of engineers. But there’s a lot of things that is in the minds of these engineers that’s not written, but it’s important.

So once you capture all the how, the decisions were made, what discussions happened, all of that along the way, then it’s easier for an AI agent to go and look at the background context of this thing that you have in the Postgres database.

So I think context graphs are going to be important. Graph databases, I have actually tried a few of them. I’m not very impressed with any of them. Like if you asked me today saying, hey Shiv, you’re making a startup, what database do you want to choose? I could close my eyes and I could say just use Postgres. I love that database. If you ask me that about OLAP, I would say just use ClickHouse for now. And if you ask me about graph database, I don’t know.

If there is a vendor that was doing graph really, really well, if they did a decent enough job to market that to somebody like me and you who’s been in this space for a bit, then we should have known. Either there’s a good product that’s not marketed well, or there is no good product which just works.

So the graph databases have to come along really well to support this context graph, and a lot of other things around chatting with your data, making graphs on top of your data. Where three years down the line, I don’t know if we will still have a Grafana dashboard kind of a thing where you hook up a data source and that data source is set up behind with some database or something.

So I think those visualizations could be done in a chat box, which is amazing, because back in the day, how did the world work, Josh? So an exec tells their analyst team that they need a certain view of a graph, and then the moment the exec looks at the graph they’re like, okay, but then I have the next question. And then do more history at the next graph.

Josh Lee [00:40:13]

Yeah.

Shivji Kumar Jha [00:40:15]

Yeah. So the database will change, and it will change in different ways. The AI persona in every layer and all of that, the protocol layer has to change. A lot of use cases have to become part of the database, as to you can talk in natural language, you can ask questions, you can plot graphs, you can make decisions.

So databases have mostly been constrained to only giving you data. Can they help you make a decision? This is something that I’m looking forward to. Can you just talk to the database and make a decision?

Josh Lee [00:40:49]

Yeah, it’s like a git blame for organizational decisions.

Shivji Kumar Jha [00:40:55]

Absolutely.

Josh Lee [00:40:57]

Yeah, that would be really cool. If you’re listening and you want to build this, this sounds like a cool opportunity. Talk to me about it, because I’d love to hear about it. But yeah, that is a gap that you’ve identified for sure.

Okay. Well, I like to ask a few spicy questions at the end to wrap things up. So let’s get a little spicy. We’re in Bangalore, so it’s only appropriate. What should more engineers be experimenting with today that they’re not?

Shivji Kumar Jha [00:41:28]

If you look at the AI space, a lot of people are building apps. That’s great for demo. Building production apps is a completely different challenge. So you have all these evals, AI gateways and all of these things that you should definitely know. It’s not enough to build an app with AI. I think that’s going to very quickly be commoditized.

So if you are an engineer, and an engineer is experienced, you should really know how to run these things reliably, cost effectively, in production. Cost is a big deal. I still see a lot of posts on social media where a certain CEO allotted a budget to use AI. We are not even midway in this year and the results are exhausted, because tokens are not cheap. GPUs are not cheap, right?

So cost optimization has always been a very essential part of engineering, because engineering is a big cost on the balance sheet. So we have to really optimize them. What I’m looking forward to is, can we make reliable something close to the deterministic apps that we used to have in the past?

The other thing I think often about, Josh, as an engineer, is that there’s this concept of Delta 4. If you go beyond Delta 4 to a Delta 8, chances are that your consumer will not be able to accept that change. There’s only so much change we can take at a point of time as a human.

And in the past we make these apps where all the code is written down and it just does the same thing every single time. So they trust this technology. Now you suddenly have a huge delta, delta to the rough, delta 20, 40, 100, I don’t know, where you’re going to have agentic architecture and these agents are going to create a havoc. They are going to do a lot of different things. Agents don’t promise you that they will make the same decisions all the time.

In this world, as an engineer, it’s very interesting to think, you know what, my customer is still the same. My user of the product still expects to get stuff done with technology. That’s the whole job technology is trying to do. And the job still has to get done. So you have to be reliable. How do you make these AI based, agent based apps as reliable as technology was in the pre-era, right?

And then we talked about all of the changes that the database market will go through. There are other markets that will go through changes. Observability is a topic that you are deeply interested in. I have spent some time around agentic observability. That’s a whole different game to the system observability or business observability that we did in the past.

I see a lot of observability happening around AI, where, much like an API gateway, you have an AI gateway today. On the AI gateway, you make some insights. Maybe if you look at a product, for some feature you say I’ll not use the most powerful model, a lower model is fine. You make a router which decides where I need to use a very powerful model, where a normal model is fine, and all of that. That is one aspect of it.

Then there are other parts where, in the past system observability and business observability were very different things. Similarly, I think only deciding that I have to use this model for this task is not enough. You have to go inside your product, you have to understand what your agents are doing over time, you have to collect all of the data, and then you have to make sure you are making the most cost-optimal decision for the design of your app. And I don’t see a lot happening in a generic sense. That’s a gap that engineers should look at.

And there are a lot of other opportunities. For instance, I don’t know if you have seen a prescription that a doctor writes in India. It’s a handwritten note and the doctor writes anything in any corner of the paper. It’s handwritten, so it’s hard for a model, but the handwriting is terrible. As a human I can’t read it, but the LLM models are actually very, very good at reading these things.

So my hope is that amid all the gloom saying AI will replace a lot of engineers, what is going to also happen is that a lot of places where we could not make technology do things will be an extra field where technology is being used. So that space is going to open up. We’re going to make technology do a lot more things than it could in the past.

So the hope I have, Josh, is that engineers should go learn some of these things. And I really hope, as well as wish, that a lot of people are losing jobs, so that’s terrible. But it’s a thing in transition. Eventually we’re going to have a lot of newer jobs, and then it’s going to be a better world.

Hopefully, Josh, we don’t have to be five to six days in office, 12 hours. That would be lovely. You can take more holidays. You can go out with family. You can go have a

Josh Lee [00:47:29]

But the specific example you gave was so great, right? Because earlier we talked about open source giving us faith in humanity, right? And your example has given me faith in the machines as well.

Shivji Kumar Jha [00:47:36]

That’s a wonderful way to go.

Josh Lee [00:47:42]

I think that’s the best note to end on, right there.

Shivji Kumar Jha [00:47:46]

Right. Thanks. Wonderful talking to you.

Josh Lee [00:47:47]

Thank you. It’s wonderful to have you.