Datatypes Are Awful – Use Arrows

Introduction

In this episode, Josh and co-host Robert Hodges, CEO of Altinity, are joined by Matt Topol, co-founder of Columnar and a PMC member on Apache Arrow, Apache Iceberg, and Apache Magpie. Matt wrote the first book on Arrow, In-Memory Analytics with Apache Arrow, which he is quick to note is full of puns, including a chapter heading about ODBC taking an arrow to the knee. Robert, who has been working on databases since the early 1980s and worked on connectivity protocols himself, turns out to be an ideal interviewer for this one.

The core argument is that the industry spent two decades making databases faster and left the wire alone. Analytical engines went column-oriented, adopted vectorization and SIMD, and then handed their results to transport protocols still designed for rows. The result is that data gets transposed out of columns into rows to leave the database, then transposed back into columns on the client for the DataFrame or visualization library waiting to receive it, paying CPU and a full copy at each step. Matt traces the argument to the 2017 VLDB paper by the future DuckDB creators.

Arrow is the fix at the format level, a columnar in-memory representation whose bytes on the wire are identical to its bytes in memory, so there is nothing to serialize. ADBC is the fix at the client level, a deliberately familiar ODBC-style interface defined in terms of Arrow data, with around twenty functions instead of hundreds. The conversation covers record batches, memory mapping and shared memory, Hugging Face, NVIDIA RAPIDS, RDMA and the experimental dissociated IPC protocol, adoption at dbt and Power BI, and a long shared lament about data types.

Episode Highlights

• [00:00:59] Matt introduces Columnar, three Apache PMCs, and a book full of puns
• [00:04:14] What Apache Arrow is, and why serialization was eating the CPU budget
• [00:06:56] Robert on being present at the creation of tabular data stream in the 1980s
• [00:07:29] The 2017 VLDB paper by the future DuckDB creators on client protocol redesign
• [00:08:23] Column-oriented engines feeding row-oriented transports, and paying twice to transpose
• [00:10:31] ClickHouse had columnar connectivity early, but the spec was the C++ source
• [00:11:07] Interoperability: DuckDB, Snowflake, BigQuery, and Polars are already Arrow
• [00:12:22] What ADBC is, and why it deliberately looks like ODBC and JDBC
• [00:15:18] Record batches, and what a result set stream actually looks like
• [00:16:44] Why the format is contiguous, and how that unlocks SIMD
• [00:17:54] NVIDIA RAPIDS is Arrow on the GPU
• [00:20:47] Streaming results without materializing the whole set first
• [00:21:38] The dbc CLI and the Driver Foundry: a package manager for ADBC drivers
• [00:24:44] Data types: better or worse for humanity, and why timestamps break everything
• [00:30:11] ODBC has hundreds of functions to implement, ADBC has about twenty
• [00:31:45] Five to twenty times faster, and why expose Arrow if you are building a database
• [00:35:50] Memory mapping, shared memory, and how Hugging Face caches data sets
• [00:37:47] The dissociated IPC protocol, RDMA, and UCX
• [00:42:24] OTel Arrow: faster transport and smaller payloads without JSON conversion
• [00:46:06] Adoption: dbt core v2, and Power BI replacing its drivers with ADBC
• [00:52:05] The Trino complaint, and a 400x PR that died on the vine

Episode Transcript

Josh Lee [00:00:10]

Hello, hello, hello, and welcome, everyone, to this episode of Unevenly Distributed. We’re here to talk about data, databases, AI, and all things related. I am one of your hosts, Josh Lee. I am joined by my co-host, Robert. Hey, Robert.

Robert Hodges [00:00:24]

Hey, Josh. How’s it going? Hey, Matt. It’s great to connect with you.

Matt Topol [00:00:32]

We’re already way down the road. The punishment has begun.

Robert Hodges [00:00:38]

Yeah, let the punishment begin. So, just the background on me. I run Altinity, but I’m a total database geek. I’ve been working on databases since the early 80s. And I love this topic that we’re going to be discussing today.

Josh Lee [00:00:53]

Yes. And with that in mind, Matt, hello, welcome. And would you like to introduce yourself?

Matt Topol [00:00:59]

Yeah. So I am Matt Topol, co-founder at Columnar. Also PMC member of Apache Arrow, PMC member of Apache Iceberg, and PMC member on Apache Magpie. We’ll get back to Magpie later. It’s new.

Robert Hodges [00:01:20]

Yeah, I remember that one.

Matt Topol [00:01:22]

But anyway, we, Columnar, we focus a lot on data connectivity using Arrow, particularly with ADBC, which we’ll talk about. Apache Arrow itself, if you’re not familiar with it, don’t worry, we’ll cover that in a second.

And then also I wrote the book on Apache Arrow, In-Memory Analytics with Apache Arrow, go check it out. Great puns. I love my puns. The book has lots of puns in it. One of my personal favorites, ODBC takes an arrow to the knee.

[laughter]

That is one of the chapter headings.

Robert Hodges [00:02:02]

Okay. Where is this book available?

Josh Lee [00:02:04]

You were going to say that, but I think we might want to get that out in front so people can just order one while they’re listening.

Matt Topol [00:02:09]

I mean, Amazon, Packt. It’s published by Packt, but it’s available on Amazon, available from Packt’s site directly. If you have an O’Reilly subscription, it’s available to read on O’Reilly’s site. Pretty much anywhere you can get technical books, you can probably find it.

Josh Lee [00:02:24]

Cool. Just because we were having fun with the puns, one of my other favorites in that book, chapter headings, is TL;DR, computers are magic.

Matt Topol [00:02:38]

Right before I explain how virtual paging works.

[laughter]

Robert Hodges [00:02:44]

Yeah, that’s, I think chapter headings and the little quotes are kind of an underrated part of literature. My all-time favorite was in, I forget which book it’s in, but it’s in some weighty computer science tome, and the chapter begins with a quote from Alfonso, the something or other, king of Naples. And he says, “Had I been present at the creation, I would have provided some hints for the better ordering of the universe.”

Matt Topol [00:03:15]

There you go. Yeah, I mean, who hasn’t had that feeling when they looked at a piece of code?

My primary goal when I was writing the book was to make sure that it is not a dry text, to make sure that reading it is interesting and keeps your attention, because I’ve gone through lots of dry technical books and they’re awful.

And so the book is very much written in my tone, and I have a lot of fun with jokes, comments, and things as I go along, hopefully. And that’s one thing that’s been complimented, the fact that it is engaging because it’s not a dry text. So I highly recommend, if you’re interested in this topic further, absolutely the book is a great place to start, especially if you’re not that familiar with Arrow already.

Robert Hodges [00:04:07]

Yeah, so that gets us into the whole topic of today. What is Arrow and why is it a big deal?

Matt Topol [00:04:14]

Yeah. So Apache Arrow is a column-oriented in-memory data format. The key thing to remember is that Arrow itself is a spec. It’s an Apache project. It was started in 2016, co-created by Wes McKinney of pandas fame, and also co-created by a few others like Julien Le Dem, Jacques Nadeau, other really big name data people.

And one of the things that it was created to do is to address the fact that, at the time, Hadoop was the big thing, and people were realizing that, well, Parquet was out already as your storage format, and that was column-oriented. And then as you passed data from node to node, from system to system, every single system had its own format internally of how it represented the bytes, and you spent more CPU cycles just serializing and deserializing the data as you went from node to node or system to system than you did on the actual computations.

So Arrow came about as a way of getting this kind of unified in-memory representation that is still extremely efficient to operate on, but that on the wire is identical to the in-memory bytes. So you don’t have to serialize or deserialize it at all. Just grab the network buffer, for all you care, and just use the buffer directly.

And so being column-oriented, you get all the benefits of really simple and easy-to-use vectorization and SIMD when you operate on it. And there are implementations for Arrow in, name a programming language, there’s probably an implementation for Arrow.

And so for the project, we manage the format, the spec, but also official implementations in – I think we had like 13 to 15 languages that have implementations of Arrow. And then all the CI is set up so that you can be sure that if you’re using two implementations of Arrow, or using multiple languages, that if you pass the data out of one system, no matter what programming language you’re using, you can then receive and use that data and process it and interact with it without any issues, and super efficiently, because like I said, you’re cutting out that serialize and deserialize step.

Robert Hodges [00:06:56]

Yeah. And I think it’s useful, maybe you could talk a little bit about what came before and why this is such a break. Because TCP/IP based communications between databases and clients has been around since the 80s. I was personally present at the creation at CIA with tabular data stream, and it was a big step forward. But what you’re doing is you’re dealing with a new set of problems. What was the older way of moving data around between databases and clients, and why is it broken now?

Matt Topol [00:07:29]

Well, so there was a great paper in 2017, a great VLDB paper by Hannes Mühleisen and Mark Raasveldt. And if you don’t recognize the names, they’re the guys who went on to create DuckDB.

Robert Hodges [00:07:47]

Ah, yeah.

Matt Topol [00:07:48]

And they published this paper in 2017 for VLDB called Don’t Hold My Data Hostage, a case for client protocol redesign. And that whole paper was based around the idea of pointing out that while we’ve spent so much time in the last couple of decades improving database performance, improving CPU usage, memory usage, improving application performance, almost no one has spent a lot of time on the data transport part of it, right?

And so what’s ended up happening is that we’ve improved all of these systems to use vectorization, SIMD. Your analytical systems are almost all column oriented, but our transport protocols are still stuck in the 80s with row-oriented transport protocols.

Robert Hodges [00:08:38]

Mhm.

Matt Topol [00:08:41]

And even when they’re using binary instead of like a JSON thing, it still ends up being row oriented, and you’re paying a huge cost to transpose all of those columns into rows as it leaves the database.

Then your client gets it. And then most of the time you’re feeding these into dataframes and visualization libraries and all these other things, which are of course column-oriented for the same reasons. And so you’re transposing it all back.

Robert Hodges [00:09:15]

And in cost terms, not only is that the CPU usage of doing the transposition, it’s a copy.

Matt Topol [00:09:23]

You know, if you send a million rows, you’re sending a million rows, then you copy them to transpose it, and then you send it across the network, and then you copy to transpose it, and so on.

And so all of the common ways are either proprietary, a proprietary format that is just whatever they did, and so you can’t really build an ecosystem on it, or they’ve been row-oriented things like JSON, like Protobuf. And even while Protobuf can represent a column oriented thing, everyone still represents the tabular data in rows, because it’s just a natural thing, even though all of the advancements in hardware and software have made us, when we’re doing analytical systems, use columns. And now paying that cost.

Robert Hodges [00:10:17]

Right. And when you say proprietary, it doesn’t necessarily mean that it’s locked up by some database vendor and you have to pay licensing fees. It’s just enough that it’s a format that’s not documented.

Matt Topol [00:10:29]

Well, yeah.

Robert Hodges [00:10:31]

And I think ClickHouse®, which I love, and you’re, I think, well familiar with at this point, has had column based connectivity for almost since its inception, but it didn’t really have a spec. Well, the spec was the C++ files that implement it. You want to know how it works? Go read the code. Is there a problem here? For a long time, we didn’t think there was.

Matt Topol [00:10:56]

But ClickHouse has also, a long while ago, like a really long while ago, they added support for Arrow input and output.

Robert Hodges [00:11:06]

Yeah.

Matt Topol [00:11:07]

And that’s the key. The key is the interoperability that Arrow provides. DuckDB outputs and inputs Arrow zero copy.

Robert Hodges [00:11:15]

Yep.

Matt Topol [00:11:17]

Snowflake returns Arrow data if you wanted to. BigQuery returns Arrow data if you wanted to. ClickHouse already said. Polars, the dataframe library, is just Arrow under the hood. It’s all already Arrow, which means you get your data from whatever your source is, and if it’s in Arrow format you can just wrap it with Polars and not have to do any copies.

Robert Hodges [00:11:44]

Right. Right. And so you’ve explained Arrow. Can you talk a little bit about what ADBC is? For me as a programmer, because that’s where I’m going to touch the connectivity layer, I’m going to get a connection, and I’m going to put some data in, I’m going to ship it off. Tell me a little bit about how that works.

Matt Topol [00:12:04]

Absolutely. One of the funny things about Arrow is that because Arrow is such a low-level technology, most application developers only ever interact with it through other applications that are using Arrow under the hood.

ADBC is Arrow Database Connectivity. It sounds familiar to ODBC or JDBC. And you’re right, it is intentionally familiar, because the stated goal is very similar.

Back in the late 80s, early 90s, ODBC was created because you have all these databases and you have all these systems and they all had their own ways of interacting, and application developers were like, well, this is annoying. Every time I want to talk to everything, I need to code an entire new interface. And so ODBC was created, technically owned by Microsoft, as a way of unifying the systems to have a client interface.

Drivers implement the interface. Application developers code against the interface. And it made perfect sense at the time. And all of the interactions are row oriented because that’s what you did. That’s how it worked back then. JDBC came along a little bit later because everybody wanted a native Java thing for it, because reasons.

And so ADBC is a similar concept. It is a defined client spec interface for interacting with databases. The difference being that it’s defined in terms of Apache Arrow data, and thus the data that flows both ways through the system is column oriented. If your data source returns Arrow, or returns columnar data, that means you get to avoid those transposes that you would otherwise have to do with ODBC.

And of course it’s Apache licensed and so on and so forth, and you have a client interface definition which is basically just the C header file.

Robert Hodges [00:14:09]

Mhm.

Matt Topol [00:14:13]

And then you have drivers that implement that interface. And then you have driver managers, bindings in the various programming languages, so that you can load any arbitrary driver you want and have nice native ergonomic interfaces to actually interact with it.

And so you just code against the interface, load whatever driver you want on the fly, and interact with whatever data source you want. And no matter what the source is, you get Arrow data back.

Robert Hodges [00:14:39]

And am I as a programmer going to see familiar connectivity constructs, like the notion of a connection, for example?

Matt Topol [00:14:45]

Absolutely. The notion of a query and a result set, and the notion of explicitly designed metadata.

Robert Hodges [00:14:52]

Yeah. But structurally you have a database object.

Matt Topol [00:14:56]

Yep.

Robert Hodges [00:14:57]

That holds driver global state and things like that.

Matt Topol [00:15:02]

Yeah.

Robert Hodges [00:15:02]

You have a connection object.

Matt Topol [00:15:03]

Yep.

Robert Hodges [00:15:04]

Which represents just a single connection. You have a statement object.

Matt Topol [00:15:08]

Yep.

Robert Hodges [00:15:09]

And then you can call set query on the statement object.

Matt Topol [00:15:13]

Execute, execute update, whatever. You get back a stream, right?

And a stream in this case is just a stream of what’s called Apache Arrow record batches. The record batch you can think of just like a slice of the result set.

Robert Hodges [00:15:27]

Yep.

Matt Topol [00:15:28]

It’s all the columns, some chunk of rows, and you just get a stream of those record batches.

Robert Hodges [00:15:35]

Ah, interesting. So the rows are the whatever columns you selected, Arrow will take care of aligning them so that from a programming perspective you can think of it as something you can process as a set of rows. Is that what you’re saying, essentially?

Matt Topol [00:15:50]

Yeah, I mean, you’re going to get back, it’s going to tell you how many rows were given.

Robert Hodges [00:15:56]

Yeah. Yep.

Matt Topol [00:15:57]

And you get back that record batch, which is just basically, here is the group of columns, right? And you’re guaranteed that for a given record batch all of those columns have the same number of rows and they’re aligned, right?

Robert Hodges [00:16:11]

Now, is there, and excuse my ignorance, but one of the things that’s interesting about columnar formats is that they are vectors and they can be just tossed onto SIMD processors, or processed using SIMD instructions. Is there a way to apply SIMD even at the low level where you’re dealing with this stream coming through? Does it give you the opportunity to use that, or are you still doing it on conventional CPUs?

Matt Topol [00:16:44]

You get that benefit because of the Arrow format. It is a column oriented in-memory representation, so you’ve now got a bunch of little operations you want to perform.

Every column in a record batch is a contiguous chunk, right? I mean, it follows the format. For example, a string column is actually two buffers. It’s one buffer of integers that give you, for a given index, start and end, which are indexes into the other buffer, which is just all of your string data as one continuous thing, right? That’s how the Arrow format does strings.

Robert Hodges [00:17:29]

Mhm.

Matt Topol [00:17:31]

But all of it is always contiguous like that, specifically so that you can leverage SIMD optimizations and SIMD instructions for any processing you want to do after you get it back.

Robert Hodges [00:17:42]

Cool.

Matt Topol [00:17:44]

Yeah. That’s why things like P is used internally.

And actually one of my favorite examples of that is NVIDIA RAPIDS.

Robert Hodges [00:17:52]

Mhm.

Matt Topol [00:17:54]

If you’re familiar at all with the NVIDIA RAPIDS ecosystem of GPU dataframes and data processing for NVIDIA GPUs, NVIDIA RAPIDS is Arrow on the GPU.

Robert Hodges [00:18:08]

Got it. So you can basically have a client pulling Arrow and then pushing it over to a GPU. Is that the basic, where you can then begin your

Matt Topol [00:18:16]

And then the entire NVIDIA RAPIDS cuDF, that whole ecosystem, operates on Arrow formatted data on GPUs.

Robert Hodges [00:18:26]

Right. So I’m just trying to think how to describe this in terms of a metaphor, because I’m thinking, the TCP/IP based sort of row based connectivity we had in the 80s, it’s like Cyndi Lauper. Who I really liked.

[laughter]

But she was big in my youth. What do you think is the equivalent today? What would we want to compare this to?

Matt Topol [00:18:54]

Well, I would say Arrow is more just like, you know, Charli XCX.

Robert Hodges [00:19:00]

I was just trying to think who would be, how can we use musical? Josh is going to lose it here in a little bit.

Matt Topol [00:19:09]

It would be more like, what do I call it, like a particular vocal track that everyone samples.

Robert Hodges [00:19:27]

Yeah. Oh yeah. I know what you mean. That one EDM sample that is

[laughter]

the one thing that everyone samples.

Matt Topol [00:19:41]

Yeah, that always, that you can always pick it out because it’s right there, but no one ever uses it direct on its own, right?

[laughter]

Because it enables all of this interaction, interoperability.

Robert Hodges [00:19:56]

Yeah, I’m going to submit Charli XCX Speed Drive as my proposed song that you should listen to while you’re coding quickly and processing data quickly. That was my favorite earworm from a couple years ago.

[laughter]

So listen to that and then dive into our ClickHouse SimCity.

Matt Topol [00:20:21]

Yeah, exactly.

[laughter]

Robert Hodges [00:20:22]

For those who haven’t seen it, if you go to Altinity demos, there’s a ClickHouse, I don’t actually know if it’s on the demos page yet, but look for ClickHouse SimCity, you’ll find it.

Matt Topol [00:20:32]

Yeah. You need to run it on a fast processor and just dial it up.

Robert Hodges [00:20:40]

Yeah. Well, so this is definitely a complete… what’s interesting is this streaming process here. Just all kidding aside, that’s where the connectivity experience becomes really different.

Matt Topol [00:20:47]

I mean, because one of the big benefits is the fact that you can get this streaming interaction where you don’t need to materialize the entire result set before you start operating on it.

Robert Hodges [00:21:01]

Yeah. Exactly.

Matt Topol [00:21:03]

Exactly. And that enables that really easily.

It also, so I don’t know about you, but have you ever actually dealt with the pain of trying to find, download, and install an ODBC driver?

Robert Hodges [00:21:21]

Oh, I try to, not for a while. I mean, God knows I’ve dealt with a lot of JDBC. I was a Java programmer. We’ve gone through the cycle of everything use JDBC, and then because that’s got the repetition being slow, we went back to using separate connectors for everything. Yeah, that was not good.

Matt Topol [00:21:38]

ADBC is trying to bring us back to that universal connector thing, right?

Since there’s all of that pain on trying to get the actual drivers themselves, we at Columnar built out a package manager for ADBC drivers. It’s an open source utility. It’s called dbc.

You can install it pretty much any way you want. You want pip, install it, go for it. You want to uv, install it. You want to npm install it, whatever. The point is it’s a little CLI. We like to call it uv for ADBC drivers.

At Columnar we set up a CDN and we set up what we call the Driver Foundry. If you go on GitHub, github.com, the ADBC-drivers organization, where we can collect all the contributing ADBC drivers, having shared infrastructure, shared validation suite, shared utilities and so on. And then we build them, sign them, put them on the CDN, and then you can use dbc to install any of those drivers from the CDN at a known place that all the driver managers know where to look.

Robert Hodges [00:22:52]

Mhm.

Matt Topol [00:22:56]

And so installing drivers for ADBC is as simple as just doing dbc install snowflake, dbc install clickhouse.

Robert Hodges [00:23:06]

Right. Right. And so maybe you could just tell us a little bit about what the Columnar business is? Because you’ve got a company that’s clearly built on connectivity in the worst way. I mean, because that’s your heart and soul.

Matt Topol [00:23:23]

Oh yeah. So Columnar is enabling people to distribute drivers in a way that looks kind of like that little ODBC, those little crazy ODBC popups that you still see bubbling up from the depths of Windows.

[laughter]

dbc to install ADBC drivers on the CDN.

Robert Hodges [00:23:47]

So you go searching and finding those popups. Right. Well, I’m really glad you’re doing that, because again, kidding aside, the thing about ODBC that was so great was that you had a single interface. Obviously that was the same with JDBC, which I used for decades, and is what we’re doing with ADBC too. And that did more for developer sanity than almost anything I can think of, at least in the database field. Just having a single interface that worked portably across a bunch of different implementations.

Matt Topol [00:24:18]

Exactly. It’s way more portable. In fact, ADBC and JDBC is way more portable than the SQL that you’re actually using.

Robert Hodges [00:24:25]

Oh, yeah.

Matt Topol [00:24:26]

ADBC doesn’t make any attempt at the SQL interactions.

Robert Hodges [00:24:31]

Yeah. It’s like JDBC or ODBC. Have to use the right SQL dialect.

Matt Topol [00:24:36]

Yeah. That’s an exercise for the reader. But that’s just called using SQL.

Robert Hodges [00:24:44]

Yes.

[laughter]

But since we’re down fairly low in talking about low-level connectivity, data types: do you think they’ve been better for humanity or worse? And you as a connectivity person should be able to comment on this.

Matt Topol [00:25:05]

I mean, data types are the bane of my existence. When you’re trying to deal with, one of the benefits of ADBC is that the drivers do the mapping of the database type to the Arrow type system for you.

Robert Hodges [00:25:22]

Mhm.

Matt Topol [00:25:23]

Rather than you having to do that mapping yourself. This is important because it is not easy to do in any way, shape or form, right? Because database systems don’t necessarily agree on types.

Robert Hodges [00:25:40]

Mhm.

Matt Topol [00:25:42]

And sometimes they don’t agree with themselves, or they say they support ANSI SQL and then they don’t, or the ANSI SQL is the minimum and they have all these extra things on top of it. Data types are awful.

Robert Hodges [00:25:58]

Yeah. For something that’s the title of the episode. For something that is that simple. It’s just like, hey, an integer. Well, how difficult can it be to represent an integer? Well, let me bring up the subject of signed versus unsigned. Let’s talk about that.

Matt Topol [00:26:16]

Actually, the integers themselves, for the most part, aren’t too big a problem. What really trips up a lot is when you start having more semantic interactions, like timestamps.

Robert Hodges [00:26:28]

Yes.

Matt Topol [00:26:30]

How you represent the timestamp affects the range of timestamps you can represent.

Robert Hodges [00:26:38]

Exactly.

Matt Topol [00:26:39]

And so the Arrow format base timestamp is a 64-bit integer, and then you attach a unit to it.

Robert Hodges [00:26:50]

Yeah.

Matt Topol [00:26:51]

So it’s a 64-bit integer and it can either be nanoseconds, milliseconds.

Robert Hodges [00:26:55]

So it’s variable precision, in other words.

Matt Topol [00:26:57]

Yeah. The problem is ANSI SQL defines timestamps as having a certain range regardless of their precision. And so the Arrow format, for example, can’t represent the full ANSI SQL representation in nanoseconds only. Now, Arrow does have a thing to represent it as a separate type.

But then you have a database where they have their timestamp type that they return using a structured thing of epoch offset and this and that, along with the precision, and it’s like, okay, now how do I map that over to this thing?

And then you have numerics, which is the worst part, because when you get the data back: is that an integer? Is that a float? Is it a float 64? Is it a decimal 128? Is it decimal 256?

Robert Hodges [00:28:01]

Right. And we’re not even talking, you’ve brought up timestamps. We haven’t even mentioned time zones. And I think, to keep this thing from just going to a really terrible place, we shouldn’t even talk about it.

Matt Topol [00:28:13]

Yeah. It’s not a horror podcast.

Robert Hodges [00:28:16]

Data types

[laughter]

data types suck.

Matt Topol [00:28:20]

Yeah. But again, that is one of the benefits of having this unified

[laughter]

that sucky experience is partaken of by somebody who wrote the driver and not you. And that’s generally us, but also, I mean, Databricks contributed the Databricks driver.

Robert Hodges [00:28:37]

Yeah.

Matt Topol [00:28:39]

We helped ClickHouse with their ADBC driver. It was primarily written by the ClickHouse engineer.

Robert Hodges [00:28:44]

Yeah.

Matt Topol [00:28:45]

Same thing for the BigQuery driver, which was contributed by a community member. Some of our ADBC drivers are just straight up open source in the Apache Arrow ADBC repo because they were just part of the project there. Others were written specifically by Columnar. And right now, at this point, the drivers we have, we can support maybe 30 to 40 different data systems.

Robert Hodges [00:29:14]

Mhm.

Matt Topol [00:29:16]

Because if it has the same wire protocol, it can use the same driver, right? So we have a Postgres driver. Anything that uses the Postgres wire protocol can use the Postgres driver, right?

Robert Hodges [00:29:27]

Right. And so because you don’t have dependencies on like information schema or something like that, because that seems to be where specifically does have a dependency on that, and therefore

Matt Topol [00:29:40]

Yeah. Because with the weirdness of, that’s one of the little nasty things about wire protocols, is when the client connects, yes it can make a connection, but now it thinks there’s specific metadata it can go look up in tables, and sometimes it’s there, sometimes it’s not.

And so the ADBC APIs also expose a bunch of introspection functions, if you need them, to safely use.

Robert Hodges [00:30:06]

Yeah. Yeah.

Matt Topol [00:30:11]

A lot of it was inherited and designed to be similar to ODBC, JDBC stuff that people are familiar with, right? Except much, much smaller, because the biggest problem with ODBC, aside from the whole row versus column thing, is that the API surface is insane. If you’ve ever tried to build an ODBC driver, I’m sorry.

Robert Hodges [00:30:36]

Oh, this is

Matt Topol [00:30:38]

This is literally just hundreds upon hundreds of functions you have to implement to be a compliant driver. ADBC has like 20 total.

Robert Hodges [00:30:52]

Oh yeah, I still, because I worked on ODBC drivers, and I remember at one point, there were these, I think they were published by Microsoft, these books on ODBC functions, and I spent the better part of a year in appendix D, which just covered data types. It was 60 pages, small print, narrow margins.

But you know what, I wanted to ask, because this connectivity is not just the joy of fixing bugs in nanosecond timestamps. It’s seeing the cool things that people do with your creation. I’m just curious. So Arrow opens up some very interesting possibilities for integration, for high performance. Is there anything that stands out for you, something really creative that you’ve seen was opened up because of this technology?

Matt Topol [00:31:45]

Well, offhand the first thing to point out is just that we’ve seen, using ADBC with a data system that is already column oriented, you’re seeing 5, 10, 20x speedups.

Robert Hodges [00:32:03]

Yeah.

Matt Topol [00:32:04]

Just because we eliminate those extra copies and transposes.

Robert Hodges [00:32:08]

Yeah.

Matt Topol [00:32:09]

Getting those speedups turns into CPU cost, that turns into memory allocation. And they’re just being lower to the ground because you’re just not allocating these big buffers. To just zero copy, you take directly what you got from the data source and you just pass it on to the client.

Robert Hodges [00:32:28]

Yeah.

Matt Topol [00:32:29]

But for example, dbt recently announced that dbt core, the way they’re doing connectors is just ADBC now.

Robert Hodges [00:32:38]

Mhm. Oh cool.

Matt Topol [00:32:41]

Like dbt Fusion was built on using ADBC for their connectors, and now dbt core itself, dbt core v2, their database connectors are just ADBC, which means that anything that has an ADBC driver can be used with it.

But the coolest thing I see about this whole Arrow ADBC and so on is the fact that if you’re building a new data system, you’re building a new database, if you expose Arrow as your interface, use say Arrow Flight SQL as your server API, you don’t have to build clients at all, because it already exists and it’s already performant, right?

If your system server API matches the standard, this entire ecosystem of tools already exists and saves you a ton of development, right? And we’ve seen, obviously Dremio uses Flight SQL as their interface, and then there’s the ADBC Flight SQL driver, just works.

Robert Hodges [00:33:54]

Mhm.

Matt Topol [00:33:56]

Apache Doris added an Arrow Flight SQL interface to their server side and now everything can just work. InfluxDB uses Flight SQL. LanceDB outputs Arrow. And so the composability of all of this is what I see as very creative.

Robert Hodges [00:34:18]

And that was kind of Wes McKinney’s, I think, the germ of this idea, that you should have this interchange format among all these systems that were core.

Matt Topol [00:34:31]

Yeah, that interchange format that’s also efficient to operate on in itself. I mean, Apache DataFusion is an entire query engine whose internal representation is just Arrow, right?

It actually started as a subproject of the Arrow project, and then got enough community involvement and got big enough that they became their own top level Apache project, right? And so DataFusion and all of it, and the Cambrian explosion of databases that we’re seeing that are just DataFusion under the hood, is all powered by Arrow.

Robert Hodges [00:35:10]

Mhm. Yeah. It’s one of these things that it’s kind of like a snowball, right? Once it reaches a certain size, it then has this huge momentum.

Can I switch gears and ask you a slightly different question? So one of the things that fascinated me about Arrow early on was you could use it over a TCP/IP connection where underneath, the driver was just streaming a bunch of buffers across the wire. But I was always fascinated about the idea that you could use shared memory access. Can you comment on that? How big is that, and is that something that is widely used?

Matt Topol [00:35:50]

It’s not that widely used, that I would say, but it is used in a lot of places. Basically because the raw bytes on disk, Arrow has what’s called an IPC format.

Robert Hodges [00:36:04]

Mhm.

Matt Topol [00:36:07]

An IPC file format. And so because the bytes on disk are identical to the bytes you would need in memory, you can get big benefits from memory mapping the file, right? Or using shared memory.

Robert Hodges [00:36:19]

Yep.

Matt Topol [00:36:20]

As long as you define the shared memory as read only, so that you don’t have all the collisions and all the fun concurrency problems. But it means that you can just use the shared memory, because at its core Arrow is just pointers to buffers, right?

And so for example, Hugging Face is a great example. Hugging Face’s entire on disk caching structure and format for their data sets and data processing is Arrow IPC, and then they memory map it.

Robert Hodges [00:36:57]

Yep. Yeah, that totally makes sense.

Matt Topol [00:37:00]

Operate on larger than memory sizes of data.

Robert Hodges [00:37:03]

Yep.

Matt Topol [00:37:05]

Without blowing your RAM out, because it only has to actually allocate the data when you touch and read those bits and pieces of it. So if you only wanted one column from a 50 gig data set.

Robert Hodges [00:37:13]

Yep.

Matt Topol [00:37:19]

You only actually allocate the one column, right? Because it can memory map it. And the same thing works true for shared memory.

Robert Hodges [00:37:29]

That’s a great example. Yeah, because I think for ClickHouse users, I imagine most of them are across a network somewhere. So this isn’t relevant. But this notion of being able to memory map these files and then do high performance processing on them, even across the network. I mean, there is RDMA.

Matt Topol [00:37:47]

Yes. So a while back I had put together a proposal that’s currently experimental in the Arrow project called dissociated IPC protocol. And the goal was to enable Arrow IPC across things like UCX and RDMA, and allow you to actually do Arrow IPC transfer of data with remote shared memory.

Robert Hodges [00:38:17]

Yeah.

Matt Topol [00:38:18]

And it’s still experimental because, well, no one’s really using it yet and it hasn’t been fully vetted and proved out, because I need people to experiment with it, give feedback to it, and reiterate.

Josh Lee [00:38:31]

Just for anyone who doesn’t know, what is RDMA, and what was the one other one you mentioned?

Matt Topol [00:38:38]

RDMA is remote direct memory access.

Josh Lee [00:38:39]

Memory access. Yeah. Okay. And there was one other one you mentioned, UCX.

Matt Topol [00:38:41]

UCX is a transport protocol for high speed, you know, like you can do shared memory across it. You can do RDMA over it. It’s an entirely different stack from like TCP/IP.

Robert Hodges [00:38:59]

Yeah. This is, we might just have something approaching a use case for what you just described with RDMA. So we just partnered up with a company called Nirvana Labs to host ClickHouse.

Now, one of the things about them, they come out of the crypto web3 world and these folks are into speed in a big way. So they have a cloud, and one of the features of this cloud is that it’s highly rack aware, and you can allocate hosts which are in the same rack with a high-speed top of rack switch.

Matt Topol [00:39:29]

That’s exactly where this stuff flies.

Robert Hodges [00:39:36]

Because, for example, and stop me if I’m going too far with this, but if you are participating in markets where you’re trying to do arbitrage across relatively small periods of time, having that access between the thing that holds your data and the thing that draws conclusions from it, it’s really critical to make that as fast as possible.

So this sounds like something the RDMA, it would be really kind of fun to put a benchmark together on that, to see if we can

Matt Topol [00:40:11]

I mean, this and then of course that’s the kind of thing when you got. Yeah. Go ahead.

Robert Hodges [00:40:18]

Then once we get it to work, we can go and find a problem that needs that solution.

[laughter]

Matt Topol [00:40:22]

But that’s one, the thing is, if you make communications fast

Robert Hodges [00:40:26]

Yeah.

Matt Topol [00:40:28]

If you make stuff go, because you get to eliminate that serialize and deserialize overhead.

Robert Hodges [00:40:37]

Exactly. And I think the obvious place is, there’s got to be people in the financial industry that are doing this kind of thing already, because these are people that already are building specialized networks that bounce signals off the layers of the atmosphere because it’s too slow to go through fiber optic repeaters.

Matt Topol [00:40:56]

So yeah, I mean there’s FINOS, the Fintech Open Source Foundation.

Robert Hodges [00:41:02]

Yeah.

Matt Topol [00:41:04]

And one of their big projects was Perspective, which is the whole visualization library and things for sites. And Perspective uses Arrow.

Robert Hodges [00:41:22]

Yeah, that totally makes sense. And so this is delightful, because I started out as a connectivity person, at CIA among other things. And it’s cool to see this innovation going on in something which, well, God knows ODBC has not changed a lot in a few decades.

Matt Topol [00:41:42]

Yeah. Oh my, ODBC was always the bane of my existence. And if you ever tried to configure your DSNs for ODBC, it’s also awful.

Robert Hodges [00:41:56]

Yeah.

Matt Topol [00:41:57]

So we also, ADBC takes a stab at that. We have our connection profiles. We’re using TOML because it’s better than INI.

[laughter]

Robert Hodges [00:42:09]

You said you’re using TOML because it’s better than INI. Oh, okay. Man, that’s, you’ve got to be careful who you say that to. That’s the kind of stuff that starts fist fights, you know.

Matt Topol [00:42:18]

Oh, I like being spicy.

Josh Lee [00:42:24]

Yeah,

[laughter]

we, oh, Robert, we didn’t even get you, you nerds, man. We didn’t even get my favorite use case for this in-memory sharing, which is observability. And there’s like a whole new OTel collector pipeline built around Arrow. But that’s okay, we’ll bring that back on for another episode because we’re out of time. But just to touch on it so that we can hold people for that future thing.

Matt Topol [00:42:49]

When they were doing a bunch of testing and benchmarking using the OTel Arrow exporters and connectors, they saw 30 to 40% reduction in cost, or rather 30, 40% improvement in speed, which was attributed to not having to do the conversion to and from JSON.

Robert Hodges [00:43:18]

Yep.

Matt Topol [00:43:20]

And smaller payloads.

Robert Hodges [00:43:22]

Yep.

Matt Topol [00:43:24]

And the faster processing. And they’re still building even further on top of that, because the collectors and there’s some other bits of the OTel workflow that aren’t fully there yet for Arrow.

Josh Lee [00:43:34]

Yeah. There’s sort of an end to end pipeline that’s emerging there.

Matt Topol [00:43:38]

Just doing the, but even just doing the network transport.

Robert Hodges [00:43:40]

Yep.

Matt Topol [00:43:41]

Collector, exporter, and then dumping it into something. That in and of itself, doing Arrow and OTel Arrow, improved performance so much.

Robert Hodges [00:43:52]

Yeah. And it simplifies architectures, because with ClickHouse, if you’re processing JSON it’s so compute intensive in some cases you actually have to put it out to a pre-ingest layer.

Matt Topol [00:44:04]

Yep.

Robert Hodges [00:44:05]

Comes through, and that works, but it uses resources. And the other thing is it makes the systems more complex to operate.

Matt Topol [00:44:11]

Exactly.

Robert Hodges [00:44:12]

And then Arrow just kind of flows through and works. So okay, data types are a benefit to humanity.

Matt Topol [00:44:19]

I think we, data types are awful. Arrow just works.

[laughter]

Robert Hodges [00:44:22]

Yeah, you want to have data types.

Matt Topol [00:44:26]

Yeah, but databases have gone off the deep end with it.

Robert Hodges [00:44:31]

Yeah, you want to have data types, but you don’t want to personally work on them. I think you want to have everybody agree on

[laughter]

I want one set, one taxonomy of data. And I think Jim Gray once said, talking about lock managers, he said, if you find out that your job is to write the database lock manager, the first step is to find out if somebody else doesn’t want to do it instead.

[laughter]

So yeah. So, man, this is, Josh is giving us the hook, but this has been the best geekfest I’ve had in months.

Matt Topol [00:45:10]

Oh, I’m glad. I’m glad. I am happy. I love connectivity. Connectivity has always been a big thing I love, because it’s such an untapped thing in a lot of ways, where everybody’s always trying to come up with better things to do. But open source is the best thing here. Not because of the actual development but because of the standardization.

Robert Hodges [00:45:36]

Yeah. Absolutely. And I think what’s interesting about what you guys have done with Arrow is you’ve actually, this is a major technology inflection point, and you guys have actually gotten it to work. And I know it’s not been quick.

Matt Topol [00:45:56]

Oh, I mean, Arrow is 10 years old this year.

Robert Hodges [00:45:59]

Exactly. Yeah.

Matt Topol [00:46:02]

ADBC is three years old now.

Robert Hodges [00:46:05]

Yeah.

Matt Topol [00:46:06]

We’re seeing a lot of pickup with ADBC. Power BI announced earlier this year that they’re replacing all of their internal Power BI drivers, they’re going to be using ADBC. I had to laugh because they published their little post and every single one of the current connectors was a Simba ODBC driver for this thing.

Robert Hodges [00:46:37]

Mhm.

Matt Topol [00:46:38]

And it’s being replaced by ADBC now.

Robert Hodges [00:46:43]

Yeah. So congratulations. That in and of itself is a piece of adoption, on top of the dbt.

Matt Topol [00:46:52]

Yeah. On top of the work that Columnar is doing with Snowflake for the Snowflake ADBC driver, for the Databricks ADBC drivers and all those things that we’re doing. And we’re trying to get as many, we want to encourage more and more vendors to contribute ADBC drivers and work with us. We’re happy to work with vendors to build and open source these drivers and then add them to our CDN.

Robert Hodges [00:47:18]

Yep.

Matt Topol [00:47:19]

So that users can get pre-compiled, signed, secure, easily distributable, easily installable drivers, right?

Robert Hodges [00:47:26]

And if people want to learn more about this, or to use it, they just contact you at Columnar. Is that how they should

Matt Topol [00:47:38]

I mean, if you go to dbc.how, that’ll give you everything you need to know about the dbc CLI for installing and downloading and managing ADBC drivers.

Robert Hodges [00:47:51]

Okay.

Matt Topol [00:47:52]

columnar.tech is our site. We frequently will have blog posts up about advances with ADBC and with dbc, and publishing new releases. We put out a whole bunch of new drivers and new versions of drivers with some bug fixes just last week. So we’re constantly improving and building the drivers.

We have a few private drivers that are not on the public CDN that you can get free trials for, if you go to console.columnar.tech. But the more and more we’re doing this, expect some more announcements coming from Columnar in the next month or two, of an actual product we’ll sell.

[laughter]

Robert Hodges [00:48:41]

Yeah. Oh, open source. Isn’t that against the principles of open source to do that?

Matt Topol [00:48:45]

But I mean, we’re a business.

Robert Hodges [00:48:48]

Yeah, there you go.

Matt Topol [00:48:49]

So the three of us co-founders of Columnar are all Arrow PMC members.

Robert Hodges [00:48:57]

Yeah.

Matt Topol [00:48:59]

And so we as a company are very, very heavily tied to the Arrow ecosystem.

Robert Hodges [00:49:04]

Yep.

Matt Topol [00:49:06]

By virtue of that, and by virtue of what we know and we’re good at. But we’re also the best positioned to know what to do with this, because my colleague David Li, who’s our co-founder, is the guy who created ADBC.

Robert Hodges [00:49:22]

Yep.

Matt Topol [00:49:24]

So we know what we’re doing.

Robert Hodges [00:49:27]

I’m looking at your site. It looks great. I think you guys are going to do really well in this market. There’s a lot of interest in speed and moving, and it’s all about data.

Matt Topol [00:49:38]

So yeah, the big key thing that we’re focused on is enabling multi-data source applications in organizations.

Robert Hodges [00:49:48]

Yeah.

Matt Topol [00:49:49]

Because ADBC gives you that singular interface. And so more and more organizations are relying on two, three, five, more separate database platforms.

Robert Hodges [00:50:03]

Mhm.

Matt Topol [00:50:05]

And rather than having to implement connectors to all of them, they can just use ADBC.

Robert Hodges [00:50:14]

Yep.

Matt Topol [00:50:14]

And get Arrow.

Robert Hodges [00:50:16]

Yeah.

Josh Lee [00:50:17]

You shared an article with that citation for the number of organizations using more than one database. And we’ll make sure that’s in the show notes for folks. But yeah, that was stark to me, how many companies are moving to multi database architecture.

Matt Topol [00:50:28]

Oh yeah. And it only keeps growing. And it’s becoming even more popular because of like Iceberg, and things like that where the actual platform you’re on is less tied to where the data is now.

Robert Hodges [00:50:40]

Yes.

Matt Topol [00:50:43]

And so you can actually use the right engine for your use case.

Robert Hodges [00:50:47]

Yeah.

Matt Topol [00:50:49]

On the same data, which then just encourages needing to be able to go to different data systems for the different use cases.

Robert Hodges [00:50:57]

Yep.

Matt Topol [00:50:58]

And so having something like ADBC where you can just use the same application interface and just point it to that source or that source or that source, without having to have multiple different connectors or native things or all of these things. It enables better observability, maintainability, auditability, all these things which, like I said, keep note for future Columnar announcements regarding stuff in terms of centralizing all this.

But it’s all open source Apache. Our drivers are public CDN drivers that are free to use, so please use them, give us feedback. That’s the biggest thing we need right now, we need more feedback.

Robert Hodges [00:51:48]

Yep.

Matt Topol [00:51:50]

Tell us, file issues on the repos. Tell us what drivers, what systems you want drivers for next.

Robert Hodges [00:51:55]

Yep.

Matt Topol [00:51:57]

Help us bug your vendors, to tell them you want Arrow data.

Robert Hodges [00:52:03]

Yeah.

Matt Topol [00:52:05]

So that you can make the ADBC drivers better. Because there’s so many systems. My favorite example, and I hammer on them all the time, is Trino. Column oriented compute engine, only outputs JSON.

Robert Hodges [00:52:26]

Ouch.

Matt Topol [00:52:28]

Yeah. Somebody, either earlier this year or last year, put together a PR on the Trino repo where they added an option for the spooler to output Arrow directly, and saw 400x performance improvement.

Robert Hodges [00:52:45]

Yeah, I’ll bet.

Matt Topol [00:52:49]

And that PR died on the vine.

Robert Hodges [00:52:52]

Oh, that’s a, huh. That seems short-sighted.

Josh Lee [00:52:57]

Okay. Well, let’s not, on that depressing note.

[laughter]

Robert Hodges [00:53:03]

It’ll get, yeah, basically says bug your vendors.

Matt Topol [00:53:07]

Yep.

Robert Hodges [00:53:08]

I like that. Hey, this has been wonderful to have a chance to meet and hear about what’s going on in the Arrow ecosystem. It’s really great, and congratulations again.

Matt Topol [00:53:18]

Thank you. Thank you. It’s just been a fantastic conversation, and good questions too. That’s what makes all these little things better, is having the right conversations, interactions, joking around.

Robert Hodges [00:53:32]

Just the good reader that makes the good book.

[laughter]

Matt Topol [00:53:37]

Look at you with your wine.

Robert Hodges [00:53:39]

I believe that’s some critic made that motto up.

[laughter]

So yeah. Well, good. It sounds like we’ll be seeing you again. Hopefully we’ll get together at a conference, and maybe see if we couldn’t do a talk together. This is such a great subject and there’s so much to cover.

Matt Topol [00:54:01]

Oh yeah, I love it. I’m happy to come back on, meet up with you guys at a conference. I’m happy for it. Just let me know.

Robert Hodges [00:54:07]

Cool. Okay, cool.

Josh Lee [00:54:09]

Great. Thanks. Thanks, guys. It’s been great.

Matt Topol [00:54:12]

Thank you very much.