ReplacingMergeTree and Crazy ClickHouse Stuff: The December SF Bay Area ClickHouse Meetup
The SF Bay Area ClickHouse Meetup is the oldest US ClickHouse meetup. We held our first meeting on May 4, 2017 in Santa Clara. The meetup now has 1,186 members. 2022 was a busy year with 5 virtual meetups, a legacy from moving online in early 2020 due to COVID.
Here is a quick summary about our latest meetup on December 14th, 2022, which featured talks by Alexandr Dubovikov, Alexey Milovidov, and your humble correspondent. As usual, the talks were streamed to YouTube, which provides video for the event.
The first two talks cover a topic near and dear to my heart: the ClickHouse ReplacingMergeTree Engine, known to connoisseurs as “RMT.” It’s a versatile engine that handles mutations but also allows high-performance analytic queries. Here are the talks with video references plus slide deck links.
The first talk up was Adventures with the ClickHouse ReplacingMergeTree Engine. Robert Hodges of Altinity introduced ReplacingMergeTree and its uses. Altinity leans on RMT to mirror MySQL data to ClickHouse in their Altinity Sink Connector project. The talk showed how ReplacingMergeTree can handle not just inserts but also simulate updates and deletes. In the presentation, you’ll find notes on performance (such as getting ORDER BY right) as well as the use of the FINAL keyword to resolve updates. It concludes with two important PRs (implicit FINAL and automatic row deletion) that will make RMT even better in the months to come. Slides here.
Next up was ReplacingMergeTree in Telecom Apps. Alexandr Dubovikov of QXIP presented an important industrial use case of using ReplacingMergeTree to track the state of phone calls in real-time from call detail records (CDRs). He introduced a number of tricks to ensure RMT high performance including the use of settings like do_not_merge_across_partitions_select_final, min_age_to_force_merge_seconds, and min_age_to_force_merge_on_partition_only. He also demoed performance, showing how RMT reaches MergeTree performance levels in recent ClickHouse versions. Slides here.
After much discussion of RMT and questions from the crowd, we switched away to a completely different topic.
That was Alexey Milovidov of ClickHouse Inc presenting Doing Crazy Stuff with ClickHouse. Alexey presented new, unusual, and completely crazy things you can do with ClickHouse. Did you know you can just pull down ClickHouse now using curl? How about obfuscating schema and data? What about special tricks with clickhouse-local like using it to replace grep? You can even use implement a completely serverless ClickHouse like the service at https://clickhouse.glitch.me/play. There are many more in the talk. (We also learned that ClickHouse can even crash Zoom!) Slides here.
As usual there were many questions with lots of insights from participants. Our meetups tend to be database nerd fests and this was no exception. Thanks to all speakers and participants for attending.
Our New Year’s resolution is to schedule the talks well in advance and let everyone know about them well in advance. Accordingly the next meetups are already scheduled for February 22nd and April 19th 2023. We’re looking for talks. Send email to info@altinity.com or DM Robert Hodges on our ClickHouse Slack workspace or telegram.
Happy holidays, ClickHouse fans. See you in 2023!!