Open Source Software at Altinity
Go and open the door.
Maybe outside there’s
a tree, or a wood,
a garden,
or a magic city.
Miroslav Holub
Open source has been fundamental to Altinity since our founding in 2017. We recognized ClickHouse was the first ever analytical open source database that could compete with the best proprietary offerings. We could see how the open source model accelerated ClickHouse development and adoption. We could also see how open source licenses give users the confidence to make major application investments without fear of vendor lock-in. We built Altinity to follow these principles.
So we like open source. Even so, we were surprised to see how many projects Altinity contributes to or maintains outright. This article tells our open source story – how we started, our current work, and where we are going in the future.
How it Began
Historically, our first open source project was ClickHouse RPM builder. At the beginning of ClickHouse’s great march to domination in analytical databases, it was lacking some basic features. RPM packages were one of the missing items. We therefore started maintaining an RPM builder and publishing RPM packages in 2017-2020. We stopped this project when the ClickHouse team added proper RPM support to the main project in ClickHouse version 20.8.
Another interesting dinosaur is ClickTail, which we started to work on in 2018. That was the first attempt to have a universal solution for log streaming into ClickHouse. We wrote a blog article about it, and the project collected quite a few stars.
Similarly, we started ClickHouse-MySQL Data Reader in 2017 to enable streaming of MySQL replication logs into ClickHouse, way before other solutions emerged. We talked about this project at conferences that time. It collected 325 GitHub stars, and was used in several production installations. Unfortunately, the Python implementation did not scale for high workloads, and the project usage declined over time. We are working on a better solution now. We will talk about that below.
ClickHouse and Ecosystem Projects
As you could see in the previous section, from the very beginning we worked on open source tools that help users to adopt ClickHouse. In 2018 and 2019 we began contributions to ClickHouse itself. Later on we also took ownership of several ClickHouse tools and connectors developed by others.
ClickHouse server – Since 2018 our team has contributed more than 400 merged pull requests, which have included everything from trivial bug fixes to significant features, like S3 functions, tiered storage, codecs and many more.
Clickhouse-cpp library – Our team is a major contributor and maintainer of this project. Altinity contributors have made over 190 commits to the project, comprising the majority of commits since 2020.
Clickhouse-odbc driver – Altinity likewise is a major contributor and maintainer of this project. Over half the commits on the project come from Altinity engineers. We significantly rewrote it to improve performance three years ago, and it’s now pretty stable.
Altinity backup for ClickHouse (aka clickhouse-backup) – This started as a pet project of devops engineer AlexAkulov. With Altinity additions it evolved into a production-ready backup tool for self-managed ClickHouse installations. We constantly keep improving it, and we accept a lot of community contributions as well. We also use it ourselves in Altinity.Cloud.
Altinity Grafana DataSource for ClickHouse – This plugin enables integration of Grafana with ClickHouse. It was originally developed by ClickHouse pioneers from Vertamedia, who were also authors of chproxy and other tools. The datasource is deployed by thousands of users. Since 2019 our team has maintained this project. We keep adding new features and make sure it is up to date with the rapid development of Grafana itself.
Altinity Projects for ClickHouse
We are not just contributing to existing open source projects, of course. We have continued to develop more open source tools to make ClickHouse work better.
Altinity Kubernetes Operator for ClickHouse (aka clickhouse-operator) – We started development in late 2018. This is our most successful open source project so far – it just hit 1000 stars! The operator makes it easy to manage ClickHouse in Kubernetes, especially ClickHouse clusters. There are thousands of production installations using the operator; it also powers several cloud services including our own Altinity.Cloud. It’s now one of the most popular Kubernetes operators on GitHub.
The Operator is accompanied by the Altinity Dashboard that provides basic UI for operator management. We hope to spend more time on this in the near future.
Altinity Tableau Connector for ClickHouse – we developed this connector in order to make Tableau easier to work with ClickHouse via ODBC and JDBC drivers. It passes the Tableau conformance tests and is listed in Tableau Exchange. It works with both Tableau Desktop and Server. We were delighted to learn from Tableau that it is now one of the more popular connectors in the Exchange.
Altinity Homebrew Tap for ClickHouse – Mac users, we love you and give you the native ClickHouse experience! Feel free to submit formulas for new releases, if you need those.
TestFlows – This open source project is not directly related to ClickHouse but we use it extensively in order to test ClickHouse and Altinity software. TestFlows is a very flexible test framework that allows us to align our tests with functional requirements. For ClickHouse itself we have developed over 6500 test scenarios that we execute for all our ClickHouse builds. TestFlows is being developed by the Altinity QA team. Please read our blog article if you want to learn more.
New Altinity Projects
We are always looking for new ways to help users build applications on ClickHouse. In the previous section we mentioned ClickHouse-MySQL Data Reader. We were surprised that it was quite popular despite many limitations. Users need to stream data from their existing databases to ClickHouse for analytics, and they are looking for solutions. We’re therefore focused on enabling high-performance replication from OLTP databases into ClickHouse.
The latest addition to Altinity open source software is Altinity Sink Connector for ClickHouse. It is a high-performance Kafka connector for ClickHouse with exactly-once semantics. Our initial use case is real-time streaming from MySQL to ClickHouse. We hope to see it reach production readiness over the summer. PostgreSQL and MongoDB will follow soon after. Read our blog article for more detail. We’ll have much more to say on this in the months to come.
Altinity Knowledge Base
Last year our support team launched the Knowledge Base https://kb.altinity.com in order to provide tips, tricks and advice on ClickHouse usage to the community. The content of the knowledge base is stored in an open source GitHub repository, and we welcome community contributions. The open source model allows users to share content easily and freely.
Cloud Services
Altinity.Cloud is our flagship product. It is built from many components, some of them being proprietary, but many more are open source. We extensively use Altinity projects like clickhouse-operator, clickhouse-backup and Grafana datasource. We also adopt, patch and contribute back to the community improvements on other tools, like ansible/receptor. We plan to release in open source some of our core components as well, once we feel they are mature enough to be shared.
Conclusion
We believe in open source. It is a great model for collaboration, and, arguably, one of the most important movements in software development in the last 30 years. People from different cities, countries or continents work together to make software better. ClickHouse as we know it today would not exist without open source. Altinity would not be as successful without open source. Most importantly, thousands of creative applications that enable people to understand the world better would not exist.
We are excited and proud to be a part of the open source movement, and we are committed to remain so. Please visit all our projects, use them, and contribute to them. Contact us if you would like to learn how to adapt them for your projects.