Announcing ARM64 Altinity Stable Builds for ClickHouse

Altinity was the first to provide ARM builds for ClickHouse back in 2021, but we paused support for them due to a lack of test coverage. Today, we are excited to announce full support for AARCH64 (ARM64) architecture in Altinity Stable Builds for ClickHouse, starting with the latest release, 23.3.13.7.altinitystable. 

You can try ARM binaries using Docker or install them directly using our APT repository containing DEB packages or YUM repository containing RPM packages by following the installation instructions on the Altinity Stable Builds page.

The ARM Docker images are also immediately available for use by our customers on our Altinity.Cloud and now allow all our customers to take advantage of the benefits of Altinity Stable Builds for ClickHouse, irrespective of server architecture.

For more information, refer to the Altinity Stable Builds documentation. If you are not sure if Altinity Stable Builds are right for you, check out the Why Every ClickHouse User Should Appreciate Altinity Stable Builds article, where we talk about why many users choose them over other distributions.

Multi-arch Docker Images

One of the simplest ways to use the new ARM64 builds is by using Docker. Starting with 23.3.13.7.altinitystable, Docker images for our Altinity Stable Builds for ClickHouse are now multi-arch. You can find them on Docker Hub.  

Click the `altinitystable` link to find currently available images. 

If you are on an AMD64 machine or using an ARM64 server, you can run the same command to pull the Docker image: 

docker pull altinity/clickhouse-server:23.3.13.7.altinitystable

When you enter the running container, the following queries can be used to confirm that everything is working as expected:

SELECT version()

┌─version()────────────────┐
│ 23.3.13.7.altinitystable │
└──────────────────────────┘

SELECT
    name,
    value
FROM system.build_options
WHERE name = 'SYSTEM_PROCESSOR'

┌─name─────────────┬─value───┐
│ SYSTEM_PROCESSOR │ aarch64 │
└──────────────────┴─────────┘

Notable Differences in Behavior Between ARM64 and AMD64 Builds

Given that ARM64 and AMD64 are completely different architectures, some aspects, such as floating point arithmetic, operate differently. You can expect that results for some queries might differ between the AMD64 and ARM64 architectures. Here are a few common examples. 

Floating Point Results

Differences in floating point results can be easily seen using aggregate functions over floating-point numbers.

For example,

SELECT stddevPopStable(number * 3.23456711e-11)
FROM numbers(1000000)

On AMD64 returns: 0.000009337390958258145

But on ARM64, the same query returns: 0.000009337390958258144

As can be seen, the results differ by the rightmost decimal digit. This also means that the hashes of the results might not be the same between ARM64 and AMD64.

We clearly saw these differences in behavior when we were porting our test suites, such as our window functions suite, where we had to create separate snapshots for each architecture.

For example, the query

SELECT simpleLinearRegression(salary, empno) OVER (ORDER BY salary, empno ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS func FROM empsalary

had the following differences between results on each architecture:

AMD64: 0.0017991004497751124,-2.473763118440779
ARM64: 0.0017991004497751124,-2.47376311844078

Also, differences were detected between the results of the tests that were using cityHash64 over floating-point results.

For example, the query

SELECT anyLast(id), anyLast(time), cityHash64(groupArray(w))
FROM (SELECT id, f_timestamp AS time, exponentialTimeDecayedAvg(1)(id, f_timestamp)
OVER (ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING) AS w
FROM datetimes2 ORDER BY id)

produced different hash values due to the exponentialTimeDecayedAvg function returning a floating-point result.

AMD64: 9999 2020-01-01 02:46:39 6998628102243875284
ARM64: 9999 2020-01-01 02:46:39 9677279760188586566

Aggregate Function States

Aggregate function states can also be different for the two architectures. Here is an example of differences in states detected by our tests for the corrState function used with Nullable(Int64) and UInt64 data types:

SELECT hex(corrState(nullable_int64_,uint64)) FROM table_eef28883_6380_11ee_b6ee_25d441995fb1

AMD64: 010000000000002240DC980956B98ECB43826D1DBA536A13441E7A36E1E10CD647C805563E2B5DF1475AA1782F51A20E48
ARM64: 010000000000002240DC980956B98ECB43826D1DBA536A13441E7A36E1E10CD647C705563E2B5DF1475AA1782F51A20E48

Due to this and other possible incompatibilities in binary representation of states and intermediate results, running heterogeneous clusters that mix both AMD64 and ARM64 ClickHouse instances is not recommended. Though tests do not cover this, you can share data via textual or intermediate formats like Parquet.

No HDFS and GRPC

Two ClickHouse capabilities are disabled for ARM completely, since corresponding libraries are not available for this architecture. Those are the HDFS Engine, the hdfs() table function, and the GRPC connection protocol.

This can be checked by querying the system.build_options table:

SELECT *
FROM system.build_options
WHERE name in ( 'USE_GRPC', 'USE_HDFS')

┌─name─────┬─value─┐
│ USE_HDFS │       │
│ USE_GRPC │       │
└──────────┴───────┘

VectorScan Instead of HyperScan

Originally, ClickHouse used the Intel HyperScan library for some advanced search scenarios. HyperScan was not available for the ARM architecture, so ClickHouse ARM builds were failing on certain search functions. Starting from 22.9, ClickHouse has migrated to the VectorScan library, which is a portable fork of HyperScan, so it is not an issue anymore.

Added Test Coverage for AARCH64 (ARM64)

The road to adding support for ARM64 architecture was not easy, and one of the main pain points was that our QA team had to port all the Altinity integration tests for ClickHouse to ARM64 architecture to make sure that our ARM64 and AMD64 test coverage was the same when running our Altinity test suites.

If you are interested, you can find our open-source test suites for ClickHouse on GitHub at https://github.com/Altinity/clickhouse-regression.

Test coverage is not exactly the same between the two architectures. One major difference is that the community integration tests have not yet been ported to ARM64 and therefore are not executed as part of the standard CI/CD pipeline.

You can check the exact test coverage for our Altinity Stable Builds by looking at the publicly available build reports that accompany each of our releases. For example, you can check our v23.3.13.7.altinitystable build report to find a link to the GitHub Actions pipeline that runs all our tests.

Overall, due to differences in test coverage between AMD64 and ARM64 platforms and the fact that the ARM64 platform is not yet as popular and well used as AMD64, using ARM binaries should be done with caution.

Conclusion and Where to Find Out More

ARM64 support is an important step in our journey to deliver 100% open-source ClickHouse builds for enterprise users who value stability and long-tail maintenance for business-critical analytics. We will continue to improve the build process and extend our testing over the coming releases. We look forward to your feedback on this latest improvement or any aspect of our work on Altinity Stable Builds.

For information about Altinity Stable Builds and the philosophy behind them, check out our blog article Why Every ClickHouse User Should Appreciate Altinity Stable Builds. For details on builds including how to install, check out the Altinity Stable Builds for ClickHouse documentation. You can also run Altinity Stable Builds in Altinity.Cloud by selecting a stable build version when you start or upgrade ClickHouse clusters.

Please log build issues in our ClickHouse fork on GitHub or join the conversion on our public Altinity Slack Workspace. You can contact us directly for information about support. In the meantime, take our new 23.3.13.7.altinitystable out for a spin!