How SenseOn Delivers Proactive Threat Detection with Altinity’s Support for ClickHouse®

SenseOn, a cybersecurity SaaS company, enables organizations to protect their IT environments from malicious activities.

The core of their product is built around ClickHouse®, a powerful open-source OLAP database. Its flexibility, scalability, ease of use, SQL syntax, ingestion rate, and query performance (as a function of data volume) have been crucial to SenseOn’s growth for seven years.

In 2023, SenseOn recognized the opportunity to expand support for larger customers and extend retention windows, providing users with more time to retain essential data for effective threat detection.

Adoption of ClickHouse

SenseOn originally built its SaaS platform using both ElasticSearch and ClickHouse for different roles within their real-time analysis pipeline. 

However, ClickHouse’s ingest performance was comparatively better and its resource requirements were many times lower for CPU, memory, and disk. So, they shifted entirely to ClickHouse to process large amounts of telemetry data for real-time threat detection.

Additionally, SenseOn gives customers direct access to query, visualize, and analyze their own telemetry data. With ClickHouse, SenseOn can optimize this self-service feature to operate on real-time data across any dimension. 

SenseOn can provide this level of customization and scalability using a sampling key, a powerful ClickHouse feature. Sampling keys enable querying massive datasets by analyzing a representative sample, a crucial capability for building large-scale, high-performance dashboards like SenseOn’s.

Challenge: Optimizing the Sampling Keys for Scalability

While the use of sampling keys has allowed SenseOn to develop new product capabilities, their usefulness depends heavily on correctly defining them. For example, poorly chosen sampling keys can reduce compression efficiency and increase storage costs.

To optimize performance, SenseOn refined the schema for their tables, which significantly improved sampling. However, this change had an unintended consequence: it increased disk usage and reduced compression efficiency, adding 50 extra bytes per row (as opposed to 7 bytes per row).

This change resulted in approximately 30% extra storage usage, pushing some ClickHouse instances to the brink of running out of disk space and potentially creating higher operational costs.

Solution: Altinity Support for ClickHouse

The sampling key update created a trade-off: better sampling for decreased table compression. To mitigate this issue, SenseOn turned to Altinity’s support team.

Altinity quickly helped SenseOn diagnose how changing a column that modified the table order impacted overall compression.  

Based on SenseOn’s need to optimize both sampling and storage, the Altinity team recommended some adjustments:

• Change the compression algorithm—test different codecs, including ZSTD(1), ZSTD (2), and ZSTD(3), to strike a balance of compression efficiency and performance
• Use the system.parts_column table to make sure data is ordered correctly

Altinity’s step-by-step instructions in implementing these suggestions enabled SenseOn to achieve substantial disk savings without any noticeable performance penalties. 

As SenseOn’s CTO James Mistry notes, “Everything from the responses we get, the timeliness of responses to the automation that has been built, [Altinity Support] just works really, really well. And it fits really nicely into developers’ workflows.” 

Conclusion

Over the past year and a half, this partnership has continued to help SenseOn accelerate its development, optimize operations, save engineers’ time, and improve their customer experience.

Speaking about the ongoing benefits of working with Altinity, Mistry adds, “There’s tremendous value in being able to run things by Altinity.”

Altinity provided timely diagnosis, debugging, and advice on everything from user management, database migration, and upgrades to I/O prioritization, backup configuration, and INSERT performance.

Mistry notes, “We have made better decisions earlier about things like designing schemas and configuring ClickHouse. Because we’ve talked to Altinity about it before it’s gone into production, we made better decisions. We’ve definitely been told stuff that we didn’t know or that we haven’t necessarily foreseen, which means we’ve made things better, to begin with.”