ClickHouse Monitoring with Graphite
Apr 20, 2018
Introduction
Monitoring is an important part of operating any software in production. ClickHouse is not an exception here. ClickHouse provides an introspection to various system metrics using tables in ‘system’ schema but they only show current or cumulative numbers. It is sufficient in order to understand what’s going on with the system right now, but can not show time trends or what has been happening a minute ago. The proper monitoring solution needs to store metrics somewhere as well as provide visualization capabilities. ClickHouse does not have such a tool packaged, but there are several 3rd-party monitoring solutions that can be used. Graphite is one of the popular options, and it can be natively integrated with ClickHouse. In this article, we will describe how to configure Graphite for monitoring ClickHouse metrics.
Install Graphite
Graphite installation is described in details in its manual
Feel free to choose any type of installation.
Setup ClickHouse – Graphite integration
ClickHouse configuration for Graphite is specified in ClickHouse config file. Edit main config file /etc/clickhouse-server/config.xml
or add a separate /etc/clickhouse-server/conf.d/graphite.xml
, and append following section:
<graphite>
<host>192.168.74.150</host>
<port>2003</port>
<timeout>0.1</timeout>
<interval>60</interval>
<root_path>one_min_cr_plain</root_path>
<metrics>true</metrics>
<events>true</events>
<asynchronous_metrics>true</asynchronous_metrics>
</graphite>
<graphite>
<host>192.168.74.150</host>
<port>2003</port>
<timeout>0.1</timeout>
<interval>1</interval>
<root_path>one_sec_cr_plain</root_path>
<metrics>true</metrics>
<events>true</events>
<asynchronous_metrics>false</asynchronous_metrics>
</graphite>
Settings description:
host
– host where Graphite is running.port
– Carbon plain text receiver port (2003 is default in Graphite). In general, Graphite has multiple ports open, with the following default values:80
nginx2003
carbon receiver – plaintext this one should receive data from ClickHouse2004
carbon receiver – pickle2023
carbon aggregator – plaintext2024
carbon aggregator – pickle8080
Graphite internal gunicorn port (without Nginx proxying).8125
statsd8126
statsd admin
interval
– interval for sending data from ClickHouse, in seconds.timeout
– timeout for sending data, in seconds.root_path
– prefix used by Graphite.metrics
– should data fromsystem_tables-system.metrics
table be sent.events
– should data fromsystem_tables-system.events
table be sent.asynchronous_metrics
– should data fromsystem_tables-system.asynchronous_metrics
table be sent.
Multiple <graphite>
sections can be used for sending different data at different intervals.
NOTE, ClickHouse restart is required in order to activate graphite configuration.
Monitoring
Navigate to Graphite web monitoring tool in a browser as http://host/dashboard
in order to see Graphite metrics.
If everything is properly configured, and ClickHouse sends its data to Graphite, entries with prefixes should be displayed, as specified in ClickHouse’s configuration, e.g.:
<root_path>one_sec_cr_plain</root_path>
For all available metrics Graphite will display monitoring graphs.