ClickHouse Confidential: Using Kubernetes Secrets with the Altinity Operator

ByRobert Hodges
Subscribe

Secrets are a vital Kubernetes resource type that solves the problem of passing credentials securely to cloud native applications. The Altinity Kubernetes Operator for ClickHouse has abundant support for using Secrets to protect ClickHouse installations. In this article we’ll explain how Secrets work, then show how to use them with the operator. 

The recent Altinity operator version 0.23.0 release makes integration with Secrets more consistent and easier to use than in previous versions. We’ll  show the new features but also show older behavior as well. All features described in this article are available in open source, and code samples are available in GitHub. 

Finally, Altinity.Cloud has abundant support for Secrets as well. This is especially important when using Bring Your Own Cloud (BYOC) or Bring Your Own Kubernetes (BYOK) deployment models. We have a blog article coming out shortly that covers those features separately. 

Kubernetes Secrets

Secrets come in a variety of flavors. The default Secret type is an Opaque Secret, which is a dictionary of one or more name-value pairs. Let’s illustrate using a Secret resource with SHA256-hashed values to use as passwords in ClickHouse. Here is an example.

apiVersion: v1
kind: Secret
metadata:
  name: db-passwords
type: Opaque
stringData:
  default_password_sha256: fe9f…1fda8
  root_password_sha256: 6dd5…9a9f'

To load the Secret into Kubernetes, put the resource into file db-passwords.yaml and apply it with kubectl as shown below. You can read the current resource state back using jq to suppress the resource metadata, which improves readability. 

kubectl apply -f db-passwords.yaml
kubectl get secret/db-passwords -o json | jq 'del(.metadata)'
{
  "apiVersion": "v1",
  "data": {
    "default_password_sha256": "MjlkNjliZjRlM2JmZmNlMDUxNGYxOTMxNmY0ZTk0ZjE4MTdiNWY0N2Q1YWMxNmU4ZjFhNGFjYzZmZTU2MGEwOA==",
    "root_password_sha256": "NGJiNjZmN2I0N2M5M2RiMjgwZjQ3MDllOGNhODFjYjE2NWU4NGZiZGExZjY0MDVjNDU1NWE5ZmRjMzM5MWQ2ZQ=="
  },
  "kind": "Secret",
  "type": "Opaque"
}

Note that the resource you read back appears to show different values. In fact they are the same. The Secret values are listed using the data: property, which means they are base64-encoded. If you run each value through the base64 -d command you will see the original values. This is significant: Kubernetes does not do anything special to hide the resource values to authorized users. 

Opaque Secrets are sufficient for the examples in this article. For more information check out the Kubernetes documentation. It offers a nice explanation of Secrets including the different types of Secrets and how to use them properly. 

It’s easy to automate the above steps using the sample Bash script generate-passwd-secret.sh. It generates random password values and applies them to Kubernetes where they can be used for the examples that follow. 

Using Secrets to protect ClickHouse user passwords

Database credentials are one of the prime use cases for Secrets, so the Altinity operator has built-in syntax to pull passwords from Secrets into a ClickHouseInstallation resource. Let’s start with the syntax to set passwords on ClickHouse users without using a Secret. In this example ClickHouseInstallation resource, both the default and root users use the SHA256-hashed password of “topsecret”.

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "secure"
spec:
  configuration:
    users:
      default/password_sha256: "5333…7a721"
      root/networks/ip: "::/0"
      root/password_sha256: "5333…7a721"
      root/profile: "default"
      root/access_management: 1
    clusters:
      #Etc.

In the above example the SHA256 hash values are present in the CHI resource definition, hence visible in GitHub (if you store them there) or anywhere else it is stored. 

Let’s now use the password syntax available in the Altinity Operator to pull the passwords from the Secret we created. If you are using ClickHouse operator 0.23.0 you can configure the CHI resource definition using the valueFrom property to fetch the value from a Kubernetes Secret.

# User passwords from Secrets for operator 0.23.0 *AND ABOVE*.
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "secure"
spec:
  taskID: "1"
  configuration:
    users:
      default/password_sha256_hex:
        valueFrom:
          secretKeyRef:
            name: db-passwords
            key: default_password_sha256
      root/networks/ip: "::/0"
      root/password_sha256_hex:
        valueFrom:
          secretKeyRef:
            name: db-passwords
            key: root_password_sha256
      root/profile: "default"
      root/access_management: 1

The valueFrom: syntax is widely used in Kubernetes to read values from Secrets. With 0.23.0 the operator is now consistent with this syntax, which we’ll use in other cases as well. 

Operator versions 0.22.2 and below support a different syntax, which is now deprecated. It uses a specially property value syntax that the Altinity operator recognizes as a reference to a Secret value. If you have an older operator version you’ll need to use this instead. 

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "secure"
spec:
  taskID: "1"
  configuration:
    users:
      default/k8s_secret_password_sha256_hex: db-passwords/default_password_sha256
      root/networks/ip: "::/0"
      root/k8s_secret_password_sha256_hex: db-passwords/root_password_sha256
      root/profile: "default"
      root/access_management: 1
    clusters:
      #Etc.

Voila! In both cases SHA256 passwords disappear from the CHI resource. You can repeat this for yourself in the GitHub examples directory by running the following commands. 

./generate-apply-passwd-secret.sh
kubectl apply -f chi-security-01-secrets.yaml

Picking up new password values from Secrets

It’s good security hygiene to rotate passwords from time to time. It’s easy enough to regenerate our Secret resource with new values using the generate-passwd-secret.sh script. However, the ClickHouse Operator won’t notice the changes automatically because it does not automatically track changes to Secret resources. 

Instead, you need to use a trick to force reconciliation, in which the operator checks for changes in the environment. Notice that the above examples included a taskID property, which looks like this: 

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "secure"
spec:
  taskID: "1"

Use kubectl to change taskID to a different value like “2” as shown below. This will force reconciliation, so that your new passwords will appear. 

kubectl patch chi secure --type=merge -p '{"spec":{"taskID":"2"}}'

You can watch the operator logs using a command like the following. You’ll see it reconcile and apply changes.  

kubectl logs -f pod/<operator name>

Once the changes are applied you will be able to login with the new values from your Secret. In both cases, this occurs without a ClickHouse server restart. 

Using Secrets to set environment variables for cloud access

Most public cloud users are familiar with the pattern of passing credentials into applications using environment variables. Here’s a typical example using aws-cli. 

export AWS_ACCESS_KEY_ID=<id>
export AWS_SECRET_ACCESS_KEY=<key>
aws s3 ls
<listing of S3 buckets>

Let’s see how to use Kubernetes Secrets to enable S3 access for a ClickHouse server running in Kubernetes. First, we need to construct a suitable Secret with the AWS credentials. Use the sample script generate-s3-secret.sh to create a Secret like the following: 

apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
type: Opaque
stringData:
  AWS_SECRET_ACCESS_KEY: =m…Mk
  AWS_ACCESS_KEY_ID: AK…OE
  AWS_S3_ENDPOINT: "https://…"

IMPORTANT NOTE: The sample script generates the Secret from environmental variables in the build environment. The AWS credentials themselves are never stored in a file outside of Kubernetes, where they can accidentally be checked into GitHub. 

Next, we need to map the Secret values to an S3 endpoint defined using <s3> tags. These are a standard way to provide credentials for object storage. (See the ClickHouse docs for details.) They normally look like the following when you configure them by hand. 

<clickhouse>
  <s3>
    <my_bucket>
      <endpoint>https://my-bucket.us-west-1.s3.amazonaws.com/a/b/</endpoint>
      <access_key_id>AK…OE</access_key_id>
      <secret_access_key>AK…OE<secret_access_key>
    </my_bucket>
  </s3>
</clickhouse>

However, we won’t configure them by hand. Instead, we’ll use new valueFrom syntax available in the 0.23.0 Altinity Operator, which is illustrated in example chi-security-02-s3.yaml. It looks like the following. 

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "secure"
spec:
    settings:
      s3/my_bucket/endpoint:
        valueFrom:
          secretKeyRef:
            name: s3-credentials
            key: AWS_S3_ENDPOINT
      s3/my_bucket/secret_access_key:
        valueFrom:
          secretKeyRef:
            name: s3-credentials
            key: AWS_SECRET_ACCESS_KEY
      s3/my_bucket/access_key_id:
        valueFrom:
          secretKeyRef:
            name: s3-credentials
            key: AWS_ACCESS_KEY_ID

From this the operator generates the S3 endpoint configuration exactly as shown above. 

For older versions of the operator you can generate the endpoint tags explicitly by populating Secret values from environmental variables. This is shown in example chi-security-02-s3-pre-0.23.0.yaml. It uses the handy “from_env” attribute, which ClickHouse provides to populate XML tags from named environmental variables. 

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "secure"
spec:
  configuration:
    files:
      config.d/s3.xml: |
        <clickhouse>
          <s3>
            <playground>
              <endpoint from_env="MY_AWS_S3_ENDPOINT"/>
              <secret_access_key from_env="MY_AWS_SECRET_ACCESS_KEY"/>
              <access_key_id from_env="MY_AWS_ACCESS_KEY_ID"/>
            </playground>
          </s3>
        </clickhouse>
    # Etc.

Next, we need to get the environmental variables. The Altinity Operator uses a standard Kubernetes trick to map Secret values to environmental variables that are available to ClickHouse which it starts. 

  templates:
    podTemplates:
      - name: replica
        spec:
          containers:
          - name: clickhouse
            image: altinity/clickhouse-server:23.3.8.22.altinitystable
            env:
            - name: MY_AWS_S3_ENDPOINT
              valueFrom:
                secretKeyRef:
                  name: s3-credentials
                  key: AWS_S3_ENDPOINT
            - name: MY_AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: s3-credentials
                  key: AWS_ACCESS_KEY_ID
            - name: MY_AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: s3-credentials
                  key: AWS_SECRET_ACCESS_KEY

You may wonder why the examples renamed the environment variables using the prefix MY_. It turns out that if you just populate AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ACCESS directly these values are available within ClickHouse. Users can use them to select from any S3 URL, not just those listed in <s3> tags. 

Important security notes: Kubernetes secrets move values safely but they are still visible inside the pod. Otherwise ClickHouse itself can’t see the values. 

  1. Environmental variables defined inside the pod as we showed in the second example are visible to processes in the pod. Anyone who can run ‘kubectl exec’ on the pod can see them using the ‘env’ command. 
  2. Older versions of ClickHouse show environmental values on disk in /var/lib/clickhouse/preprocessed_configs. ClickHouse pre-processes all configuration files and renders the final values down to a small set of files that it actually reads. Passwords, AWS credentials, and other values are visible here, even if you use the from_env attribute. See https://github.com/ClickHouse/ClickHouse/pull/53818 for information on how to suppress this behavior. 

You should therefore always restrict access to running pods if they use valuable credentials. 

Picking up new Secret values for environmental variables

You may need to change credentials contained in environmental variables from time to time. Like Secret values used for ClickHouse passwords, it’s not enough to change the Secret. Unfortunately, if you use environmental variables, it’s also not enough to change the taskID property either. In this case you have to restart the ClickHouse pod. 

While it’s possible to delete the ClickHouse pod using ‘kubectl delete’, this will disrupt running applications. A better way is to do a rolling restart on ClickHouse using the operator restart: “RollingUpdate” property. Executing the following patch command will restart the operator gracefully, which is to say, one replica at a time.

kubectl patch chi secure --type=merge -p '{"spec":{"restart":"RollingUpdate"}}'

One more thing: the restart property is ephemeral. The operator removes it from the ClickHouseInstallation after the restart completes. 

Using Secrets to transmit X509 certificates and private keys

Features like TLS encryption require public and private keys to be mounted as files. Kubernetes Secrets can do this as well. The next example shows how to transmit X509 certificates and keys. Once again, we start by constructing a Secret resource like the following example. 

apiVersion: v1
kind: Secret
metadata:
  name: server-crt
type: Opaque
stringData:
  ca.crt: |-
    -----BEGIN CERTIFICATE-----
    MIIDDTCCAfWgAwIBAgIUYw8yEunsC3CX/RxSTJd1/bVoLvswDQYJKoZIhvcNAQEL
    …
    l3RNRDnrNMB2Pypl/v9vKzU=
    -----END CERTIFICATE-----
  server.crt: |-
    -----BEGIN CERTIFICATE-----
    MIICsDCCAZgCFD9KsduOqdgCSB6ftWw+w4e5j+shMA0GCSqGSIb3DQEBCwUAMBYx
    …
    mt8IlJLU4vdpgY/6vyyFVoNldUk=
    -----END CERTIFICATE-----
  server.key: |-
    -----BEGIN PRIVATE KEY-----
    MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQC3opGOkN6gBDMN
    …
    iC+MAzRkF994/d528Br3Yw==
    -----END PRIVATE KEY-----

Next we need to map our certificates into the ClickHouse server pod and ensure ClickHouse sees them. The Altinity operator version 0.23.0 offers convenient valueFrom syntax for mapping secret values automatically to files. The commands are in sample file chi-security-05-certs.yaml. Here are the relevant parts. 

spec:
  configuration:
    files:
      ca.crt:
        valueFrom:
          secretKeyRef:
            name: server-crt
            key: ca.crt
      server.crt:
        valueFrom:
          secretKeyRef:
            name: server-crt
            key: server.crt
      server.key:
        valueFrom:
          secretKeyRef:
            name: server-crt
            key: server.key
      openssl.xml: |
          <clickhouse>
            <openSSL>
              <server>
                <loadDefaultCAFile>false</loadDefaultCAFile>
                <caConfig>/etc/clickhouse-server/secrets.d/ca.crt/server-crt/ca.crt</caConfig>
                <certificateFile>/etc/clickhouse-server/secrets.d/server.crt/server-crt/server.crt</certificateFile>
                <privateKeyFile>/etc/clickhouse-server/secrets.d/server.key/server-crt/server.key</privateKeyFile>
                <verificationMode>relaxed</verificationMode>
                <cacheSessions>true</cacheSessions>
                <disableProtocols>sslv2,sslv3</disableProtocols>
                <preferServerCiphers>true</preferServerCiphers>
              </server>
              <client>
                <loadDefaultCAFile>false</loadDefaultCAFile>
                <caConfig>/etc/clickhouse-server/secrets.d/ca.crt/server-crt/ca.crt</caConfig>
                <cacheSessions>true</cacheSessions>
                <disableProtocols>sslv2,sslv3</disableProtocols>
                <preferServerCiphers>true</preferServerCiphers>
                <verificationMode>relaxed</verificationMode>
                <invalidCertificateHandler>
                    <name>AcceptCertificateHandler</name>
                </invalidCertificateHandler>
              </client>
            </openSSL>
          </clickhouse>

Volume mounts are created individually for every mounted secret file using the following pattern:

/etc/clickhouse-server/secrets.d/<file_name>/<secret_name>/<key_name>

Since every secret is automatically mapped as a separate volume, not a sub-path, changes to Secret values are picked up automatically. That makes it possible to rotate certificates without any restarts. You just regenerate the Secret. It’s a big improvement over the older behavior before version 0.23.0. 

Speaking of that older behavior, operator versions prior to 0.23.0 need to mount files to volumes explicitly in the pod template. This is a standard feature of pod specifications in Kubernetes. The commands to get it done are in sample file chi-security-05-certs-pre-0.23.0.yaml.

  templates:
    podTemplates:
      - name: replica
        spec:
          containers:
          - name: clickhouse
            image: altinity/clickhouse-server:23.3.8.22.altinitystable
            volumeMounts:
            - name: server-crt-volume
              mountPath: "/opt/certs/ca.crt"
              subPath: ca.crt
            - name: server-crt-volume
              mountPath: "/opt/certs/server.crt"
              subPath: server.crt
            - name: server-crt-volume
              mountPath: "/opt/certs/server.key"
              subPath: server.key
          volumes:
            - name: server-crt-volume
              secret:
                secretName: server-crt

Now that the files are available, we can pass in the usual ClickHouse TLS configuration using the <openSSL> tag. This just changes the paths, so it’s pretty much identical to the previous example. 

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "secure"
spec:
  configuration:
    # Etc.
    files:
      openssl.xml: |
          <clickhouse>
            <openSSL>
              <server>
                <loadDefaultCAFile>false</loadDefaultCAFile>
                <caConfig>/opt/certs/ca.crt</caConfig>
                <certificateFile>/opt/certs/server.crt</certificateFile>
                <privateKeyFile>/opt/certs/server.key</privateKeyFile>
                # Etc.

In the older confirmation changes to certificate files 0.23.0 require a restart of ClickHouse. See the procedure above in the section on changes for environmental variables.  

Additional exercises for the reader

This article provided lots of examples of using Kubernetes Secrets with the Altinity Operator. There’s always more to do on security, so of course we left a few things out. 

One interesting problem is how to check Secrets into GitHub in a secure way. If you need to do this, check out projects like Bitnami Sealed Secrets for Kubernetes or Mozilla SOPS. Both provide ways to encrypt values in Secrets before pushing them to GitHub, and decrypt them again once they arrive in Kubernetes. 

Conclusion

Secrets are a powerful Kubernetes mechanism to share credentials and other sensitive data with applications like ClickHouse clusters. This article showed three ways to combine Secrets with the Altinity operator to set user passwords, insert cloud credentials into environment variables, and mount files used by TLS encryption. We also covered issues like how to ensure Secret changes are picked up. 

For more information, check out the Altinity Operator Hardening Guide to get up-to-date documentation with additional details. And watch out for a future article on using Secrets in Altinity.Cloud. 

Altinity provides enterprise support for operating ClickHouse on Kubernetes. We can even run ClickHouse for you in Altinity.Cloud. If you have further questions, feel free to contact us at the Altinity website or launch a free trial. Meanwhile, have fun with ClickHouse on Kubernetes!

Share

Related: