Managing EBS gp3 Volumes in EKS

ByAlexander Zaitsev
Subscribe

Gp3 is the current generation of general purpose Elastic Block Storage (EBS) in AWS. It provides flexibility to configure performance characteristics, and optimal balance between performance and cost. We rely heavily on gp3 in Altinity.Cloud. In this article we will explain how to use gp3 volumes in AWS Elastic Kubernetes Service (EKS). We will also introduce a new open source software from Altinity that allows to change parameters of EBS volumes from inside EKS clusters.

gp3 Storage Class

For some strange reason, EBS storage in general is not supported in EKS by default. Users have to follow a few steps described in AWS documentation in order to deploy the EBS CSI driver that enables gp3 and other volume types. Once the EBS CSI driver is installed, we may create a storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  fsType: ext4
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

The usage is extremely simple: one may then reference the storage class in a PersistentVolumeClaim (PVC) and that’s it.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-gp3-volume
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: gp3

Note the ‘allowVolumeExpansion’ flag in the storage class definition. The volume class is expandable, so it is possible to increase the storage capacity dynamically by changing storage requests in PVC.

Unfortunately, volume expansion is the only modification that is allowed for an existing volume using the PVC interface. In the next sections, we will explain how to apply additional customizations to the new and existing EBS volumes.

gp3 Storage Class Parameterized

One may notice parameters in the storage class definition, like ‘fsType’ and ‘type’. What other parameters are supported and what do they allow users to control? Luckily, the EBS CSI driver is an open source project! That helps us to find the reference for supported parameters:

https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/parameters.md

Storage class parameters allow us to define different storage classes for different needs. For example, we may define a storage class with enabled encryption and 500MB/s throughput:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-encrypted-500
provisioner: ebs.csi.aws.com
parameters:
  encrypted: 'true'
  fsType: ext4
  throughput: '500'
  type: gp3
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Customized storage classes work fine for new volumes, allowing users to provision volumes with different performance characteristics. But what if we want to change throughput of an existing volume? AWS allows a wide spectrum of changes to EBS volumes, but not from inside Kubernetes. This is due to a limitation of Kubernetes CSI specification, which restricts the modifyVolume action to size only. Users are stuck with the initial configuration, or have to go to AWS Console to perform changes. As this is dependent on upstream community approval AWS are unable to implement this functionality in EKS. This is of course not convenient. 

EBS Params Controller

In order to modify existing EBS volumes from EKS we have developed the EBS Params Controller. The controller allows us to manage volume parameters in a “Kubernetes” way: using annotations applied to PVC. There are two sets of annotations: ‘spec’ annotations, and ‘status’ annotations.

‘Spec’ annotations are set by a user in order to specify a change to the volume parameter. For example, adding annotation ‘spec.epc.altinity.com/throughput: 500’ sets the volume throughput to 250MB/s. Here is an example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-gp3-volume
  annotations:
    spec.epc.altinity.com/throughput: 250
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: gp3

Note, that annotations are not just set on new PVCs. More importantly, annotations can also be set to an existing PVC, which will trigger volume modification!

‘Status’ annotations are set by a controller in order to communicate the current parameters of the volume back to the user. They can also report volume modification status and last modification time. AWS does not allow you to modify volume more often than once in 6 hours, so knowing the previous modification time is important. 

annotations:
  spec.epc.altinity.com/throughput: '250'
  status.epc.altinity.com/iops: '3000'
  status.epc.altinity.com/mod-end-time: '2023-05-22T09:16:39.000Z'
  status.epc.altinity.com/mod-start-time: '2023-05-22T09:11:30.000Z'
  status.epc.altinity.com/mod-state: completed
  status.epc.altinity.com/throughput: '250'
  status.epc.altinity.com/type: gp3

The annotation approach is very powerful. Using it we can modify multiple volumes at once. For example, the command below changes all volumes of ClickHouse cluster my-clickhouse:

kubectl annotate --overwrite pvc -n my-namespace -l clickhouse.altinity.com/cluster=my-clickhouse spec.epc.altinity.com/throughput=500

In addition to throughput, it is possible to change provisioned IOPS and even the volume type! Moreover, even if volume is changed from an outside, e.g. using AWS console, the volume state will be correctly read and reported back by the controller.

Under the hood, the controller subscribes to PVC change events, checks for ‘spec’ annotations, and uses AWS API in order to apply changes. It also updates the ‘state’ every 5 minutes.

Conclusion 

Using gp3 volumes in EKS clusters is easy though it requires some extra configuration steps. The EBS CSI driver provides a good interface to create differently configured gp3 volumes. It can not, however, make modifications to existing volumes except for extending the volume size. There is no interface for that yet. It may appear in the future, since users keep asking about it.

This is where a custom controller may help. It allows users to modify EBS volume parameters using PVC annotations and also read the current state back. The controller is very simple and open source. Feel free to use and contribute: 

https://github.com/Altinity/ebs-params-controller

The EBS Params Controller helps us manage storage conveniently in our own Altinity.Cloud, so we are confident it is useful. We’re delighted to be able to share it with the wider Kubernetes community. Enjoy!

Share

Related: