Persistent Storage with Kubernetes Cluster

Kubernetes is extremely useful tool when one talks about deploying complex micro-services based solutions. It helps to simplify the process from providing out of box resilience to ability to scale services horizontally at a blink of an eye. However, at some point you might question yourself how to use persist data that are used by the containers deployed to Kubernetes?

With standalone docker there are couple of strategies that could be used. Docker can map the docker managed volumes or the host directory to the running container, hence if container is destroyed the data will be kept in the above objects and hence could be mapped to the new container should it be re-created.

In theory the host directory can also be used as a persistent storage on Kubernetes eco system, but then something should be able to sync data between the directories situated on each node. This is theory and still should be confirmed.

Another approach is to use network storage such as NFS or iSCSI to share the data. This is a step by step guide to set up and use iSCSI LUNs with Kubernetes.

First, ensure that iscsi-initiator-utils package is installed on each node. In order to access the iSCSI drives the following pre-requisites must be met:

1. The nodes are connected to the iSCSI portal preferably using the separate network interface.
2. iSCSI target portal is configured to allow multiple connections and some form of security is in place, i.e. it authorises only connection from the range of IPs or CHAP authentication is configured.
3. One or more LUNs are created on the target iSCSI portal.

Configure the iSCSI initiator on each node of the cluster. Then run the following command to discover the accessible LUNs (on each node):

  sudo iscsiadm -m discovery -t sendtargets -p

This will create iSCSI targets locally. Next log into the iSCSI portal by running the following command:

  sudo iscsiadm -m node --login

Or to login to the specific target:

  sudo iscsiadm -m node -T <target IP>

Then create configuration for the persistent volume (for example in the file storage/mysql.yaml)

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: <name>
  annotations:
    volume.beta.kubernetes.io/storage-class: "slow"
spec:
  capacity:
    storage: <capacity>
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  iscsi:
    targetPortal: <access IP address>
    iqn: <target name>
    lun: 0
    fsType: ext4
    readOnly: false

Create it in the Kubernetes:

  kubectl create -f storage/mysql.yaml

That is kind of it. The persistent storage is available to the Kubernetes containers. In order to use it you can create a persistent storage claim for example using the following configuration file (storage/mysql.yaml)

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: <name>
  annotations:
    volume.beta.kubernetes.io/storage-class: "slow"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: <capacity>

Then create it:

  kubectl create -f storage/mysql.yaml

Bare in mind that persistent volume capacity should match the persistent volume claim capacity. (At least for now). In order to use it (for example with MySQL container) create the following file (deployment/mysql.yaml)

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mysql
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: mysql
        environment: production
    spec:
      containers:
        - name: mysql
          image: mysql:5.6
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: password
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-data
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-data
          persistentVolumeClaim:
            claimName: mysql

Create a deployment by running:

  kubectl create -f deployment/mysql.yaml

Your persistent volume is attached to the pod. If you kill the pod a replica set associated with the deployment will create a new pod which will be attached to the same persistent volume.

The downside of this strategy is that it only allows one pod to be connected with the read/write permission (when connected to the iSCSI persistent volume). This means that replica set can only have one pod, i.e. it is not horizontally scalable, however, Kubernetes will still make the deployment resilient as in case of failure pod will be replaced automatically.

Alternatively, you can attache the LUN directly to the pod. Here is the example of configuration (pods/mysql.yaml)

---
apiVersion: v1
kind: Pod
metadata:
  name: mysql
spec:
  containers:
  - name: mysql
    image: mysql:5.6
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: mysql-data
  volumes:
  - name: mysql-data
    iscsi:
      targetPortal: <access IP address>
      iqn: <target name>
      lun: 0
      fsType: ext4
      readOnly: false

In this case LUN is attached directly to the pod and not using claims mechanism.