Amazon Web Services Posted April 10 Share Posted April 10 Overview Amazon EKS Anywhere (EKS Anywhere) is an opinionated and automated deployment of the Amazon EKS Distro that enables users to create and operate Kubernetes clusters on user-managed infrastructure. EKS Anywhere does not include a Container Storage Interface (CSI) driver for persistence. In this post, we setup OpenEBS to provide persistence using the disks available in the Kubernetes cluster. There are platform specific CSI drivers available when deploying EKS Anywhere on vSphere, Apache Cloud Stack, and Nutanix. These drivers are supported by the teams behind those platforms. In a bare metal deployment, we must consider the Open Source and partner supported CSI drivers, and there are many. In this post we deploy OpenEBS as one such solution. Note that OpenEBS is not related to Amazon Elastic Block Store (Amazon EBS). OpenEBS is an Open Source project in the Cloud Native Computing Foundation portfolio. It is designed to use the storage available on Kubernetes nodes available as replicated container attached storage to applications within the cluster. We are deploying OpenEBS with Jiva to our cluster to provide storage for a sample application. OpenEBS deploys “HostPath” PersistentVolumes that act as persistent replicas for our Jiva volume, which are attached directly to the container through a PersistentVolumeClaim. OpenEBS supports several “Data Engines”. We are using Jiva here as it is the most flexible. You may want to consider Mayastor if you are focused on NVMe disks. Note that as of this writing Bottlerocket does not support iSCSI. Many Kubernetes CSI drivers that work on Bare Metal have iSCSI as a prerequisite. For this reason, we are using Ubuntu 20.04 as our cluster operating system. Prerequisites For this example we have a previously deployed an EKS Anywhere (0.18.7) Kubernetes (1.28) cluster on bare metal using Ubuntu (20.04). The example cluster is built as three stacked (meaning ETCD and Kubernetes API services running on the same node) nodes. Walkthrough In this post we walk you through the following steps. Install Open-iSCSI on our nodes Install OpenEBS 3.4 on our cluster through Helm Set our default StorageClass Test Dive deeper Cleaning up Install Open-iSCSI We need to set up Open-iSCSI on each worker node in our data plane. Since our data plane is already established, we SSH into each worker node and add that. For a production environment we would customize an image-builder pipeline to add the open-iSCSI package. EKS Anywhere provides documentation on customizing the image-builder images. SSH into each worker node and install open-iSCSI: sudo apt-get update sudo apt-get install -y open-iscsi sudo systemctl enable --now iscsid sudo systemctl status iscsid Install OpenEBS That is the only SSH related task. From here on out we are working with Helm charts and Kubernetes manifests. The first of these is the OpenEBS Helm chart. We use Helm chart values to enable the Jiva container attached storage data engine. helm repo add openebs https://openebs.github.io/charts helm repo update helm upgrade openebs openebs/openebs \ --install \ --namespace openebs \ --create-namespace \ --set jiva.enabled=true To verify everything is installed and running, we run kubectl get pods -n openebs, which should show a similar output as follows, and the random characters at the ends are different. $ kubectl get pods -n openebs NAME READY STATUS RESTARTS AGE openebs-jiva-csi-controller-0 5/5 Running 0 37m openebs-jiva-csi-node-8gh25 3/3 Running 0 37m openebs-jiva-csi-node-gpfwk 3/3 Running 0 37m openebs-jiva-csi-node-srxm9 3/3 Running 0 37m openebs-jiva-operator-69bd68fccc-6kjss 1/1 Running 0 37m openebs-localpv-provisioner-56d6489bbc-2jd62 1/1 Running 0 37m openebs-ndm-9ftqw 1/1 Running 0 37m openebs-ndm-j2jpt 1/1 Running 0 37m openebs-ndm-kn4zb 1/1 Running 0 37m openebs-ndm-operator-5d7944c94d-wtlkf 1/1 Running 0 37m pvc-9676b5f9-9368-49ba-ba85-e50c87fa6c65-jiva-ctrl-65798b7jhdgh 2/2 Running 0 28m pvc-9676b5f9-9368-49ba-ba85-e50c87fa6c65-jiva-rep-0 1/1 Running 0 28m pvc-9676b5f9-9368-49ba-ba85-e50c87fa6c65-jiva-rep-1 1/1 Running 0 28m pvc-9676b5f9-9368-49ba-ba85-e50c87fa6c65-jiva-rep-2 1/1 Running 0 28m Our next verification step is to check all the StorageClasses through kubectl get sc and we should see the following. NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE openebs-device openebs.io/local Delete WaitForFirstConsumer false 12m openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 12m openebs-jiva-csi-default jiva.csi.openebs.io Delete Immediate true 12m OpenEBS has deployed three StorageClasses in total. The openebs-device and openebs-hostpath StorageClasses are used to create volumes on the nodes in our data plane. In our example cluster, we have three worker nodes in our data plane, and the default number of replicas Jiva creates is three. When we create a PersistentVolumeClaim to request storage, we use the openebs-jiva-csi-default StorageClass. When we use the openebs-jiva-csi-default StorageClass, Jiva creates three openebs-hostpath based PersistentVolumes as replicas, and a volume attached to our container that acts like a proxy reading and writing to the replica volumes that were created. Now that we have our openebs-jiva-csi-default StorageClass, we can use it to create dynamic persistent volumes by referencing it as our StorageClass. Let’s look at a sample application. Set the default StorageClass We can also mark our openebs-jiva-csi-default Storage Class as the default StorageClass for Kubernetes to use if a PersistentVolumeClaim is created without specifying the storageClassName to use. The following command patches in is-default-class label. kubectl patch storageclass openebs-jiva-csi-default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' If we run kubectl get sc openebs-jiva-csi-default, then we see that openebs-jiva-csi-default is now our default StorageClass. NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE openebs-jiva-csi-default (default) jiva.csi.openebs.io Delete Immediate true 5m31s Testing To test that our persistence layer is working, we deploy a PersistentVolumeClaim to request storage from Jiva. We test and verify that everything is working by deploying a BusyBox container that doesn’t really do anything. We use that container to create a file and test that it still exists after we delete the container. First, create busybox.yaml with the following contents. kind: PersistentVolumeClaim apiVersion: v1 metadata: name: example-jiva-csi-pvc spec: storageClassName: openebs-jiva-csi-default accessModes: - ReadWriteOnce resources: requests: storage: 5Mi --- kind: Deployment apiVersion: apps/v1 metadata: name: busybox labels: app: busybox spec: replicas: 1 strategy: type: RollingUpdate selector: matchLabels: app: busybox template: metadata: labels: app: busybox spec: containers: - resources: limits: cpu: 0.5 name: busybox image: busybox command: ['sh', '-c', 'sleep 3600'] imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /var/output name: demo-vol1 volumes: - name: demo-vol1 persistentVolumeClaim: claimName: example-jiva-csi-pvc This file defines our PersistentVolumeClaim, and it asks for the storage to be provisioned with the openebs-jiva-csi-default storage class. Additionally, we define a Kubernetes Deployment of busybox with a volume called demo-vol1 that uses our PersistentVolumeClaim as backing. Then, that volume is mounted into our container at /var/output. Now let’s deploy our PersistentVolumeClaim and our busybox Deployment. $ kubectl apply -f busybox.yaml persistentvolumeclaim/example-jiva-csi-pvc created deployment.apps/busybox created If we execute kubectl get pods, then we see our new busybox container come up shortly. $ kubectl get pods NAME READY STATUS RESTARTS AGE busybox-86b8c7df5f-59r4m 1/1 Running 0 41m Once we are in the “1/1 Running” state we can test creating a file in our persistence. We connect to that running pod and create a file. $ kubectl exec -i -t busybox-[random] -- /bin/sh Now that we are inside the container we can see that our /var/output directory is empty. $ ls -l /var/output total 16 drwx------ 2 root root 16384 Apr 28 17:15 lost+found The lost+found directory is default and we can ignore it. The fact that it exists means things are working. Let’s create a file quickly to see that things are truly working: $ echo world > /var/output/hello $ ls -l /var/output total 20 -rw-r--r-- 1 root root 6 Apr 28 18:06 hello drwx------ 2 root root 16384 Apr 28 17:15 lost+found Now that our hello file exists, let’s delete this container and test that we get re-attached to our PersistentVolume as expected. First we need to exit that exec session. $ exit Now we’re back out or normal prompt, we have exited the Busybox container. Let’s delete the pod we used previously, and wait for Kubernetes to start a new one. Since our Deployment spec says that there should always be one replica, Kubernetes makes sure of that by starting a new one. $ kubectl delete pod busybox-[random] Let’s wait for Kubernetes to start a new one. Once it is at the 1/1 Running state we can resume. $ kubectl get pods --watch NAME READY STATUS RESTARTS AGE busybox-86b8c7df5f-4r679 0/1 ContainerCreating 0 54s busybox-86b8c7df5f-4r679 1/1 Running 0 54s Now we connect to our new pod and make sure we find the hello file that we expect. $ kubectl exec -i -t busybox-[random] -- cat /var/output/hello world Success! We’ve proven that we can create the file in the persistent layer outside of our pod and have it restored in a new pod. You can stop here if you’d like, or follow along into a deep dive of how this all works. Dive deeper With our Busybox Deployment and PersistentVolumeClaim still deployed we can see some of what OpenEBS and Jiva are doing for us. Let’s look at our PersistentVolumeClaim first. $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE example-jiva-csi-pvc Bound pvc-9523fdf1-4d87-4341-a623-404b9cc4142b 4Gi RWO openebs-jiva-csi-default 75m The example-jiva-csi-pvc PersistentVolumeClaim we created is Bound to the pvc-9523fdf1-4d87-4341-a623-404b9cc4142b Volume that was created for us. Let’s look at the volumes on our cluster now. $ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-275ed8d5-fca9-4b04-bbfc-ba4de0c0fb8b 4Gi RWO Delete Bound openebs/openebs-pvc-9523fdf1-4d87-4341-a623-404b9cc4142b-jiva-rep-2 openebs-hostpath 78m pvc-60bca1e6-4548-4bd2-a704-265a4842b52a 4Gi RWO Delete Bound openebs/openebs-pvc-9523fdf1-4d87-4341-a623-404b9cc4142b-jiva-rep-0 openebs-hostpath 78m pvc-9523fdf1-4d87-4341-a623-404b9cc4142b 4Gi RWO Delete Bound default/example-jiva-csi-pvc openebs-jiva-csi-default 78m pvc-ff76dcc5-74ca-4c33-b7ff-c35b63f55769 4Gi RWO Delete Bound openebs/openebs-pvc-9523fdf1-4d87-4341-a623-404b9cc4142b-jiva-rep-1 openebs-hostpath 78m What’s this? And why are there four PersistentVolumes created when we only requested one? This is what OpenEBS is doing for us behind the scenes. If we look at the list of CLAIMs in the output, then we can see our default/example-jiva-csi-pvc claim we expected to find. However, we also see three other claims that all look the same, except the claims end with jiva-rep-0, jiva-rep-1, and jiva-rep-2. This is because OpenEBS and Jiva create three replicas of any volume we request by default. This can be changed by passing —set jiva.defaultPolicy.replicas=n when installing the helm chart. The PersistentVolume mounted to our container is like a proxy. When we write to our volume, the writes are actually replicated to these three volume replicas. If we describe our replica-0 volume, then we see that it’s actually a local volume on the disk. $ kubectl describe pv pvc-60bca1e6-4548-4bd2-a704-265a4842b52a Name: pvc-60bca1e6-4548-4bd2-a704-265a4842b52a Labels: openebs.io/cas-type=local-hostpath Annotations: pv.kubernetes.io/provisioned-by: openebs.io/local Finalizers: [kubernetes.io/pv-protection] StorageClass: openebs-hostpath Status: Bound Claim: openebs/openebs-pvc-9523fdf1-4d87-4341-a623-404b9cc4142b-jiva-rep-0 Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 4Gi Node Affinity: Required Terms: Term 0: kubernetes.io/hostname in [eksa-cp01] Message: Source: Type: LocalVolume (a persistent volume backed by local storage on a node) Path: /var/openebs/local/pvc-60bca1e6-4548-4bd2-a704-265a4842b52a Looking into the details of the PersistentVolume we can see that it is a LocalVolume type, and that it’s actual location on disk for this node is /var/openebs/local/pvc-60bca1e6-4548-4bd2-a704-265a4842b52a. However, we don’t have to worry about this, as OpenEBS manages all of this for us. The /var/openebs path is default, but this is configurable through the helm chart values. Cleaning up Now all that’s left to do is clean up. We can tell Kubernetes to delete our PersistentVolumeClaim and Deployment that we applied earlier. $ kubectl delete -f busybox.yaml persistentvolumeclaim "example-jiva-csi-pvc" deleted deployment.apps "busybox" deleted After a moment those resources are deleted and we can see that all four PersistentVolumes that OpenEBS was managing for us have been removed. $ kubectl get pv No resources found Conclusion In this post we have seen how OpenEBS can leverage the existing storage on nodes in an EKS Anywhere cluster to provide persistence support for the workloads we deploy. To learn more about EKS Anywhere, you can visit the the EKS Anywhere page and test it out for yourself.View the full article Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.