Backup kubernetes cluster including pv using velero backup

In this post, I will backup kubernetes cluster along with pv to minio. Minio is object storage, you can use it as alternative solution of AWS S3. When you deploy your application to kubernetes cluster, it is better to install those apps in separate namespace other then default.

Download velero binary

wget https://github.com/vmware-tanzu/velero/releases/download/v1.5.3/velero-v1.5.3-linux-amd64.tar.gz
mv velero-v1.5.3-linux-amd64.tar.gz  /tmp/
tar -xzvf velero-v1.5.3-linux-amd64.tar.gz 
mv velero-v1.5.3-linux-amd64/velero  /usr/local/bin/

Create the credential file

vim credentials-velero

 [default]
 aws_access_key_id = ************
 aws_secret_access_key = **************************

Apply the yaml file

kubectl apply -f https://raw.githubusercontent.com/vmware-tanzu/velero/main/examples/minio/00-minio-deployment.yaml

Install velero

velero install \
    --provider aws \
    --bucket velero \
    --plugins velero/velero-plugin-for-aws:v1.1.0 \
    --use-restic \
    --secret-file ./credentials-velero \
    --use-volume-snapshots=true \
    --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=https://minio-ip:9000,insecureSkipTLSVerify="true" \
    --cacert ./public.crt \
    --snapshot-location-config region=minio 

Now we will use restic for backing up volume (pv). Edit the deploy velero

kubectl edit deploy velero -n velero

Add the following line

    spec:
      containers:
      - args:
        - server
        - --features=
        - --default-volumes-to-restic # Add this one

Restart the velero pods, simply scale down and then scale up

kubectl scale deploy velero -n velero --replicas=0
kubectl scale deploy velero -n velero --replicas=1

Schedule backup

velero schedule create k3s-ha-full-backup --schedule "0 1 * * *"

This creates a Backup object with the name  <SCHEDULE NAME>-<TIMESTAMP>. The default backup retention period, expressed as TTL (time to live), is 30 days (720 hours);

How to restore?

A disaster happens and you need to recreate your resources.

Update your backup storage location to read-only mode (this prevents backup objects from being created or deleted in the backup storage location during the restore process):

kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \
    --namespace velero \
    --type merge \
    --patch '{"spec":{"accessMode":"ReadOnly"}}'

Create a restore with your most recent Velero Backup:

velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

When ready, revert your backup storage location to read-write mode:

kubectl patch backupstoragelocation <STORAGE LOCATION NAME> \
   --namespace velero \
   --type merge \
   --patch '{"spec":{"accessMode":"ReadWrite"}}'

References
https://velero.io/docs/v1.5/disaster-case/
https://devops.cisel.ch/backup-kubernetes-pv-with-velero-the-new-approach

Leave a Reply

Your email address will not be published. Required fields are marked *