Kubernetes Backup on Raspberry Pi

storage server in data center rack

Once you start using your Raspberry Pi cluster for workloads with persistence, you’re probably going to want to implement a decent Kubernetes backup strategy.

I have been using my Raspberry Pi Cluster for a number of workloads with persistence requirements, from WordPress sites with MySQL databases, to Minecraft servers.

This post will detail how to get a basic install of Velero up and running on an ARM based Raspberry Pi Kubernetes cluster that uses NFS volume mounts for container persistence.

In other words, a solution that will allow you to backup both your Kubernetes resource states, and the storage that those resources are mounting and reading/writing from and to.

First though, I’ll revisit how I have been backing up my Pi cluster up until recently.

A Simple and Naive Backup Approach

One way of dealing with backups is a fairly brute force, copy data off and upload to another location method. I’ve been using this for a while now, but wanted a more Kubernetes-native method as an extra backup.

I have had a cronjob that does a mysql dump of all databases and then gzips and uploads this to BackBlaze B2 object storage.

Another cronjob handles compression and upload of a variety of different NFS volumes that pods mount. This cronjob runs in a BSD jail on my FreeNAS storage server, where the same Kubernetes NFS storage is mounted to.

You can see the details of how I do this in my Cheap S3 Cloud Backup with BackBlaze B2 post.

Kubernetes Backup of State and Volumes with Velero

Velero is nice in the way that it is pluggable. You can use different plugins and adapters to talk to different types of storage services.

It works well enough to backup your pod storage if you’re running Kubernetes on a platform that where that storage runs too. For example, if you’re running Kubernetes on AWS and using EBS persistent volumes. Another example would be VMware with vSAN storage.

But in this post we’re dealing with Kubernetes on Raspberry Pi using NFS shared storage. If you haven’t yet setup shared storage, here is a guide to setting up NFS storage on Raspberry Pi Kubernetes.

We’ll assume you’re also wanting to backup your state and pod volume storage to AWS S3. You can quite easily modify some of the commands and use S3 API compatible storage instead though. E.g. minio.

Install Velero

On a management machine (where you’re setup with kubectl and your cluster context), download Velero and make it executable. I’m using a Raspberry Pi here, so I’ve downloaded the ARM version.

curl -L -O https://github.com/vmware-tanzu/velero/releases/download/v1.5.1/velero-v1.5.1-linux-arm.tar.gz
tar -xvf velero-v1.5.1-linux-arm.tar.gz
sudo mv velero-v1.5.1-linux-arm/velero /usr/local/bin/velero
sudo chmod +x /usr/local/bin/velero

Create a dedicated IAM user for velero to use. You’ll also setup your parameters for S3 bucket and region, and add permissions to your IAM user for the target S3 bucket. Remember to change to use the AWS region of your preference.

Now you’re ready to install Velero into your cluster. Apply the magic incantation:

velero install --provider aws --plugins velero/velero-plugin-for-aws-arm:main --bucket $BUCKET --backup-location-config region=$REGION --secret-file ./credentials-velero --use-restic --use-volume-snapshots=false

There are a few things going on here that are different to the standard / example install commands in the documentation…

  • The plugins parameter specifies the ARM version of the Velero AWS plugin. I found that the install command and the usual velero-plugin-for-aws selection tries to pull down the wrong docker image (for x86 architecture). Here we specify we want the ARM version from the main branch.
  • Use Restic Integration. This enables the Restic open source integration for persistent volume backup. This is the magic that allows us to backup our NFS mounted volumes. Also handy if you happen to be using other file storage types like EFS, AzureFile, or local mounted.
  • Disable volume snapshots. We’re on a Pi cluster with NFS storage, so we don’t want to use volume snapshots at all for backup.

After the Velero install completes, you should get some output to indicate success.

Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

You should also see your pods in the velero namespace up and running for Velero and Restic.

kubectl -n velero get pods
NAME                     READY   STATUS    RESTARTS   AGE
restic-589rm             1/1     Running   0          7m29s
restic-g6swc             1/1     Running   0          7m29s
velero-555695d95-dbzmd   1/1     Running   0          7m29s

If you see pod initialization errors, then double check you didn’t specify the normal velero plugin for AWS that would have caused the incorrect docker image architecture to be used.

With that complete, you’re ready to run your first backup.

Initiating a Manual Backup

Request a backup with the velero backup command. In the example below I’ll target my demo namespace. I’ll get volume backups for everything in this namespace that uses supported persistent volumes with the --default-volumes-to-restic flag.

If you don’t specify that flag, all backups will be opted-out by default for volume backups. That is, you opt-in to restic volume backups with this flag.

velero backup create demo-backup --include-namespaces=demo --default-volumes-to-restic

You can request backup state and details using the velero backup describe command.

velero backup describe demo-backup --details

It’s worth running a backup of a test namespace with some test workloads. Then simulate failure by deleting everything, followed by a velero restore.

This should prove the backup process works 100% and the DR strategy to be sound.

Don’t forget to do ocassional test DR scenarios to exercise your deployed backup solution!

Conclusion

It’s a good idea to backup your Kubernetes cluster once you start running services with persistence. Even if it is just for personal use.

A quick and dirty approach is to script out the export and backup of Kubernetes resources and mounted container storage. In the long run however, you’ll save time and get a more fully featured solution by using tools that are specifically designed for this use case.

Using Placeholder Templates With Xargs In The Pipeline

pipes running along a wall

Using placeholder templates with xargs gives you a lot more power than simply using xargs to append args onto the end of a command.

I previously blogged about how to use xargs to append arguments to another command in the pipeline. This post goes into a bit more detail and shows you a more powerful way of using xargs.

Basic Xargs Example

With a typical, simple xargs command, you might append one argument onto the end of an existing command in the pipeline like this:

echo "this is a basic xargs example" | xargs echo "you said:"

The above command results in the output:

you said: this is a basic xargs example

Placeholder Templates With Xargs Example

Now, here is a simple example of using a placeholder, or template in your command, and passing your argument into that.

echo "this is a basic xargs example" | xargs -I {} echo "you said: {}"

The output is exactly the same as with the basic example. However, you can now use the curly braces placeholder to move the argument placement to anywhere in your command.

You’re no longer constrained to it being on the end of the echo (or whichever command you’re using). You can also do multiple placements of the argument in your command.

For example:

echo "FOO" | xargs -I {} echo "you said: {}. Here is another usage of your sample argument: {}. And here is yet another: {}"

A Slightly More Practical Example

Enough of the simple echo examples though. How about using this for a more practical, real world example?

In the following example, we want to list a bunch of AWS S3 buckets, and then do a summary output of their total size in GiB. We cut out the bucket name using cut from the initial listing that is returned with aws s3 ls.

aws s3 ls | cut -d' ' -f3 | xargs -n1 aws s3 ls --summarize --human-readable --recursive s3://

Using xargs to append the bucket name from the pipeline looks like it would work, as we only need it right at the end of the aws s3 ls command. There is an issue though, xargs would add a space, and we want the bucket name appended to s3:// without a space.

Using The Template or Placeholder

This is where using a placeholder or template with xargs can come in handy.

aws s3 ls | cut -d' ' -f3 | xargs -n1 -I {} aws s3 ls --summarize --human-readable --recursive s3://{}

It’s also worth noting that you can change your template or placeholder token with the -I parameter. It doesn’t have to be {} as in the examples above.

In summary, your usage of xargs can be levelled up by using the -I parameter to leverage placeholder or template tokens.

Select, match and pipe output into another command with jq, grep, and xargs

select-match-and-pipe-output-with-jq-grep-xargs

This is a quick reference post if you’re looking to pipe output into another command on Linux using xargs.

The pipeline is immensley powerful and you can leverage it to act on different stages of your full command to do specific selecting, matching, and manipulation.

Say you are running an executable that outputs a bunch of JSON and you want to select certain a certain subset of this data, pattern match it, and then send that matched data into another command.

This is the perfect use case for a mixture of the jq, grep and xargs commands in the pipeline.

Practical example with xargs

Here is a practical example where you might want to list all your AWS CodePipeline pipelines, match only on certain ones, and then execute (Release Changes) on each of them.

aws codepipeline list-pipelines | jq -r '.pipelines[].name' | grep project-xyz | xargs -n1 aws codepipeline start-pipeline-execution --name

The piped command above does the following:

  • Lists all AWS CodePipelines with the command aws codepipeline list-pipelines
  • Uses jq to ‘raw’ select the name from each pipeline object in the pipelines[] array that the above command outputs
  • Sends each pipeline name into grep to match only those containing the string “project-xyz”
  • Pipes the resulting pipeline names using xargs into the command aws codepipeline start-pipeline-execution --name. The -n1 argument tells xargs to use at most max-args of 1 per command line.

Jarvis standing desk and workspace configuration

Jarvis standing desk setup

I recently invested in a standing desk for my home office. On the recommendation of a friend, I purchased the Jarvis Laminate Standing Desk from fully.com.

I’ve been working predominantly from home for around 1.5 years now (at least 4 days a week), and since going fully remote at 5 days a week after COVID-19 I decided to invest more in the home office.

After having noticed myself slouching at my desk on more than one occasion, I got curious about standing desks. After reading about the apparent benefits of standing more (as opposed to sitting at a desk all day), and on the recommendation of a friend I pulled the trigger on this fully motorised setup.

Here are the specifications and complete configuration I went for:

  • Jarvis Laminate Standing Desk (160 x 80 cm, black finish)
  • Jarvis Lifting Column, Mid Range
  • Jarvis Frame EU, Wide, Long Foot Programmable
  • WireTamer Cable Trays (2x)
  • Topo Mini Anti-fatigue mat

The total for the desk parts and standing mat came up to around £600 excluding VAT, which I think is a great price considering the health benefits it should bring about over the longer term.

The build

I got stuck in one evening after work, and thought it might be 2 hour job. I was very wrong. Here is photo I took of the chaos I unleashed in the office after taking down my old desk and beginning assembly of the Jarvis.

jarvis desk assembly, defeat.
Note the crossed legs, socks, and beer as I take a breather, almost admitting defeat for the night.

Soon I was making progress though.

jarvis standing desk progress
jarvis desk assembled and upright

Multi-monitor configuration

I’ve got two computers that needed setting up. One is my PC, the other is an Apple Mac Mini (the main work machine). So next on the list was a decent adjustable multi-monitor stand. I ended up getting:

  • FLEXIMOUNTS F6D Dual monitor mount LCD arm

This was the strongest arm I could find. I could not (at the time) find anything that would fit my 34″ Acer Predator ultra wide LCD. (It’s pretty heavy).

Although this mount is only compatible up to 30″ LCDs, it seems to cope with my Acer Predator on one side, and my LG 5K Retina 27″ LCD display on the other.

the monitor dual mount setup on the desk

Around midnight that evening I finally had everything configured and in order. I booted up both machines and raised the desk to try it out.

Adjusting to, and actually working at the desk

I’m not fully committed to standing all day (at least not yet). I tend to spend around 3 hours standing each day, and the remainder sitting. I’m slowly increasing the standing time as I go on.

The Top Mini anti-fatigue mat definitely helps. I also noticed wearing my light, minimalist running shoes feels good while standing and working too.

topo mini anti-fatigue mat
The topo mini mat

Some more angles of the full setup

Lastly, here are some photos of the final setup. Certainly a farcry and total overhaul since the days of this home office setup of mine circa 2009!

In summary, I’m very happy with the new setup. My office is a little narrow and crammed, but this configuration helps me move around a lot more and I feel better for it.

This is post #8 in my effort towards 100DaysToOffload.

Kube Chaos game source and executable release

kube chaos screenshot destroying a pod

Source code here: https://github.com/Shogan/kube-chaos

I recently blogged about a small game I made that interfaces with any Kubernetes cluster and allows you to enter cluster nodes and destroy live, running pods.

I mentioned I would make the source available if there was any interest, so here it is.

After a little bit of code clean-up, and removing my 2D Shooter Bullet and Weapon System and 2D Neon Grid asset code from the game, the source is now available. These modules are game development modules/assets that I sell on the Unity Asset Store, and as such could not include in the source here. The fancy grid/background effect is therefore missing in this version.

I’ve also added an initial Windows build in the releases section, which is ready to go. Just unzip, run and enter your kube context/namespace details to get going.

the kube chaos start screen

If you decide to compile from source yourself, just make sure you have Unity 2019.4 or later installed and you should be good to go. I’ve not yet tested on macOS or Linux, but

This is post #7 in my effort towards 100DaysToOffload.

3D Printing Useful Parts

water tap design perspective view from fusion 360

I purchased my first 3D printer about 6 months ago after having putting the idea off for a year or so before that. For a while now I’ve had a fascination with being able to use 3D printing for hobby projects and bespoke creations.

From purchase until just recently I had only been printing 3D models for fun. Figurines, Star Wars replicas, and other random items. I even printed a Millenium Falcon (in two halves) and painted it for my oldest son.

After printing these pre-made 3D models, my oldest son and I sat together and created a simple ‘dropship’ model in Blender which we also printed.

These have all been a lot of fun, but after a while I had a hankering to use the 3D printer for some good around the house too.

3D printing useful parts

The printer I am using is the Elegoo Mars. It uses UV photocuring technology to print models out of a liquid resin. The printing plate lifts up from the resin basin and UV light is shone from underneath, emitted by an LCD panel. This cures the resin on the print plate one layer at a time.

elegoo mars uv lcd resin printer

The main benefit of this compared to FDM printers is that you can get a really good level of detail (resolution). One of the bigger downsides to this however is that the cured resin is more brittle.

The Elegoo Mars was a very cheap entry point for me to get started and I’m happy with the limitations (especially when considering the great detail that is achievable).

Printing a fake water tank handle

The reason I had this idea was to put in a fake water tank handle for our outdoors water butt.

Our 1.5 year old is currently a huge fan of running water and has learned how to open this tap and drain our collected rain water. One option would have been a lockable cover to go over the tap handle, but why do that when you can create your own solution?

I pulled out the tap handle and observed how it lets the water out when turned. Basically the tap handle shaft has a circular cut out on one side. Turning this aligns the hole with the opening in the water tank and allows water to flow through. Turning it further closes the hole again.

The ‘fake’ tap handle solution is pretty simple, replicate the part, but close the hole in the shaft.

Design

I’ve used Blender for more creative projects in the past, but decided to move onto something more suited for CAD and 3D printing. A friend recommended Fusion 360. After trying it out for this project I can highly recommend it.

It allows you to design your parts in stages. If at some point you want to reconfigure the design and change measurements, the process is simple. You go to that stage, apply the new measurement, and the software will ‘replay’ that through the remaining stages. Your whole design adjusts accordingly.

top view of the design
water tap design in fusion 360
The water butt tap design with no hole in the central shaft

Result

The resulting print shown below is actually the second iteration. The first design I made didn’t have a strong enough handle.

Here is the ‘fake’ water tap 3D print in action.

In the video above you can see the water tap in action. Turning it to any angle keeps the water flow blocked now. On the ground you can see the original water tap which is now just simply kept out of reach for when we actually need to use the water.

It is possible to pull the fake tap out as you can see above, but for a small toddler it is not easily possible. In order to keep it water tight, I had to get the measurement just right so as not to leak, but also not be too tight to remove again.

This is post #6 in my effort towards 100DaysToOffload.