Docker Compose Backups in TrueNAS Scale

A good friend at work recently demoed some of the applications they have running in their homelab setup to me. These were applications that I'd heard about, but never actually saw in action, and their demo was nothing short of the Steve Jobs iPhone reveal moment back in 2007. Stuff I had been doing manually for years now had a nice web front end, as well as automation that would have been killer during those times. I needed to have it!

Since then, I've sunk some considerable time into my own setup, to the point where it has moved away from being a purgeable plaything I couldn't care less about to a production setup that I would really be sad about losing.

I already had cloud backups for container config and data, but there didn't seem to be an obvious way to back up the Docker Compose configurations available to custom apps starting with Electric Eel. I knew where they were (/mnt/.ix-apps), and even found a full/complete Compose Backups at /mnt/.ix-apps/backups, but those were tagged "system-update", seemingly only triggered when TrueNAS Scale ran an upgrade.

With the amount of iterations I've made to the setup, I wanted something that I had more control over, rather than just being at the mercy of system updates. Preliminary searches didn't yield anything conclusive, until I stumbled upon this thread:

The post links to the source code for middlewared, which from a layman perspective seemed to be one of the core components driving TrueNAS Scale.

From there, I kept poking around til I found this file:

Bingo - this is exactly what I needed to trigger! The question that remained wasn't the "what", it was the "how".

From that same TrueNAS thread, people were suggesting that the TrueNAS API was the way to go about it. This might have been true at that time, but on my TrueNAS version (Fangtooth), the API was deprecated. Further down, they mentioned something called midclt, which a quick Google resolved to:

Even closer! Now I just needed to carve out the right commands and wrap around them for ease of repeated use. At the end of it all, here's what I came up with:

#!/bin/bash

ARCHIVE_DEST=<path>
BACKUP_PATH=<path>

backup () {
    PREVIOUS_UPDATE=$(cli -c "app docker list_backups" | grep -v "system-update" | grep -oP '\d{4}-\d{2}-\d{2}_\d{2}:\d{2}:\d{2}')
    cli -c "app docker delete_backup backup_name=\"$PREVIOUS_UPDATE\""
    BACKUP_NAME=$(cli -c "app docker backup" | tail -n 1)
    tar cJf $ARCHIVE_DEST/$BACKUP_NAME.tar.xz $BACKUP_PATH/$BACKUP_NAME/
}

prune () {
    KEEP=10
    find $ARCHIVE_DEST -maxdepth 1 -type f -name '*.tar.xz' \
    -printf '%T@ %p\n' | sort -nr | tail -n +$((KEEP + 1)) | cut -d' ' -f2- | while read -r old_file; do
        rm -f "$old_file"
    done
}

backup

if [ "$(ls $ARCHIVE_DEST | wc -l)" -gt 10 ]; then
    prune
fi

Some background on the script:

My TrueNAS instance has definitely gone through more than one system update, and yet I only found one system update entry. In lockstep with this observation, I wrote the script such that only one manual backup should be loaded in the system backup location at one time. All the others should reside in an archival path somewhere else.
Went with the more modern tar.xz over tar.gz - may go back and try tar.zst in the future.
Don't want infinite growth in these backups, so rotate them and only keep 10 most recent ones.
The backups at /mnt/.ix-apps/backups didn't seem to have any kind of encryption so I opted not to do any to the resulting tar.xz files. Though, I am putting them on an encrypted ZFS Dataset, and all my cloud backups receive rclone encryption, so I feel that they are reasonably protected with the weakest link having little difference to the unencrypted system backups.

I have the above running in a weekly Cron job that I configured through the TrueNAS Advanced Settings, and it has a companion Cloud Sync Task that syncs the ARCHIVE_DEST to my cloud backup storage at a similar cadence.

Now I have greater peace of mind with this setup, knowing that in addition to the config/data of the applications, I also have the corresponding Docker Compose files backed up to pull off a full restoration if needed!