Please, describe the problem here and provide additional information below (if applicable) …
We’ve successfully implemented a backup server while using this manual: Backup Datastore: Restic |
The KVM nodes can reach the backup server over SFTP and can also access the Sunstone server passwordlessly with the user oneadmin. Backups are also successfully made and stored on the backup server.
We use Restic on the Sunstone front-end to perform incremental backups of the VMs that use Ceph RBD as the storage back-end on the backup server over SFTP.
We run into two issues with the back-up system. The first one is that we get the following error when we go over the “KEEP_LAST“ value:
Mon Nov 24 14:40:35 2025 : Error flattening backup increments: Command failed:export LANG=Cexport LC_ALL=Cset -e -o pipefail; shopt -qs failglobexport RESTIC_BIN=“/var/tmp/one/datastore/restic/restic"export RESTIC_PASSWORD=‘REDACTED’”$RESTIC_BIN" ‘–no-lock’ ‘–json’ ‘–repo=/var/lib/one/datastores/108/job_4’ snapshots ‘ac909d09’ | jq --slurp ./var/lib/one/remotes/datastore/restic/restic.rb:203:in
query_snapshot': bash: line 6: jq: command not found (StandardError)from /var/lib/one/remotes/datastore/restic/increment_flatten:152:in’
The snapshots aren’t flattened, but the back-up is performed. What’s interesting is that the documentation mentions:
In both the Enterprise and Community editions of OpenNebula, the correct version of restic is included as a dependency.
And under the “Repository Maintenance and Troubleshooting“ heading the Restic command is also mentioned, to for instance check whether a repository is locked, but the Restic command isn’t available on the front-end server. We’re now wondering what could be the curlpit here and if we’re missing something that might cause the Restic command to be missing.
Versions of the related components and OS (frontend, hypervisors, VMs):
frontend node:
- Debian 12
- Opennebula 7.0.1 fully installed according to Single Front-end Installation |
KVM nodes:
- Debian 12
- Opennebula 7.0.1 fully installed according to KVM Node Installation |
- Ceph 19.2.3 Squid
- qemu-system-x86_64 version 10.0.2 from bookworm-backports
Restic repository:
- Debian 13
- OpenZFS storage with a symlink to /var/lib/one/datastores
Steps to reproduce:
- Build a fresh Opennebula cluster on Debian 12 with a Ceph datastore that’s configured with Opennebula according to Ceph Datastore |
- Configure a Debian 13 machine to store the backups on
- Configure the Restic datastore with the following config:
NAME = “RBackups”
TYPE = “BACKUP_DS”DS_MAD = “restic”
TM_MAD = “-”RESTIC_PASSWORD = “REDACTED”
RESTIC_SFTP_SERVER = “10.60.1.210”
BRIDGE_LIST = “10.60.2.10,10.60.2.20,10.60.2.30”
It doesn’t matter here whether the “BRIDGE_LIST“ is configured or not. The problem arises no matter if it’s configured or not.
Current results:
Error:
Mon Nov 24 14:40:35 2025 : Error flattening backup increments: Command failed:export LANG=Cexport LC_ALL=Cset -e -o pipefail; shopt -qs failglobexport RESTIC_BIN=“/var/tmp/one/datastore/restic/restic"export RESTIC_PASSWORD=‘REDACTED’”$RESTIC_BIN" ‘–no-lock’ ‘–json’ ‘–repo=/var/lib/one/datastores/108/job_4’ snapshots ‘ac909d09’ | jq --slurp ./var/lib/one/remotes/datastore/restic/restic.rb:203:in
query_snapshot': bash: line 6: jq: command not found (StandardError)from /var/lib/one/remotes/datastore/restic/increment_flatten:152:in’
Snapshots aren’t flattened
Restic command is not available for any user on the front-end server
Restic is not available at /var/tmp/one/datastore/restic/restic. Also not during the execution of the back-up.
Expected results:
- No error when exceeding the KEEP_LAST varaible
- The Restic command to be available for user oneadmin to make sure we’re able to perform maintenance and troubleshooting steps when necessary
