Hello @mptek, I explained nature of non-persistent and persistent images/instances in my prev posts. I already discuss this problem with few my customers. It all depends on usage of opennebula.
If you use it in “cloud” way, them it is like amazon ec2 instances (vm instances), ami images (non-persistent images) and ebs volumes (persistent images). So customer deploys new instances based on ami image (non-persistent image) and configure everything “infrastructure as code” using tools like ansible. That programatic configuration also includes attaching and mounting ebs volume (persistent image) which can be backuped. So in case of crash, they can easily deploys and configure everything by running “program”. Important data will be on ebs volumes, so preserved.
On the other hand, many people use openenbula as “vps” like hosting and deploys non-persistent instances/images. In this case there are two possible solutions (in my subjective opinion).
First one is to make every customer VPS persistent, so clone base image and make it persistent, clone base VM template, map it to that image, configure persistent IP, etc… and deploy vm.
Second one is deploying non persistent instances and do backups of that “non-persistent” images created on background.
I am currently using ssh transfer datastores on some of my hosts. Images are deployed persistent and with driver=qcow2.
Unfortunately your backup script does not cover this scenario (TM_MAD=ssh, Image persistent, Image “Driver = SSH”).
I believe technically it should work with the same commands from your script. Am I correct? If yes, do you see a chance to implement this?
the version in the develop branch does not backup the image as I expected. To be more precise about my goal:
I deployed a VM to a KVM node with local storage and a system datastore (TMD_MAD=ssh). I checked the box “persistent” while deploying the VM. Now I would like to backup the image located in the system datastore from the running VM directly from this KVM node.
When I run your script (pointing to the system datastore using a label) it just does nothing.
I did some more testing and can report following results. In my first tests I only included the system ds (ID: 109) on the kvm node in the backup run. In my oppinion this should be have been sufficient because the whole persistent image is located there. The backup.sh just did nothing in this tests.
In my next tests I included only the image in the backup run (based on label, not the complete ds) and I think I found a bug in your script:
Snapshot is created succesfully
rsync tries to copy the image file from datastore id: 110, which is the originating image datastore only existing in the frontend node. It should copy from id: 109, which is the ssh datastore on the kvm_node.
Here is the ouptut (look at the bold part):
[oneadmin@backup addon-image-backup]$ ./backup.sh
Create live snapshot of image 5 named testpersistent-disk-0 attached to VM 16 as disk sda running on kvm01.inf.corp.loc
Run cmd: ssh oneadmin@kvm01.inf.corp.loc ‘touch /var/lib/one/snapshot_image_backup/one-16-weekly-backup’
Run cmd: ssh oneadmin@kvm01.inf.corp.loc ‘virsh -c qemu:///system snapshot-create-as --domain one-16 weekly-backup --diskspec sda,file=/var/lib/one/snapshot_image_backup/one-16-weekly-backup --disk-only --atomic --no-metadata --quiesce’ || ssh oneadmin@kvm01.inf.corp.loc ‘virsh -c qemu:///system snapshot-create-as --domain one-16 weekly-backup --diskspec sda,file=/var/lib/one/snapshot_image_backup/one-16-weekly-backup --disk-only --atomic --no-metadata’
Domain snapshot weekly-backup created
Run cmd: mkdir -p /mnt/image_backup/110/6a32b4fe232caa0b36287e2cbb91cc8e.snap
Run cmd: rsync -aHAXxWv --inplace --numeric-ids --progress -e “ssh -T -o Compression=no -x” oneadmin@kvm01.inf.corp.loc:/var/lib/one//datastores/110/6a32b4fe232caa0b36287e2cbb91cc8e /mnt/image_backup/110/6a32b4fe232caa0b36287e2cbb91cc8e.tmp
receiving incremental file list
rsync: change_dir “/var/lib/one//datastores/110” failed: No such file or directory (2)
sent 8 bytes received 100 bytes 43.20 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1650) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)
Hello, would you print here your datastores? If I understand correcttly, them persistent image is created in system? datastore on specific compute node? or it is image datastore?
Hi Kristian,
I apologize for the delay in your reply.
I’m trying to better understand the operation of ON so as not to waste anyone’s time.
In your previous post write:
Literally I understand that the VM created “non-persistent” will have as a backing-file the image of the VM template. As the example of “Snowman”:
But if I do a qemu-img info of my qcow2 disks (of non-persistent VM) i get this (no backing-file):
root @ sto1-a: / var / lib / one / datastores / 0/83 # qemu-img info disk.0
image: disk.0
file format: qcow2
virtual size: 200G (214748364800 bytes)
disk size: 12G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
Why?
It looks like a copy of the template image (complete).
The first case for the solution you listed is IMHO laborious.
As I use it, opennebula assigns various parameters automatically (eg IPv4, IPv6, hostname, rootpwd, etc …).
If you open sunstone to internet my clients could create VMs by setting only the parameters strictly necessary leaving others decided by ON (eg IPs).
Hello, I think that it deponds on datastore type. I use shared qcow2 files datastore for images and for system too. I have clustered setup with gfs2 running on top of it. So it uses backing files for non-persistent VM and are in system DS. Persistent VMs each have own image in image DS.
If you use non shared SSH DS, them image is copied from frontend node via ssh to compute node, so no backing file.
My backup script supports only qcow2 datastore or shared datastores, not ssh/local
Hello Feldsam.
You are right. I use shared QCOW2 for SYS DS and shared (only) for IMG.
ON copy the base qcow2 file to SYS DS (without backing file).
But i prefer this mode. Every VM have separate and indepndent storage file.
root@sto1-a:~# onedatastore list
ID NAME SIZE AVAIL CLUSTERS IMAGES TYPE DS TM STAT
0 Storage_VM 13.9T 95% 0 0 sys - qcow2 on
1 Immagini_VM 13.9T 95% 0 2 img fs shared on
2 Files 13.9T 95% 0 0 fil fs shared on
I will look to modify your script to “accomodate” my setup.
Do you think is a heavy mods or simple?
Hello, I also prefer to have separate image for each VM, but I use persistent disks instead of non-persistent. My backup script also counts with backuping persistent disks only.
In case of use persistent disks, oned use system ds for just deployment files (virsh xml, symlink to image datastore)
I joined OpenNebula’s forums looking for a script that would do exactly what another user you spoke to was hoping for. This other user wanted to Borg store a bunch of System states from OpenNebula. You said you back them up, and you have a script.
This script, what you share here, I hope to adopt to my needs 6 years after its release. Thank you for your work, and your generosity. I think I’ll be leaving OpenStack (as it can’t capture live states) and joining OpenNebula thanks to you.
Hello @FileneTaylor I rewritten my original script to python and I am using borg backup. I also have implemented restic and kopia for testing purposes.
It is also opensource by it is tied to opennebula 3par driver, so it doesnt backup qcow2 - not implemented.
Btw, in meantime, opennebula implements own backup mechanism, but I dont have time/prioritu to look into it
I am very excited to do the work and get Open Nebula working, configured, and backing up regularly! I did expect to use QCow2, and a lot has changed like their own Backup function… I have no idea if I’ll prefer 3Par!
Thank you deeply for sharing your kind gifts and efforts with us!! I truly hope the OpenNebula and Open Source communities recognize you and support you and your endeavors!! ( feldhost.cz !! )
I have a lot to research… I’ll gladly accept any quick notes you have left to share. . Thank you again.
Hi there, Rubén Montero, and thank you for replying!
One of the largest reasons I found this thread is because of the Borg backup feature. Part of my plan for using Borg isn’t just because of how it reduces used space, but because I plan on distributing backups of my virtual machine!
If it’s possible, or already included, to use Borg backups within OpenNebula for saving, loading, and restoring backups, that will save me a lot of trouble using Kristian Feldsam’s scripts for Node or Python. However, I do think my need is such an edge case I will be lucky to find that OpenNebula is already attuned for my needs. I think other users will either need to use the OpenNebula restore-from-backups feature to load my distributions, or they’ll need to find a way to use the backup snapshot as OpenNebula captures it.
So far, this is the closest means to what I’m working for. OpenStack doesn’t capture running states, for example. Being able to capture a running state, while being compatible with QEMU and libvirt puts OpenNebula ahead of other non-open source solutions for me. (Open Source is an important part in being assured of the contents of what I’m running.
Since I have a bunch of confusion I’m still busy clearing up, the help of others in Open Nebula would be of great assistance to me. Thank you both, utterly and sincerely.
@FileneTaylor From what I know, there is only restic implementation in enterprise - paid version. I am open to help you with developing custom backup driver, to use opennebula native backup system with borg as backend. If you are interested, please contact me on feldsam@feldhost.cz
I plan on contacting you after I have my home server reimaged in the next week. Thank you for offering your help, I’ll start a new thread and link to it to help marry the latest from OpenNebula and the aid all your versions of work provide me.
Ok, thanks. That link to the Tuning Guide makes it very clear, although I have some security related doubts with this setup (non-authenticated and non-encrypted). Network isolation should be bullet proof. Just my opinion…
)
. Thank you again.
