Delete last snapshot for qcow2 based disk ends with failure

Hi,
this a scenario:

  • do snapshot_A
  • delete snapshot_A

This operation ends with failure, due to the message:
image

Also ‘Flatten’ option does not work properly:
image

Why deleting the last snapshot is not allowed?

2 Likes

In other situations (to be honest, I don’t know what is the difference), we got the message:
image

In the first case you are trying to manage the image snapshots directly. This has some limitations as you cannot delete the active snapshot, because is currently in use by a VM.

In the second case, you are trying to manage the VM disk snapshots. The error you are facing seems to be that the snapshot you are trying to delete has some child snapshots that depend on it. You’d have to delete those first.

Note that these operations, although seemingly the same, use two different sets of API Calls.

1 Like

Hi,
thanks for such a fast response @dclavijo . So, there is no way to do this scenario?

  • a disk without any snapshot
  • create snapshot
  • delete this snapshot

In the documentation, I found that:

  • disk-snapshot-delete <vmid> <diskid> <snapshot_id>: Deletes a snapshot if it has no children and is not active.

So how to manage this kind o situation? How to delete this snapshot? Is this even possible?

Also,
is there any chance to add to OpenNebula support for external snapshots for qcow2-based disk? I think that we find the root problem - the lack of this option. If we do this kind of action manually directly on the KVM host, after rebooting, the configuration of the machine lost this entry from xml config file.

We have discussed this case in this issue. The problem is explained in detail here as why you can’t delete active snapshots nor snapshots with children.

1 Like

Hi @Mateusz_M
We don’t understand why OpenNebula doesn’t add the feature of external snapshot with qcow2 image.
They already know how to use external snapshot because they use this type of snapshot on the backup solution built-in opennebula (Restic).

I hope they will add the possibility to made external snapshot directly on sunstone/Fireedge.

Kind regard

Hi @Mateusz_M

Just a few comments:

  • OpenNebula DOES USE external snapshots for disk snapshot operation. It use external snapshots: You can see the operation here: one/src/tm_mad/qcow2/snap_create_live at master · OpenNebula/one · GitHub

  • OpenNebula DOES allow to delete snapshots through the revert operation. it cannot be active (because VM is writing changes there) it cannot have children

  • Note that OpenNebula implements a general snapshot dependency chains.We allow any revert operation, so you can end up with a backing file dependency as the one shown:

┌──────────┐     ┌──┐     ┌──┐    ┌─────────┐
│Base Image│◄────┤1 │◄────┤3 ◄────┤ WR SNAP │
└──────────┘     └──┘     └──┘    └─────────┘
                   ▲        ▲
                   │        │
                   │        │
                 ┌─┴┐     ┌─┴┐
                 │ 2│     │ 4│
                 └──┘     └──┘

(WR SNAP is the file where the VM is actually writing changes)

Let’s say I want to remove snapshot 3, I cannot commit to snapshot 1 (i.e. merge contents of 3 into 1) because it has children (2 in this case that will break). So, in general We cannot benefit of the block commit operation. That is possible with linear chains:

┌──────────┐     ┌──┐     ┌──┐    ┌─────────┐
│Base Image│◄────┤1 │◄────┤3 ◄────┤ WR SNAP │
└──────────┘     └──┘     └──┘    └─────────┘

We are using the blockcommit operation for the backup interface so any backup solution as well as the built-in ones can use this temp snapshots for backup procedures.

1 Like

OK, but we’re talking here about simple temporary snapshot without tree. You can easily block operations that could damage the tree if you see the external snapshots are present. Now the answer we also received is that we’re supposed to SSH and directly create external snapshot, but what if in the meantime the VM gets down? Power outage/VM migration etc.? OpenNebula won’t point to the correct active disk, as it is unaware about the external snapshot that has been created.

Can you please add the mode that allows the use of external snapshots and at the same time blocks the revert situation you described before? Users would have a choice. It is very rare to build sophisticated snapshot tree in production.

And one more thing I’m not sure about the answer above is that reverting is not the same as removing the snapshot. In general, it just should change the active snapshot file you’re working on. While removal of the snapshot doesn’t mean you have to remove the active snapshot - you still should be able to remove any snapshot in the chain if it is not actively used without reversion as these are not supposed to be required. If I want to revert a VM I usually have 2 options - preserve the current state or remove it. In the approach that you described - revert action would lose data that has been changed in the meantime. And both of these possibilities are provided by external snapshots.

So for temporal snapshots and backups. The best way to integrate a new backend is to use the new backup drivers. It offers several improvements:

  • Use of tmp snapshots to generate full backups. We can also expose this tmp snapshots to the backup drivers and not commit changes to the snapshot file.
  • snaphost multiple disks in parallel for crash consistency.

So no need to SSH and create the external snapshot directly, but through the drivers. This way OpenNebula coordinates all the operations. It checks quotas etc…

I agree that we can relax the delete operation so it can be actually do a blockcommit if the snapshot layout is linear and we can rebase. Note that the limitation is not for the snapshot not being external (which in OpenNebula are) but for the backing file chains.

We can add this behavior to the driver and OpenNebula core. I will update with timeframe for this future and link to github issue

ok, can you provide us links on how we can use the driver methods you mentioned? can it be invoked via API in some way? Or we have to install some kind of module or something inside the OpenNebula? we would need to have basic create/remove snapshot invoked on-demand not via Backup button in OpenNebula

The driver methods I mentioned are described here. You can also take a look to the rsync driver as a reference:. Just look into the actions descfribed under the backup section in the driver guide.

Also this actions generate a “standalone” copy of the disks, so it can be recovered anywhere. We can modify the process to have just the snapshots (without committing the full chain into them). In any case this require a xmlrpc backup operation either button or API call to happen

This, I think, will provide the best experience as it is integrated with all the OpenNebula interfaces and abstractions. It will also leverage the some new futures like backup jobs etc…

I reckon this can be a more demanding process and the ability of delete snapshots mid chain is also interesting per se. We have planned this option for the next OpenNebula release, ant it will be back-ported to next maintenance as well (it will maintain the same interface). More info and tracking in here

Thanks for the feedback :slight_smile:

.

1 Like

Quick update on this issue. The feature has been implemented and ready for release. There are further improvements we want to do related to ssh drivers (now full copies are done for snapshots) that are being address in a separate issue.