Fossil Remnants of deleted VMs inside Web Frontend VM disk

Product Support Operations
1

Hi !

I have 2 VMs which were forever stuck in power-off or undeploy state (id = 33, 34). I deleted them through VM Actions → Recover → Delete. VMs disappeared from the list.
However, on web front end VM I see remnants of several hundreds GB
/var/lib/one/datastores/0/33 and 34.
This is really strange since I ran these VMs on separate servers KVM nodes, NOT inside the web front VM.
I assume I can safely delete these dirs - please correct if I’m wrong here.

Thanks in advance.
Andrei

2

Hi Andrei,

Thank you for the details.

Before removing those directories manually, we would recommend verifying that there are no remaining references to these VMs in OpenNebula, to avoid affecting other resources.

Please check the following:

  1. Confirm the VMs are fully removed:
onevm list | grep -E '33|34'
  1. Ensure there are no disks/images still referencing these VM IDs:
onevm show 33
onevm show 34

(these should return “VM not found”)

  1. Check if any files inside those directories are still in use by other VMs (e.g. shared/base images):
ls -l /var/lib/one/datastores/0/33
ls -l /var/lib/one/datastores/0/34
  1. (Optional but recommended) verify no processes are using those files:
lsof +D /var/lib/one/datastores/0/33
lsof +D /var/lib/one/datastores/0/34

If all checks confirm that:

  • the VMs no longer exist in OpenNebula

  • no other VM is referencing these files

then it should be safe to remove the directories manually.

3

OK, thanks a lot! Problem solved.

Looks like during creating VM one need to select only KVM host, but no datastore. Then based on the KVM host selection ON engine deploys correct datastore automatically.

Is my assumption of ON internals correct ?