How to recover from FAILED state?

Community Support
1

This is on Openebula 4.10.0
The VM created succesfully. I can ssh to it. Then I tried to Migrate to other node. It put VM to FAILED state. My question is what is the right way to recover from FAILED. It appears that any options I choose (PLAY, BOOT, RECOVER) not working due to “Wrong state to perform that action”. Should I try “DELETE”? What DELETE really do?

Here is the log when it’s in failed state.
Mon Mar 30 16:14:40 2015 [Z0][DiM][I]: New VM state is ACTIVE.
Mon Mar 30 16:14:40 2015 [Z0][LCM][I]: New VM state is PROLOG.
Mon Mar 30 16:14:46 2015 [Z0][LCM][I]: New VM state is BOOT
Mon Mar 30 16:14:46 2015 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/81/deployment.0
Mon Mar 30 16:14:46 2015 [Z0][VMM][I]: ExitCode: 0
Mon Mar 30 16:14:46 2015 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Mon Mar 30 16:14:49 2015 [Z0][VMM][I]: ExitCode: 0
Mon Mar 30 16:14:49 2015 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
Mon Mar 30 16:14:49 2015 [Z0][VMM][I]: ExitCode: 0
Mon Mar 30 16:14:49 2015 [Z0][VMM][I]: Successfully execute network driver operation: post.
Mon Mar 30 16:14:49 2015 [Z0][LCM][I]: New VM state is RUNNING

Mon Mar 30 18:43:44 2015 [Z0][LCM][I]: New VM state is SAVE_MIGRATE
Mon Mar 30 18:44:35 2015 [Z0][VMM][I]: ExitCode: 0
Mon Mar 30 18:44:35 2015 [Z0][VMM][I]: Successfully execute virtualization driver operation: save.
Mon Mar 30 18:44:35 2015 [Z0][VMM][I]: ExitCode: 0
Mon Mar 30 18:44:35 2015 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Mon Mar 30 18:44:35 2015 [Z0][LCM][I]: New VM state is PROLOG_MIGRATE
Mon Mar 30 18:44:36 2015 [Z0][LCM][I]: New VM state is BOOT
Mon Mar 30 18:44:36 2015 [Z0][VMM][I]: ExitCode: 0
Mon Mar 30 18:44:36 2015 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Mon Mar 30 18:44:36 2015 [Z0][VMM][I]: Command execution fail: /var/tmp/one/vmm/kvm/restore '/var/lib/one//datastores/0/81/checkpoint' 'node-002.dc1.xxx.com' 'one-81' 81 node-002
.dc1.xxx.com
Mon Mar 30 18:44:36 2015 [Z0][VMM][E]: restore: Command "virsh --connect qemu:///system restore /var/lib/one//datastores/0/81/checkpoint" failed: error: Failed to restore domain from 
/var/lib/one//datastores/0/81/checkpoint
Mon Mar 30 18:44:36 2015 [Z0][VMM][I]: error: unsupported configuration: Unable to find security driver for label apparmor
Mon Mar 30 18:44:36 2015 [Z0][VMM][E]: Could not restore from /var/lib/one//datastores/0/81/checkpoint
Mon Mar 30 18:44:36 2015 [Z0][VMM][I]: ExitCode: 1
Mon Mar 30 18:44:36 2015 [Z0][VMM][I]: Failed to execute virtualization driver operation: restore.
Mon Mar 30 18:44:36 2015 [Z0][VMM][E]: Error restoring VM: Could not restore from /var/lib/one//datastores/0/81/checkpoint
Mon Mar 30 18:44:36 2015 [Z0][DiM][I]: New VM state is FAILED
2

Use “delete and recreate”

3

Hi,

Unfortunately, once the VM enters the failed state it can’t be recovered. The only option is to shutdown the VM manually, and then delete it from OpenNebula.

We are working to improve this in the next release. See #3654 for details.

4

Hello,
I’ve just moment ago solved this.
First of I use ONE v4.12 (Debian 7) + MySQL, shared LVM datastore, xen 4.5.

  1. When a VM get status FAILED I tried to run it using direct communication with hypervisor.
    I.e. xl create /Path/to/vm/config
    In the VM Config file was written the fitted for VM options (I mean MAC, Disk, RAM, IF)
    Finaly VM stared successfully.
  2. Turn off the VM.
  3. Stop all opennebula services.
  4. Now start working with DB.
  • Just in case make a full db backup.
    Then I’m trying to show WHAT and WHERE it was changed step-by-step.
    Here VM_ID - ID of FAILED VM
    mysql -u root -h <MYSQL_IP> -p
    mysql> delete from vm_monitoring where vmid=VM_ID; # I’m nut sure is it really need, but I don’t want to make such kind of “test” one more time.
    mysql> update vm_pool set state=8 where oid=VM_ID;
  1. Try to start back opennebula services.
  2. Check the VM state it should be “POWEROFF”.
  3. Try to start VM (in my case it looks like regular start of regular VM)

Hope it help!
P.S. Message for developers: Guys, It’s really great Cloud solution, one of the best, but offer to users just RECREATE filed VM is horrible solution! VM can be failed because of network, storage, e.t.c. reasons but environment can be repaired and failed VM also have to be too.

ONE 4.14 already has tools for recovering FAILED VM. But I can’t install it cause it requires to update to Jessie, but this release is not have Pacemaker packet witch I use.
http://docs.opennebula.org/4.14/release_notes/release_notes/whats_new.html