# Recover vms stuck in UNKOWN state

**URL:** <https://forum.opennebula.io/t/recover-vms-stuck-in-unkown-state/9732>\
**Category:** Product Support\
**Created:** [July 23, 2021, 6:35am UTC](https://forum.opennebula.io/t/recover-vms-stuck-in-unkown-state/9732 "2021-07-23T06:35:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Spencer\_Owen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/spencer_owen/32/6704_2.png) [@Spencer\_Owen](https://forum.opennebula.io/u/Spencer_Owen)\
**Post date:** [July 23, 2021, 6:35am UTC](https://forum.opennebula.io/t/recover-vms-stuck-in-unkown-state/9732/1 "2021-07-23T06:35:50Z")

</div>

We upgraded our cluster from 5.10 to 5.12. We mistakenly had a few hosts that were downgraded back to 5.10.

Even after re-upgrading those KVM hosts to 5.12, the VMs on those host are stuck in UNKOWN state.

How do we recover these VMs?

## Answer

We recovered 90% of the vms by running the followiong on the opennebula node

```auto
sudo -u oneadmin onehost sync --force

```

To recover the vms on a specific host, add the host ID

```auto
vms=$(onevm list --csv | grep unkn | tail -n 1) #Picks a single VM stuck in UNKNOWN state
vm_id=$(echo $vms | awk '{split($0,a,","); print a[1]}' )
vm_name=$(echo $vms | awk '{split($0,a,","); print a[4]}' )
kvm_name=$(echo $vms | awk '{split($0,a,","); print a[8]}' )

echo $vm_id
echo $vm_name
echo $kvm_name

sudo -u oneadmin onehost sync $kvm_name --force

```

For the remaining vms we did the following

1. Ensure the host is enabled

```auto
onehost enable $kvm_id # you can get the id with 'onehost list'
onehost show $kvm_id | grep 'STATE\|LAST' # Wait until this transisions from INIT to MONITORED

```

1. Restart libvirt-bin on the kvm host

```auto
sudo service libvirt-bin restart

```

1. Re-run the sync on the nebula host

```auto
sudo -u oneadmin onehost sync --force

```

---

<div class="post-metadata">

**Author:** ![Spencer\_Owen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/spencer_owen/32/6704_2.png) [@Spencer\_Owen](https://forum.opennebula.io/u/Spencer_Owen)\
**Post date:** [July 23, 2021, 5:53pm UTC](https://forum.opennebula.io/t/recover-vms-stuck-in-unkown-state/9732/2 "2021-07-23T17:53:44Z")

</div>

If the host _still_ isn’t updating, Try killing the monitord process, removing the pid and then forcing a resync.

```auto
# On host
ps aux | grep monitord
kill <pid of monitord>
rm /tmp/onemonitord-<hostid>.error
rm /tmp/one-monitord-<hostid>.pid

```

```auto
# On OpenNebula
onehost enable <hostid>
sudo -u oneadmin onehost enable <hostid> --force
sudo -u oneadmin onehost forcesync

```

---

<div class="post-metadata">

**Author:** ![Spencer\_Owen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/spencer_owen/32/6704_2.png) [@Spencer\_Owen](https://forum.opennebula.io/u/Spencer_Owen)\
**Post date:** [July 23, 2021, 6:29pm UTC](https://forum.opennebula.io/t/recover-vms-stuck-in-unkown-state/9732/3 "2021-07-23T18:29:05Z")

</div>

If this doesn’t resolve the issue try the following

```auto
# On KVM host
mv /var/tmp/one /var/tmp/one.backup

```

We did have a few hosts that took a couple of sync attempts to to fix

```auto
onehost enable 72
sudo -u oneadmin onehost sync 42 --force

```

---

<div class="post-metadata">

**Author:** ![Spencer\_Owen](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/spencer_owen/32/6704_2.png) [@Spencer\_Owen](https://forum.opennebula.io/u/Spencer_Owen)\
**Post date:** [July 27, 2021, 7:54pm UTC](https://forum.opennebula.io/t/recover-vms-stuck-in-unkown-state/9732/4 "2021-07-27T19:54:19Z")

</div>

> [@Spencer\_Owen](#):
>
> ```auto
> sudo -u oneadmin onehost enable <hostid> --force
> 
> ```

Should be

```auto
sudo -u oneadmin onehost sync <hostid> --force

```