# Random VMs incorrectly in POWEROFF state after upgrade to 5.12

**URL:** <https://forum.opennebula.io/t/random-vms-incorrectly-in-poweroff-state-after-upgrade-to-5-12/9903>\
**Category:** Installation & Configuration\
**Created:** [September 3, 2021, 1:13pm UTC](https://forum.opennebula.io/t/random-vms-incorrectly-in-poweroff-state-after-upgrade-to-5-12/9903 "2021-09-03T13:13:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nico\_Schottelius](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/nico_schottelius/32/6551_2.png) [@Nico\_Schottelius](https://forum.opennebula.io/u/Nico_Schottelius)\
**Post date:** [September 3, 2021, 1:13pm UTC](https://forum.opennebula.io/t/random-vms-incorrectly-in-poweroff-state-after-upgrade-to-5-12/9903/1 "2021-09-03T13:13:32Z")

</div>

Hello,

after upgrading from 5.10 to 5.12.0.4 many VMs on different hosts are seen in POWEROFF state with the infamous message:

```auto
Tue Aug 31 15:47:17 2021 [Z0][LCM][I]: VM running but monitor state is POWEROFF
Tue Aug 31 15:47:17 2021 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Tue Aug 31 15:47:19 2021 [Z0][VM][I]: New state is POWEROFF
Tue Aug 31 15:47:19 2021 [Z0][VM][I]: New LCM state is LCM_INIT

```

virsh list shows the VMs within ca. 1s, however using poll.rb, I only see 2-3 VMs in the output.

```auto
[14:43:20] server10.place6:/var/tmp/one/im/kvm-probes.d/vm$ ruby monitor/poll.rb | wc -l                                                                                   
1
[14:43:25] server10.place6:/var/tmp/one/im/kvm-probes.d/vm$
[14:43:29] server10.place6:/var/tmp/one/im/kvm-probes.d/vm$ ruby monitor/poll.rb | wc -l                                                                                   
3
[14:45:24] server10.place6:/var/tmp/one/im/kvm-probes.d/vm$ virsh --connect qemu:///system list | wc -l                                                                      
188

```

I already ran `onedb fsck`, which correct some quotas and complains about incorrect leases, but aside from that did not report any issue.

What’s the next best step for debugging this?

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ruben/32/19_2.png) [@ruben](https://forum.opennebula.io/u/ruben)\
**Post date:** [September 3, 2021, 4:05pm UTC](https://forum.opennebula.io/t/random-vms-incorrectly-in-poweroff-state-after-upgrade-to-5-12/9903/2 "2021-09-03T16:05:53Z")

</div>

The monitor process changed between those two. Best process is to remove the /var/tmp/one process in the hosts and execute a onehost sync -f to force the update of the hosts.

The you can disable/enable the hosts to trigger a restart of the monitor agent (or wait a couple of minutes for the automatic recovery…)