# Hosts entering the ERROR state **URL:** \ **Category:** Product Support\ **Created:** [May 23, 2016, 7:43am UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273 "2016-05-23T07:43:36Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Yenya](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/yenya/32/4624_2.png) [@Yenya](https://forum.opennebula.io/u/Yenya)\ **Post date:** [May 23, 2016, 7:43am UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/1 "2016-05-23T07:43:36Z")
Hello, I have ONe cluster with ~25 hosts running CentOS 7 and ONe 4.14, and occasionally some of the hosts enters the ERROR state with the following message visible in Sunstone: > ERROR Mon May 23 09:19:40 2016 : Error monitoring Host myhost25 (48): error: failed to connect to the hypervisor error: error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. ERROR MESSAGE --8\<------ Error executing kvm.rb ERROR MESSAGE ------\>8-- ARCH=x86\_64 MODELNAME=“Intel(R) Core™2 Duo CPU E8500 @ 3.16GHz” The existing VMs deployed on the host are running without problems. I have discovered that running “service libvirtd restart” fixes the problem. So it might be that libvirtd stops accepting requests from kvm.rb for some reason. Other hosts are running OK, and the problem occurs on a different host next time. I see the problem with a host entering the ERROR state about once or twice a week. Can you help me to find out where the problem could be? Thanks! ---
**Author:** ![ondrej\_hamada](https://avatars.discourse-cdn.com/v4/letter/o/ea5d25/32.png) [@ondrej\_hamada](https://forum.opennebula.io/u/ondrej_hamada)\ **Post date:** [May 23, 2016, 8:03am UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/2 "2016-05-23T08:03:38Z")
Hi, I’m seeing the same problem with ONE 4.14 and Debian (both 7 and 8) Here’s a log from libvirtd on one host that has died yesterday: `2016-05-22 22:50:04.295+0000: 11577: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:50:04.317+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:50:34.376+0000: 11575: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:50:34.376+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:51:04.387+0000: 11576: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:51:04.388+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:51:34.399+0000: 11579: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:51:34.399+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:52:04.408+0000: 11578: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:52:04.409+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:54:59.479+0000: 11577: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:54:59.480+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:55:07.505+0000: 11580: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:55:07.506+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:55:38.703+0000: 11581: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:55:38.703+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:56:09.900+0000: 11575: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:56:09.901+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:56:41.100+0000: 11578: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:56:41.100+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 22:59:59.545+0000: 11574: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 22:59:59.545+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 23:04:59.609+0000: 11577: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 23:04:59.609+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 23:09:59.673+0000: 11576: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 23:09:59.673+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 23:14:59.735+0000: 11575: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 23:14:59.735+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 23:19:59.795+0000: 11578: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 23:19:59.795+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 23:24:59.858+0000: 11574: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. 2016-05-22 23:24:59.858+0000: 11573: error : virNetSocketReadWire:1571 : End of file while reading data: Input/output error 2016-05-22 23:29:59.922+0000: 11577: error : virDBusCall:1537 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.` ---
**Author:** ![alvaro\_simongarcia](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/alvaro_simongarcia/32/4178_2.png) [@alvaro\_simongarcia](https://forum.opennebula.io/u/alvaro_simongarcia)\ **Post date:** [July 5, 2016, 12:00pm UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/3 "2016-07-05T12:00:49Z")
Hi We are also affected by this issue, same problem, after a week running libvirtd service bites the dust and we get this message: > service libvirtd status -l > Redirecting to /bin/systemctl status -l libvirtd.service > ● libvirtd.service - Virtualization daemon > Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) > Active: active (running) since Thu 2016-06-23 00:59:50 CEST; 1 weeks 5 days ago > Docs: man:libvirtd(8) > [http://libvirt.org](http://libvirt.org) > Main PID: 19551 (libvirtd) > CGroup: /system.slice/libvirtd.service > └─19551 /usr/sbin/libvirtd > Jul 05 13:42:48 hyp115.swablu.os libvirtd[19551]: error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. > Jul 05 13:42:48 hyp115.swablu.os libvirtd[19551]: End of file while reading data: Input/output error And OpenNebula changes the host status to error state and running VM to unknown but the VMs are still running and virsh connect is still working. In this case we are running: - CentOS 7.2 - libvirt 1.2.17-13 - OpenNebula 4.14.2 We didn’t have this issue before, it seems that is related with the new libvirt version and the Opennebula probes, but I’m not sure. For the moment we didn’t have this issue in our OpenNebula 5.0 testing cluster. As @Yenya said the workaround is to restart libvirtd service but it’s a bit annoying. Has anyone else notice this issue with Opennebula and libvirtd? Cheers Alvaro ---
**Author:** ![alvaro\_simongarcia](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/alvaro_simongarcia/32/4178_2.png) [@alvaro\_simongarcia](https://forum.opennebula.io/u/alvaro_simongarcia)\ **Post date:** [July 5, 2016, 12:23pm UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/4 "2016-07-05T12:23:04Z")
I can confirm that this issue also affects to Opennebula 5.0 hypervisors running libvirtd 1.2.17-13 ---
**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\ **Post date:** [July 6, 2016, 8:43pm UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/5 "2016-07-06T20:43:13Z")
Hi, What is the possibility that you have the issue described by @vholer? > [@Polkitd traps general protection ip in libmozjs-17.0.so](https://forum.opennebula.io/t/polkitd-traps-general-protection-ip-in-libmozjs-17-0-so/399/3): > > Yes, I’m as well on CentOS 7.x . It’s related to PolicyKit, which crashes after many authentication checks. I would strongly recommend to disable PolicyKit support in libvirtd on RHEL/CentOS 7.x by adding following lines into /etc/libvirt/libvirtd.conf : auth\_unix\_ro = "none" auth\_unix\_rw = "none" unix\_sock\_group = "oneadmin" unix\_sock\_ro\_perms = "0770" unix\_sock\_rw\_perms = "0770" When PolicyKit fails, also the libvirt operations initiated by KVM IM probes or KVM VMM actions fail. OpenNebula … Kind Regards, Anton Todorov ---
**Author:** ![alvaro\_simongarcia](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/alvaro_simongarcia/32/4178_2.png) [@alvaro\_simongarcia](https://forum.opennebula.io/u/alvaro_simongarcia)\ **Post date:** [July 7, 2016, 8:36am UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/6 "2016-07-07T08:36:55Z")
Hi @atodorov_storpool Nice tip! yes it seems that is related with a polkit issue. We will try the workaround suggested by @vholer for the moment. Reading the bug discussion it should be fixed in polkit-0.113… Cheers and thanks! Alvaro ---
**Author:** ![alvaro\_simongarcia](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/alvaro_simongarcia/32/4178_2.png) [@alvaro\_simongarcia](https://forum.opennebula.io/u/alvaro_simongarcia)\ **Post date:** [July 7, 2016, 1:28pm UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/7 "2016-07-07T13:28:31Z")
btw we have generated a new CentOS polkit-0.133 rpm from Fedora 22 srpm package. And we have installed this in one hypervisor to see what happens after a while… Cheers Alvaro ---
**Author:** ![bms8197](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/bms8197/32/5288_2.png) [@bms8197](https://forum.opennebula.io/u/bms8197)\ **Post date:** [February 9, 2018, 9:47am UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/8 "2018-02-09T09:47:17Z")
I know it’s an old thread but I have this issue on a Dell Optiplex i7 CPU desktop computer running CentOS 7.4 latest, OpenNebula 5.4.6 and libvirt-3.2.0. From time to time libvirtd sees to crash with the message from this thread. @Anton Todorov fix seems to work as expected! (I had most of the settings in my libvirtd.conf file, except for the `auth_unix_ro = "none" auth_unix_rw = "none"` which were disabled). ---
**Author:** ![willemgf](https://avatars.discourse-cdn.com/v4/letter/w/df705f/32.png) [@willemgf](https://forum.opennebula.io/u/willemgf)\ **Post date:** [April 16, 2018, 6:53am UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/9 "2018-04-16T06:53:23Z")
We had the same problem, and my research turned out to this fix. However, after applying this fix, it still happens our libvirtd crashes. About once a week I need to restart the libvirtd-daemon. CentOS Linux release 7.4.1708 (Core) OpenNebula 5.4.1.1 libvirt-3.2.0-14.el7\_4.9.x86\_64 libvirtd.conf has the firx as described: auth\_unix\_ro = “none” auth\_unix\_rw = “none” unix\_sock\_group = “oneadmin” unix\_sock\_ro\_perms = “0770” unix\_sock\_rw\_perms = “0770” /var/log/messages: Apr 15 06:10:29 Node-1 libvirtd: 2018-04-15 04:10:29.145+0000: 187146: error : virDBusCall:1570 : error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. Apr 15 06:10:29 Node-1 libvirtd: 2018-04-15 04:10:29.149+0000: 187143: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Anyone else who is still having this problem, or has another fix? ---
**Author:** ![pianziva](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/pianziva/32/5961_2.png) [@pianziva](https://forum.opennebula.io/u/pianziva)\ **Post date:** [January 29, 2019, 1:08pm UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/11 "2019-01-29T13:08:20Z")
Hi, I was facing this issue often getting an error State on the host node Tue Jan 29 08:02:52 2019 : Error monitoring Host [xxxxx.com](http://xxxxx.com/) (21): Timeout executing ‘if [-x “/var/tmp/one/im/run\_probes”]; then /var/tmp/one/im/run\_probes kvm /var/lib/one//datastores 4124 60 21 [mwncloud5.maintenis.com](http://xxxxxx.com/); else exit 42; fi’ i already doing this step 1. remove /var/tmp/one and then onehost sync --force but still didnt solve the problem. 2. restart libvirtd daemon but still didnt solve the problem, any other solutions ? or what type of preimum support should i buy to solve this case ? ---
**Author:** ![mabdou](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/mabdou/32/6432_2.png) [@mabdou](https://forum.opennebula.io/u/mabdou)\ **Post date:** [February 4, 2019, 9:38am UTC](https://forum.opennebula.io/t/hosts-entering-the-error-state/2273/12 "2019-02-04T09:38:08Z")
Hello, Here are some general details about the OpenNebula Systems Support Subscriptions. [http://opennebula.systems/opennebula-support/](http://opennebula.systems/opennebula-support/) You will see that there are various options available to align with your SLA requirements (schedule and speed needed for issue resolution). Take a look at the documents here, and you can reach out directly to me if you would like to discuss options in more detail. Michael Abdou mabdou@opennebula.systems. Best regards.