# OpenNebula suddenly dies after upgrade to 5.4

**URL:** <https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700>\
**Category:** Product Support\
**Created:** [July 31, 2017, 1:20pm UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700 "2017-07-31T13:20:35Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![AndreiSamasiuk](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@AndreiSamasiuk](https://forum.opennebula.io/u/AndreiSamasiuk)\
**Post date:** [July 31, 2017, 1:20pm UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700/1 "2017-07-31T13:20:35Z")

</div>

We have OpenNebula 5.2.1 installed (which was upgraded to 5.2.1 from version 4.6.0).  
When new version 5.4 was released, we decided to upgrade our installation.  
But after full upgrade process ([http://docs.opennebula.org/5.4/intro\_release\_notes/upgrades/upgrade\_52.html](http://docs.opennebula.org/5.4/intro_release_notes/upgrades/upgrade_52.html), [http://docs.opennebula.org/5.4/deployment/opennebula\_installation/frontend\_installation.html#ignc](http://docs.opennebula.org/5.4/deployment/opennebula_installation/frontend_installation.html#ignc)) when we start one ($ one start) after ~30 second oned suddenly dies and we see such strings in our logs.

/var/log/syslog  
Jul 31 15:28:05 on kernel: [2009.104310] oned[24712]: segfault at 9 ip 000000000056cd36 sp 00007f8f7a7fb5e0 error 4 in oned[400000+2fb000]  
Jul 31 15:39:10 on kernel: [2674.875929] oned[603]: segfault at 7fbfcc0003e8 ip 00007fbfcc0003e8 sp 00007fbffdffa5d8 error 15

After we come back to version 5.2.1 (restore database, /etc/one backup, original OpenNebula 5.2.1 packages), we have working installation.

Debian Jessie 8.9 x64  
root@on:/var/log/one# dpkg --get-selections | grep nebula  
opennebula install  
opennebula-common install  
opennebula-flow install  
opennebula-sunstone install  
opennebula-tools install  
ruby-opennebula install

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ruben/32/19_2.png) [@ruben](https://forum.opennebula.io/u/ruben)\
**Post date:** [July 31, 2017, 2:58pm UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700/2 "2017-07-31T14:58:21Z")

</div>

About the fail, do you have the log files? also do you have access to the core files?

---

<div class="post-metadata">

**Author:** ![AndreiSamasiuk](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@AndreiSamasiuk](https://forum.opennebula.io/u/AndreiSamasiuk)\
**Post date:** [July 31, 2017, 3:13pm UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700/3 "2017-07-31T15:13:15Z")

</div>

Yes, I have root access on all nodes. Which logs are necessary for You?

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ruben/32/19_2.png) [@ruben](https://forum.opennebula.io/u/ruben)\
**Post date:** [July 31, 2017, 3:27pm UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700/4 "2017-07-31T15:27:34Z")

</div>

Just oned.log from the frontend. and maybe you have something coredumpctl for oned… This has to be the 5.4 oned.log that crashed,

THANK YOU!!

---

<div class="post-metadata">

**Author:** ![AndreiSamasiuk](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@AndreiSamasiuk](https://forum.opennebula.io/u/AndreiSamasiuk)\
**Post date:** [August 1, 2017, 2:04pm UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700/5 "2017-08-01T14:04:55Z")

</div>

## root@on:/var/log/one# cat /var/log/one/oned.log Tue Aug 1 17:00:07 2017 [Z0][ONE][I]: Starting OpenNebula 5.4.0

```
 OpenNebula Configuration File      

```

* * *

## AUTH\_MAD=AUTHN=ssh,x509,ldap,server\_cipher,server\_x509,EXECUTABLE=one\_auth\_mad AUTH\_MAD\_CONF=DRIVER\_MANAGED\_GROUPS=NO,MAX\_TOKEN\_TIME=-1,NAME=core,PASSWORD\_CHANGE=YES AUTH\_MAD\_CONF=DRIVER\_MANAGED\_GROUPS=NO,MAX\_TOKEN\_TIME=-1,NAME=public,PASSWORD\_CHANGE=NO AUTH\_MAD\_CONF=DRIVER\_MANAGED\_GROUPS=NO,MAX\_TOKEN\_TIME=-1,NAME=ssh,PASSWORD\_CHANGE=YES AUTH\_MAD\_CONF=DRIVER\_MANAGED\_GROUPS=NO,MAX\_TOKEN\_TIME=-1,NAME=x509,PASSWORD\_CHANGE=NO AUTH\_MAD\_CONF=DRIVER\_MANAGED\_GROUPS=YES,MAX\_TOKEN\_TIME=86400,NAME=ldap,PASSWORD\_CHANGE=YES AUTH\_MAD\_CONF=DRIVER\_MANAGED\_GROUPS=NO,MAX\_TOKEN\_TIME=-1,NAME=server\_cipher,PASSWORD\_CHANGE=NO AUTH\_MAD\_CONF=DRIVER\_MANAGED\_GROUPS=NO,MAX\_TOKEN\_TIME=-1,NAME=server\_x509,PASSWORD\_CHANGE=NO DATASTORE\_CAPACITY\_CHECK=yes DATASTORE\_LOCATION=/var/lib/one//datastores DATASTORE\_MAD=ARGUMENTS=-t 15 -d dummy,fs,lvm,ceph,dev,iscsi\_libvirt,vcenter -s shared,ssh,ceph,fs\_lvm,qcow2,vcenter,EXECUTABLE=one\_datastore DB=BACKEND=mysql,DB\_NAME=opennebula,PASSWD=kar123kar,PORT=0,SERVER=localhost,USER=oneadmin DEFAULT\_AUTH=ldap DEFAULT\_CDROM\_DEVICE\_PREFIX=hd DEFAULT\_COST=CPU\_COST=0,DISK\_COST=0,MEMORY\_COST=0 DEFAULT\_DEVICE\_PREFIX=hd DEFAULT\_IMAGE\_PERSISTENT= DEFAULT\_IMAGE\_PERSISTENT\_NEW= DEFAULT\_IMAGE\_TYPE=OS DEFAULT\_UMASK=177 DS\_MAD\_CONF=MARKETPLACE\_ACTIONS=export,NAME=ceph,PERSISTENT\_ONLY=NO,REQUIRED\_ATTRS=DISK\_TYPE,BRIDGE\_LIST DS\_MAD\_CONF=NAME=dev,PERSISTENT\_ONLY=YES,REQUIRED\_ATTRS=DISK\_TYPE DS\_MAD\_CONF=NAME=iscsi\_libvirt,PERSISTENT\_ONLY=YES,REQUIRED\_ATTRS=DISK\_TYPE,ISCSI\_HOST DS\_MAD\_CONF=NAME=dummy,PERSISTENT\_ONLY=NO,REQUIRED\_ATTRS= DS\_MAD\_CONF=MARKETPLACE\_ACTIONS=export,NAME=fs,PERSISTENT\_ONLY=NO,REQUIRED\_ATTRS= DS\_MAD\_CONF=NAME=lvm,PERSISTENT\_ONLY=NO,REQUIRED\_ATTRS=DISK\_TYPE,BRIDGE\_LIST DS\_MAD\_CONF=MARKETPLACE\_ACTIONS=export,NAME=vcenter,PERSISTENT\_ONLY=NO,REQUIRED\_ATTRS=VCENTER\_INSTANCE\_ID, VCENTER\_DS\_REF, VCENTER\_DC\_REF DS\_MAD\_CONF=NAME=shared,PERSISTENT\_ONLY=NO,REQUIRED\_ATTRS= DS\_MAD\_CONF=NAME=ssh,PERSISTENT\_ONLY=NO,REQUIRED\_ATTRS= DS\_MAD\_CONF=NAME=vmfs,PERSISTENT\_ONLY=NO,REQUIRED\_ATTRS=BRIDGE\_LIST ENABLE\_OTHER\_PERMISSIONS=YES FEDERATION=MASTER\_ONED=,MODE=STANDALONE,SERVER\_ID=-1,ZONE\_ID=0 HM\_MAD=EXECUTABLE=one\_hm HOST\_MONITORING\_EXPIRATION\_TIME=43200 HOST\_PER\_INTERVAL=15 IMAGE\_RESTRICTED\_ATTR=SOURCE IM\_MAD=ARGUMENTS=-p 4124 -f 5 -t 50 -i 20,EXECUTABLE=collectd,NAME=collectd IM\_MAD=ARGUMENTS=-r 3 -t 15 kvm,EXECUTABLE=one\_im\_ssh,NAME=kvm,SUNSTONE\_NAME=KVM IM\_MAD=ARGUMENTS=-c -t 15 -r 0 vcenter,EXECUTABLE=one\_im\_sh,NAME=vcenter,SUNSTONE\_NAME=VMWare vCenter INHERIT\_DATASTORE\_ATTR=CEPH\_HOST INHERIT\_DATASTORE\_ATTR=CEPH\_SECRET INHERIT\_DATASTORE\_ATTR=CEPH\_KEY INHERIT\_DATASTORE\_ATTR=CEPH\_USER INHERIT\_DATASTORE\_ATTR=CEPH\_CONF INHERIT\_DATASTORE\_ATTR=POOL\_NAME INHERIT\_DATASTORE\_ATTR=ISCSI\_USER INHERIT\_DATASTORE\_ATTR=ISCSI\_USAGE INHERIT\_DATASTORE\_ATTR=ISCSI\_HOST INHERIT\_DATASTORE\_ATTR=GLUSTER\_HOST INHERIT\_DATASTORE\_ATTR=GLUSTER\_VOLUME INHERIT\_DATASTORE\_ATTR=DISK\_TYPE INHERIT\_DATASTORE\_ATTR=ALLOW\_ORPHANS INHERIT\_DATASTORE\_ATTR=VCENTER\_ADAPTER\_TYPE INHERIT\_DATASTORE\_ATTR=VCENTER\_DISK\_TYPE INHERIT\_DATASTORE\_ATTR=VCENTER\_DS\_REF INHERIT\_DATASTORE\_ATTR=VCENTER\_DS\_IMAGE\_DIR INHERIT\_DATASTORE\_ATTR=VCENTER\_DS\_VOLATILE\_DIR INHERIT\_DATASTORE\_ATTR=VCENTER\_INSTANCE\_ID INHERIT\_IMAGE\_ATTR=ISCSI\_USER INHERIT\_IMAGE\_ATTR=ISCSI\_USAGE INHERIT\_IMAGE\_ATTR=ISCSI\_HOST INHERIT\_IMAGE\_ATTR=ISCSI\_IQN INHERIT\_IMAGE\_ATTR=DISK\_TYPE INHERIT\_IMAGE\_ATTR=VCENTER\_ADAPTER\_TYPE INHERIT\_IMAGE\_ATTR=VCENTER\_DISK\_TYPE INHERIT\_VNET\_ATTR=VLAN\_TAGGED\_ID INHERIT\_VNET\_ATTR=FILTER\_IP\_SPOOFING INHERIT\_VNET\_ATTR=FILTER\_MAC\_SPOOFING INHERIT\_VNET\_ATTR=MTU INHERIT\_VNET\_ATTR=INBOUND\_AVG\_BW INHERIT\_VNET\_ATTR=INBOUND\_PEAK\_BW INHERIT\_VNET\_ATTR=INBOUND\_PEAK\_KB INHERIT\_VNET\_ATTR=OUTBOUND\_AVG\_BW INHERIT\_VNET\_ATTR=OUTBOUND\_PEAK\_BW INHERIT\_VNET\_ATTR=OUTBOUND\_PEAK\_KB INHERIT\_VNET\_ATTR=CONF INHERIT\_VNET\_ATTR=BRIDGE\_CONF INHERIT\_VNET\_ATTR=IP\_LINK\_CONF INHERIT\_VNET\_ATTR=VCENTER\_NET\_REF INHERIT\_VNET\_ATTR=VCENTER\_SWITCH\_NAME INHERIT\_VNET\_ATTR=VCENTER\_SWITCH\_NPORTS INHERIT\_VNET\_ATTR=VCENTER\_PORTGROUP\_TYPE INHERIT\_VNET\_ATTR=VCENTER\_CCR\_REF INHERIT\_VNET\_ATTR=VCENTER\_INSTANCE\_ID IPAM\_MAD=ARGUMENTS=-t 1 -i dummy,EXECUTABLE=one\_ipam KEEPALIVE\_MAX\_CONN=30 KEEPALIVE\_TIMEOUT=15 LISTEN\_ADDRESS=0.0.0.0 LOG=DEBUG\_LEVEL=3,SYSTEM=file LOG\_CALL\_FORMAT=Req:%i UID:%u %m invoked %l MAC\_PREFIX=02:00 MANAGER\_TIMER=15 MARKET\_MAD=ARGUMENTS=-t 15 -m http,s3,one,EXECUTABLE=one\_market MARKET\_MAD\_CONF=APP\_ACTIONS=monitor,NAME=one,PUBLIC=yes,REQUIRED\_ATTRS=,SUNSTONE\_NAME=[OpenNebula.org](http://OpenNebula.org) Marketplace MARKET\_MAD\_CONF=APP\_ACTIONS=create, delete, monitor,NAME=http,REQUIRED\_ATTRS=BASE\_URL,PUBLIC\_DIR,SUNSTONE\_NAME=HTTP server MARKET\_MAD\_CONF=APP\_ACTIONS=create, delete, monitor,NAME=s3,REQUIRED\_ATTRS=ACCESS\_KEY\_ID,SECRET\_ACCESS\_KEY,REGION,BUCKET,SUNSTONE\_NAME=Amazon S3 MAX\_CONN=15 MAX\_CONN\_BACKLOG=15 MESSAGE\_SIZE=1073741824 MONITORING\_INTERVAL=60 MONITORING\_THREADS=50 NETWORK\_SIZE=254 PCI\_PASSTHROUGH\_BUS=0x01 PORT=2633 RAFT=BROADCAST\_TIMEOUT\_MS=500,ELECTION\_TIMEOUT\_MS=2500,LOG\_PURGE\_TIMEOUT=600,LOG\_RETENTION=500000,XMLRPC\_TIMEOUT\_MS=2000 RPC\_LOG=NO SCRIPTS\_REMOTE\_DIR=/var/tmp/one SESSION\_EXPIRATION\_TIME=900 TIMEOUT=15 TM\_MAD=ARGUMENTS=-t 15 -d dummy,lvm,shared,fs\_lvm,qcow2,ssh,ceph,dev,vcenter,iscsi\_libvirt,EXECUTABLE=one\_tm TM\_MAD\_CONF=CLONE\_TARGET=SYSTEM,DS\_MIGRATE=YES,LN\_TARGET=NONE,NAME=dummy,SHARED=YES TM\_MAD\_CONF=CLONE\_TARGET=SELF,LN\_TARGET=NONE,NAME=lvm,SHARED=YES TM\_MAD\_CONF=CLONE\_TARGET=SYSTEM,DS\_MIGRATE=YES,LN\_TARGET=NONE,NAME=shared,SHARED=YES TM\_MAD\_CONF=CLONE\_TARGET=SYSTEM,DRIVER=raw,LN\_TARGET=SYSTEM,NAME=fs\_lvm,SHARED=YES TM\_MAD\_CONF=CLONE\_TARGET=SYSTEM,DRIVER=qcow2,LN\_TARGET=NONE,NAME=qcow2,SHARED=YES TM\_MAD\_CONF=CLONE\_TARGET=SYSTEM,DS\_MIGRATE=YES,LN\_TARGET=SYSTEM,NAME=ssh,SHARED=NO TM\_MAD\_CONF=ALLOW\_ORPHANS=yes,CLONE\_TARGET=SELF,DRIVER=raw,DS\_MIGRATE=NO,LN\_TARGET=NONE,NAME=ceph,SHARED=YES TM\_MAD\_CONF=CLONE\_TARGET=SELF,DS\_MIGRATE=NO,LN\_TARGET=NONE,NAME=iscsi\_libvirt,SHARED=YES TM\_MAD\_CONF=CLONE\_TARGET=NONE,LN\_TARGET=NONE,NAME=dev,SHARED=YES TM\_MAD\_CONF=CLONE\_TARGET=SYSTEM,LN\_TARGET=NONE,NAME=vcenter,SHARED=YES TM\_MAD\_CONF=CLONE\_TARGET=SYSTEM,DRIVER=,DS\_MIGRATE=NO,LN\_TARGET=NONE,NAME=vmfs,SHARED=YES VLAN\_IDS=RESERVED=0, 1, 4095,START=2 VM\_ADMIN\_OPERATIONS=migrate, delete, recover, retry, deploy, resched VM\_INDIVIDUAL\_MONITORING=no VM\_MAD=ARGUMENTS=-t 15 -r 0 kvm,DEFAULT=vmm\_exec/vmm\_exec\_kvm.conf,EXECUTABLE=one\_vmm\_exec,IMPORTED\_VMS\_ACTIONS=terminate, terminate-hard, hold, release, suspend, resume, delete, reboot, reboot-hard, resched, unresched, disk-attach, disk-detach, nic-attach, nic-detach, snap-create, snap-delete,KEEP\_SNAPSHOTS=no,NAME=kvm,SUNSTONE\_NAME=KVM,TYPE=kvm VM\_MAD=ARGUMENTS=-p -t 15 -r 0 vcenter -s sh,DEFAULT=vmm\_exec/vmm\_exec\_vcenter.conf,EXECUTABLE=one\_vmm\_sh,IMPORTED\_VMS\_ACTIONS=terminate, terminate-hard, hold, release, suspend, resume, delete, reboot, reboot-hard, resched, unresched, poweroff, poweroff-hard, disk-attach, disk-detach, nic-attach, nic-detach, snap-create, snap-delete,KEEP\_SNAPSHOTS=yes,NAME=vcenter,SUNSTONE\_NAME=VMWare vCenter,TYPE=xml VM\_MANAGE\_OPERATIONS=undeploy, hold, release, stop, suspend, resume, reboot, poweroff, disk-attach, nic-attach, disk-snapshot, terminate, disk-resize, snapshot, updateconf, rename, resize, update, disk-saveas VM\_MONITORING\_EXPIRATION\_TIME=14400 VM\_PER\_INTERVAL=5 VM\_RESTRICTED\_ATTR=CONTEXT/FILES VM\_RESTRICTED\_ATTR=NIC/MAC VM\_RESTRICTED\_ATTR=NIC/VLAN\_ID VM\_RESTRICTED\_ATTR=NIC/BRIDGE VM\_RESTRICTED\_ATTR=NIC\_DEFAULT/MAC VM\_RESTRICTED\_ATTR=NIC\_DEFAULT/VLAN\_ID VM\_RESTRICTED\_ATTR=NIC\_DEFAULT/BRIDGE VM\_RESTRICTED\_ATTR=CPU\_COST VM\_RESTRICTED\_ATTR=MEMORY\_COST VM\_RESTRICTED\_ATTR=DISK\_COST VM\_RESTRICTED\_ATTR=PCI VM\_RESTRICTED\_ATTR=EMULATOR VM\_RESTRICTED\_ATTR=RAW VM\_RESTRICTED\_ATTR=USER\_PRIORITY VM\_SUBMIT\_ON\_HOLD=NO VM\_USE\_OPERATIONS= VNC\_PORTS=START=5900 VNET\_RESTRICTED\_ATTR=VN\_MAD VNET\_RESTRICTED\_ATTR=PHYDEV VNET\_RESTRICTED\_ATTR=VLAN\_ID VNET\_RESTRICTED\_ATTR=BRIDGE VNET\_RESTRICTED\_ATTR=CONF VNET\_RESTRICTED\_ATTR=BRIDGE\_CONF VNET\_RESTRICTED\_ATTR=IP\_LINK\_CONF VNET\_RESTRICTED\_ATTR=AR/VN\_MAD VNET\_RESTRICTED\_ATTR=AR/PHYDEV VNET\_RESTRICTED\_ATTR=AR/VLAN\_ID VNET\_RESTRICTED\_ATTR=AR/BRIDGE VXLAN\_IDS=START=2

Tue Aug 1 17:00:07 2017 [Z0][ONE][I]: Log level:3 [0=ERROR,1=WARNING,2=INFO,3=DEBUG]  
Tue Aug 1 17:00:07 2017 [Z0][ONE][I]: Support for xmlrpc-c \> 1.31: yes  
Tue Aug 1 17:00:07 2017 [Z0][ONE][I]: Checking database version.  
Tue Aug 1 17:00:07 2017 [Z0][ONE][I]: oned is using version 5.4.0 for local\_db\_versioning  
Tue Aug 1 17:00:07 2017 [Z0][ONE][I]: oned is using version 5.4.0 for db\_versioning  
Tue Aug 1 17:00:07 2017 [Z0][ACL][I]: Starting ACL Manager…  
Tue Aug 1 17:00:07 2017 [Z0][ACL][I]: ACL Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][HKM][I]: Starting Hook Manager…  
Tue Aug 1 17:00:07 2017 [Z0][HKM][I]: Loading Hook Manager driver.  
Tue Aug 1 17:00:07 2017 [Z0][HKM][I]: Hook Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][HKM][I]: Hook Manager loaded  
Tue Aug 1 17:00:07 2017 [Z0][ONE][I]: oned started in solo mode.  
Tue Aug 1 17:00:07 2017 [Z0][RCM][I]: Starting Raft Consensus Manager…  
Tue Aug 1 17:00:07 2017 [Z0][FRM][I]: Starting Federation Replica Manager…  
Tue Aug 1 17:00:07 2017 [Z0][RCM][I]: Raft Consensus Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][FRM][I]: Federation Replica Manger started.  
Tue Aug 1 17:00:07 2017 [Z0][VMM][I]: Starting Virtual Machine Manager…  
Tue Aug 1 17:00:07 2017 [Z0][LCM][I]: Starting Life-cycle Manager…  
Tue Aug 1 17:00:07 2017 [Z0][VMM][I]: Virtual Machine Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][LCM][I]: Life-cycle Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][InM][I]: Starting Information Manager…  
Tue Aug 1 17:00:07 2017 [Z0][InM][I]: Information Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][TrM][I]: Starting Transfer Manager…  
Tue Aug 1 17:00:07 2017 [Z0][DiM][I]: Starting Dispatch Manager…  
Tue Aug 1 17:00:07 2017 [Z0][TrM][I]: Transfer Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][DiM][I]: Dispatch Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][AuM][I]: Starting Auth Manager…  
Tue Aug 1 17:00:07 2017 [Z0][AuM][I]: Authorization Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][ImM][I]: Starting Image Manager…  
Tue Aug 1 17:00:07 2017 [Z0][ImM][I]: Image Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][ImM][I]: Starting Marketplace Manager…  
Tue Aug 1 17:00:07 2017 [Z0][MKP][I]: Marketplace Manager started.  
Tue Aug 1 17:00:07 2017 [Z0][IPM][I]: Starting IPAM Manager…  
Tue Aug 1 17:00:07 2017 [Z0][IPM][I]: IPAM Manager started.  
Tue Aug 1 17:00:09 2017 [Z0][VMM][I]: Loading Virtual Machine Manager drivers.  
Tue Aug 1 17:00:09 2017 [Z0][VMM][I]: Loading driver: kvm (KVM)  
Tue Aug 1 17:00:09 2017 [Z0][VMM][E]: Wrong action: snap-create  
Tue Aug 1 17:00:09 2017 [Z0][VMM][E]: Wrong action: snap-delete  
Tue Aug 1 17:00:10 2017 [Z0][VMM][I]: Driver kvm loaded.  
Tue Aug 1 17:00:10 2017 [Z0][VMM][I]: Loading driver: vcenter (XML)  
Tue Aug 1 17:00:10 2017 [Z0][VMM][E]: Wrong action: snap-create  
Tue Aug 1 17:00:10 2017 [Z0][VMM][E]: Wrong action: snap-delete  
Tue Aug 1 17:00:10 2017 [Z0][VMM][I]: Driver vcenter loaded.  
Tue Aug 1 17:00:10 2017 [Z0][InM][I]: Loading Information Manager drivers.  
Tue Aug 1 17:00:10 2017 [Z0][InM][I]: Loading driver: collectd  
Tue Aug 1 17:00:10 2017 [Z0][InM][I]: Driver collectd loaded  
Tue Aug 1 17:00:10 2017 [Z0][InM][I]: Loading driver: kvm  
Tue Aug 1 17:00:10 2017 [Z0][InM][I]: Driver kvm loaded  
Tue Aug 1 17:00:10 2017 [Z0][InM][I]: Loading driver: vcenter  
Tue Aug 1 17:00:10 2017 [Z0][InM][I]: Driver vcenter loaded  
Tue Aug 1 17:00:10 2017 [Z0][TM][I]: Loading Transfer Manager driver.  
Tue Aug 1 17:00:10 2017 [Z0][TM][I]: Transfer manager driver loaded  
Tue Aug 1 17:00:10 2017 [Z0][ImM][I]: Loading Image Manager driver.  
Tue Aug 1 17:00:10 2017 [Z0][ImM][I]: Image Manager loaded  
Tue Aug 1 17:00:10 2017 [Z0][MKP][I]: Loading Marketplace Manager driver.  
Tue Aug 1 17:00:11 2017 [Z0][MKP][I]: Marketplace Manager loaded  
Tue Aug 1 17:00:11 2017 [Z0][IPM][I]: Loading IPAM Manager driver.  
Tue Aug 1 17:00:11 2017 [Z0][IPM][I]: IPAM Manager loaded  
Tue Aug 1 17:00:11 2017 [Z0][AuM][I]: Loading Auth. Manager driver.  
Tue Aug 1 17:00:11 2017 [Z0][AuM][I]: Auth Manager loaded  
Tue Aug 1 17:00:11 2017 [Z0][ReM][I]: Starting Request Manager…  
Tue Aug 1 17:00:11 2017 [Z0][ReM][I]: Starting XML-RPC server, port 2633 …  
Tue Aug 1 17:00:11 2017 [Z0][ReM][I]: Request Manager started.  
Tue Aug 1 17:00:13 2017 [Z0][ReM][D]: Req:9072 UID:0 one.system.config invoked  
Tue Aug 1 17:00:13 2017 [Z0][ReM][D]: Req:9072 UID:0 one.system.config result SUCCESS, "\<AUTH\_MAD\>…"  
Tue Aug 1 17:00:15 2017 [Z0][InM][D]: Host node2 (1) successfully monitored.  
Tue Aug 1 17:00:15 2017 [Z0][VMM][D]: VM 248 successfully monitored: STATE=a CPU=10.19 MEMORY=4194304 NETRX=45240796328 NETTX=32485128 DISKRDBYTES=542150144 DISKWRBYTES=3386170368 DISKRDIOPS=8468 DISKWRIOPS=609088  
Tue Aug 1 17:00:15 2017 [Z0][VMM][D]: VM 1248 successfully monitored: STATE=a CPU=9.17 MEMORY=6291456 NETRX=7795943535 NETTX=1029834 DISKRDBYTES=957817954 DISKWRBYTES=5948096512 DISKRDIOPS=100167 DISKWRIOPS=430323  
Tue Aug 1 17:00:15 2017 [Z0][VMM][D]: VM 1353 successfully monitored: STATE=a CPU=8.15 MEMORY=2097152 NETRX=7846344197 NETTX=4047478 DISKRDBYTES=2839144960 DISKWRBYTES=3319002624 DISKRDIOPS=164502 DISKWRIOPS=134158  
Tue Aug 1 17:00:15 2017 [Z0][VMM][D]: VM 1453 successfully monitored: STATE=a CPU=43.8 MEMORY=33554432 NETRX=45455715771 NETTX=38952371 DISKRDBYTES=976482816 DISKWRBYTES=3739892736 DISKRDIOPS=37951 DISKWRIOPS=914107  
Tue Aug 1 17:00:15 2017 [Z0][VMM][D]: VM 1459 successfully monitored: STATE=a CPU=49.92 MEMORY=67108864 NETRX=46044311300 NETTX=271108016 DISKRDBYTES=777558528 DISKWRBYTES=49238375424 DISKRDIOPS=28276 DISKWRIOPS=2593937  
Tue Aug 1 17:00:15 2017 [Z0][VMM][D]: VM 1751 successfully monitored: STATE=a CPU=2.04 MEMORY=2104496 NETRX=173623651 NETTX=19512194 DISKRDBYTES=2757950976 DISKWRBYTES=2672941568 DISKRDIOPS=135229 DISKWRIOPS=257903  
Tue Aug 1 17:00:15 2017 [Z0][VMM][D]: VM 1752 successfully monitored: STATE=a CPU=1.02 MEMORY=2101432 NETRX=173620473 NETTX=19524356 DISKRDBYTES=2806717440 DISKWRBYTES=2890356224 DISKRDIOPS=136771 DISKWRIOPS=270707  
Tue Aug 1 17:00:20 2017 [Z0][InM][D]: Host node1 (0) successfully monitored.  
Tue Aug 1 17:00:20 2017 [Z0][VMM][D]: VM 1185 successfully monitored: STATE=a CPU=3.09 MEMORY=4227184 NETRX=81672464055 NETTX=3661161235 DISKRDBYTES=1821240588 DISKWRBYTES=70083159040 DISKRDIOPS=225327 DISKWRIOPS=3084419  
Tue Aug 1 17:00:20 2017 [Z0][VMM][D]: VM 1187 successfully monitored: STATE=a CPU=4.12 MEMORY=2097152 NETRX=45237429906 NETTX=15079048 DISKRDBYTES=105780800 DISKWRBYTES=7239254016 DISKRDIOPS=26278 DISKWRIOPS=1290505  
Tue Aug 1 17:00:20 2017 [Z0][VMM][D]: VM 1361 successfully monitored: STATE=a CPU=0.0 MEMORY=4145152 NETRX=80903141659 NETTX=1016816374 DISKRDBYTES=5552362386 DISKWRBYTES=106104012800 DISKRDIOPS=225858 DISKWRIOPS=7870164  
Tue Aug 1 17:00:20 2017 [Z0][VMM][D]: VM 1402 successfully monitored: STATE=a CPU=9.27 MEMORY=2127996 NETRX=81521328462 NETTX=738258751 DISKRDBYTES=9271912448 DISKWRBYTES=12966392832 DISKRDIOPS=979832 DISKWRIOPS=1979401  
Tue Aug 1 17:00:20 2017 [Z0][VMM][D]: VM 1561 successfully monitored: STATE=a CPU=2.06 MEMORY=1048576 NETRX=7798084557 NETTX=58715 DISKRDBYTES=420557430 DISKWRBYTES=460955648 DISKRDIOPS=43381 DISKWRIOPS=67329  
Tue Aug 1 17:00:20 2017 [Z0][VMM][D]: VM 1591 successfully monitored: STATE=a CPU=4.12 MEMORY=1048576 NETRX=68476506439 NETTX=2317294 DISKRDBYTES=52211712 DISKWRBYTES=4108345344 DISKRDIOPS=567 DISKWRIOPS=547707  
Tue Aug 1 17:00:20 2017 [Z0][VMM][D]: VM 1610 successfully monitored: STATE=a CPU=3.09 MEMORY=2097152 NETRX=7764589096 NETTX=1734768 DISKRDBYTES=104329430 DISKWRBYTES=76038144 DISKRDIOPS=43905 DISKWRIOPS=10252  
Tue Aug 1 17:00:20 2017 [Z0][VMM][D]: VM 1612 successfully monitored: STATE=a CPU=7.21 MEMORY=2097152 NETRX=68438746913 NETTX=23335186 DISKRDBYTES=1243287552 DISKWRBYTES=1438043136 DISKRDIOPS=39594 DISKWRIOPS=266120  
Tue Aug 1 17:00:21 2017 [Z0][VMM][D]: VM 1613 successfully monitored: STATE=a CPU=5.15 MEMORY=4194304 NETRX=68444902611 NETTX=10791668 DISKRDBYTES=23183360 DISKWRBYTES=10223796224 DISKRDIOPS=375 DISKWRIOPS=1105733  
Tue Aug 1 17:00:21 2017 [Z0][VMM][D]: VM 1630 successfully monitored: STATE=a CPU=102.96 MEMORY=4194304 NETRX=7763642798 NETTX=6600311 DISKRDBYTES=884729954 DISKWRBYTES=10779724800 DISKRDIOPS=92634 DISKWRIOPS=1007047  
Tue Aug 1 17:00:21 2017 [Z0][VMM][D]: VM 1637 successfully monitored: STATE=a CPU=3.09 MEMORY=4194304 NETRX=7761912624 NETTX=826598 DISKRDBYTES=149280964 DISKWRBYTES=106779648 DISKRDIOPS=50430 DISKWRIOPS=12905  
Tue Aug 1 17:00:21 2017 [Z0][VMM][D]: VM 1654 successfully monitored: STATE=a CPU=0.0 MEMORY=4194304 NETRX=3096942 NETTX=0 DISKRDBYTES=766464 DISKWRBYTES=0 DISKRDIOPS=1497 DISKWRIOPS=0  
Tue Aug 1 17:00:21 2017 [Z0][VMM][D]: VM 1658 successfully monitored: STATE=a CPU=24.71 MEMORY=2097152 NETRX=68461494435 NETTX=17213418 DISKRDBYTES=49152 DISKWRBYTES=15513067520 DISKRDIOPS=12 DISKWRIOPS=1819137  
Tue Aug 1 17:00:21 2017 [Z0][VMM][D]: VM 1745 successfully monitored: STATE=a CPU=58.69 MEMORY=4194304 NETRX=25604673 NETTX=600 DISKRDBYTES=0 DISKWRBYTES=723077120 DISKRDIOPS=0 DISKWRIOPS=184214  
Tue Aug 1 17:00:21 2017 [Z0][VMM][D]: VM 1748 successfully monitored: STATE=a CPU=0.0 MEMORY=1070184 NETRX=68452670703 NETTX=600 DISKRDBYTES=270336 DISKWRBYTES=15360266240 DISKRDIOPS=27 DISKWRIOPS=1666897  
Tue Aug 1 17:00:22 2017 [Z0][InM][D]: Monitoring datastore system (0)  
Tue Aug 1 17:00:22 2017 [Z0][MKP][D]: Monitoring marketplace OpenNebula Public (0)  
Tue Aug 1 17:00:22 2017 [Z0][InM][D]: Monitoring datastore default (1)  
Tue Aug 1 17:00:22 2017 [Z0][InM][D]: Monitoring datastore files (2)  
Tue Aug 1 17:00:22 2017 [Z0][ImM][D]: Datastore files (2) successfully monitored.  
Tue Aug 1 17:00:22 2017 [Z0][ImM][D]: Datastore default (1) successfully monitored.  
Tue Aug 1 17:00:22 2017 [Z0][MKP][D]: Marketplace OpenNebula Public (0) successfully monitored.  
Tue Aug 1 17:00:24 2017 [Z0][VMM][D]: VM 1172 successfully monitored: DISK\_SIZE=[ID=0,SIZE=3138] DISK\_SIZE=[ID=1,SIZE=1]  
Tue Aug 1 17:00:24 2017 [Z0][VMM][D]: VM 1185 successfully monitored: DISK\_SIZE=[ID=0,SIZE=116917] DISK\_SIZE=[ID=1,SIZE=1]  
Tue Aug 1 17:00:24 2017 [Z0][VMM][D]: VM 1187 successfully monitored: DISK\_SIZE=[ID=0,SIZE=3482] DISK\_SIZE=[ID=1,SIZE=1]  
Tue Aug 1 17:00:24 2017 [Z0][VMM][D]: VM 1191 successfully monitored: DISK\_SIZE=[ID=0,SIZE=25572] DISK\_SIZE=[ID=1,SIZE=564] DISK\_SIZE=[ID=2,SIZE=1]  
Tue Aug 1 17:00:24 2017 [Z0][VMM][D]: VM 1248 successfully monitored: DISK\_SIZE=[ID=0,SIZE=18993] DISK\_SIZE=[ID=1,SIZE=1]  
Tue Aug 1 17:00:24 2017 [Z0][VMM][D]: VM 1258 successfully monitored: DISK\_SIZE=[ID=0,SIZE=40827] DISK\_SIZE=[ID=1,SIZE=1]  
Tue Aug 1 17:00:24 2017 [Z0][VMM][D]: VM 1273 successfully monitored: DISK\_SIZE=[ID=0,SIZE=6816] DISK\_SIZE=[ID=1,SIZE=564] DISK\_SIZE=[ID=2,SIZE=1]

---

<div class="post-metadata">

**Author:** ![AndreiSamasiuk](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@AndreiSamasiuk](https://forum.opennebula.io/u/AndreiSamasiuk)\
**Post date:** [August 1, 2017, 2:06pm UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700/6 "2017-08-01T14:06:17Z")

</div>

After oned crash in SSH console.

oneadmin@on:~$ one stop  
oned and scheduler stopped  
oneadmin@on:~$ one start  
Stale .lock detected. Erasing it.  
oneadmin@on:~$ \*\*\* Error in `/usr/bin/oned’: double free or corruption (!prev): 0x00007fb4a801e1b0 \*\*\*

---

<div class="post-metadata">

**Author:** ![AndreiSamasiuk](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@AndreiSamasiuk](https://forum.opennebula.io/u/AndreiSamasiuk)\
**Post date:** [August 1, 2017, 2:09pm UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700/7 "2017-08-01T14:09:01Z")

</div>

root@on:/var/log# coredumpctl dump 6641  
PID: 6641 (oned)  
UID: 9869 (oneadmin)  
GID: 9869 (oneadmin)  
Signal: 6 (ABRT)  
Timestamp: Вт 2017-08-01 17:07:53 +03 (46s ago)  
Command Line: /usr/bin/oned -f  
Executable: /usr/bin/oned  
Control Group: /system.slice/ssh.service  
Unit: ssh.service  
Slice: system.slice  
Boot ID: aad63f21be9c4e37b5b59913d28db616  
Machine ID: f93825527b0b4f31ab13fb93decfd029  
Hostname: on  
Coredump: /var/lib/systemd/coredump/core.oned.9869.aad63f21be9c4e37b5b59913d28db616.6641.1501596473000000000000.xz  
Message: Process 6641 (oned) of user 9869 dumped core.

```
            Stack trace of thread 6682:
            #0 0x00007f9166d06067 __GI_raise (libc.so.6)
            #1 0x00007f9166d07448 __GI_abort (libc.so.6)
            #2 0x00007f9166d441b4 __libc_message (libc.so.6)
            #3 0x00007f9166d4998e malloc_printerr (libc.so.6)
            #4 0x00007f9166d4a696 _int_free (libc.so.6)
            #5 0x000000000041deaa _ZN12CallbackableD0Ev (oned)
            #6 0x000000000056cd3e _ZN14VirtualMachineD2Ev (oned)
            #7 0x000000000056cea4 _ZN14VirtualMachineD0Ev (oned)
            #8 0x0000000000533597 _ZN7PoolSQL3getEib (oned)
            #9 0x00000000004420af _ZN18VirtualMachinePool3getEib (oned)
            #10 0x0000000000445406 _ZN27VirtualMachineManagerDriver12process_pollEiRKSsb (oned)
            #11 0x000000000053138d monitor_action (oned)
            #12 0x00000000005326a5 _ZNK18ImageManagerDriver8protocolERKSs (oned)
            #13 0x0000000000517df2 _ZN10MadManager8listenerEv (oned)
            #14 0x000000000051732e mad_manager_listener (oned)
            #15 0x00007f91678a6064 start_thread (libpthread.so.0)
            #16 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6683:
            #0 0x00007f91678aa3f8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acae5 _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000441b9d _ZN13ActionManager4loopEl (oned)
            #3 0x0000000000529cc1 image_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6673:
            #0 0x00007f91678aa3f8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acae5 _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000441b9d _ZN13ActionManager4loopEl (oned)
            #3 0x00000000004368fa vmm_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6685:
            #0 0x00007f91678aa3f8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acae5 _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000441b9d _ZN13ActionManager4loopEl (oned)
            #3 0x00000000005d605b marketplace_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6884:
            #0 0x00007f91678aa04f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acb3d _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000458a99 _ZN13ActionManager4loopEv (oned)
            #3 0x000000000046a91d rm_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6684:
            #0 0x00007f9166db2893 __select (libc.so.6)
            #1 0x0000000000517903 _ZN10MadManager8listenerEv (oned)
            #2 0x000000000051732e mad_manager_listener (oned)
            #3 0x00007f91678a6064 start_thread (libpthread.so.0)
            #4 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6678:
            #0 0x00007f91678aa04f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acb3d _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000458a99 _ZN13ActionManager4loopEv (oned)
            #3 0x00000000004ddf0c tm_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6687:
            #0 0x00007f91678aa3f8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acae5 _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000441b9d _ZN13ActionManager4loopEl (oned)
            #3 0x00000000005d98f5 ipamm_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6641:
            #0 0x00007f91678ad5c9 do_sigwait (libpthread.so.0)
            #1 0x00007f91678ad653 __sigwait (libpthread.so.0)
            #2 0x000000000041b10f _ZN6Nebula5startEb (oned)
            #3 0x0000000000413810 oned_main (oned)
            #4 0x0000000000413b3e main (oned)
            #5 0x00007f9166cf2b45 __libc_start_main (libc.so.6)
            #6 0x000000000041353b _start (oned)
            
            Stack trace of thread 6670:
            #0 0x00007f91678aa04f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acb3d _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000458a99 _ZN13ActionManager4loopEv (oned)
            #3 0x00000000005e4669 raft_manager_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6676:
            #0 0x00007f91678aa3f8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acae5 _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000441b9d _ZN13ActionManager4loopEl (oned)
            #3 0x000000000046691f im_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6672:
            #0 0x00007f9166db2893 __select (libc.so.6)
            #1 0x0000000000517903 _ZN10MadManager8listenerEv (oned)
            #2 0x000000000051732e mad_manager_listener (oned)
            #3 0x00007f91678a6064 start_thread (libpthread.so.0)
            #4 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6677:
            #0 0x00007f9166db2893 __select (libc.so.6)
            #1 0x0000000000517903 _ZN10MadManager8listenerEv (oned)
            #2 0x000000000051732e mad_manager_listener (oned)
            #3 0x00007f91678a6064 start_thread (libpthread.so.0)
            #4 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6680:
            #0 0x00007f9166db2893 __select (libc.so.6)
            #1 0x0000000000517903 _ZN10MadManager8listenerEv (oned)
            #2 0x000000000051732e mad_manager_listener (oned)
            #3 0x00007f91678a6064 start_thread (libpthread.so.0)
            #4 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6671:
            #0 0x00007f91678aa04f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acb3d _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000458a99 _ZN13ActionManager4loopEv (oned)
            #3 0x00000000005eedab frm_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6652:
            #0 0x00007f91678aa04f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acb3d _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000458a99 _ZN13ActionManager4loopEv (oned)
            #3 0x00000000005ab7da hm_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6651:
            #0 0x00007f9166db2893 __select (libc.so.6)
            #1 0x0000000000517903 _ZN10MadManager8listenerEv (oned)
            #2 0x000000000051732e mad_manager_listener (oned)
            #3 0x00007f91678a6064 start_thread (libpthread.so.0)
            #4 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6885:
            #0 0x00007f9166db0aed poll (libc.so.6)
            #1 0x00000000006055ae waitForConnection (oned)
            #2 0x00000000005fc904 ChanSwitchAccept (oned)
            #3 0x0000000000602da9 acceptAndProcessNextConnection (oned)
            #4 0x00000000005f5e3c setupSignalsAndRunAbyss (oned)
            #5 0x00000000005f5e76 _ZN8xmlrpc_c16serverAbyss_impl3runEv (oned)
            #6 0x00000000005f6209 _ZN8xmlrpc_c11serverAbyss3runEv (oned)
            #7 0x000000000046aabf rm_xml_server_loop (oned)
            #8 0x00007f91678a6064 start_thread (libpthread.so.0)
            #9 0x00007f9166db962d __clone (libc.so.6)
            
            Stack trace of thread 6679:
            #0 0x00007f91678aa04f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acb3d _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000458a99 _ZN13ActionManager4loopEv (oned)
            #3 0x00000000004d4e92 dm_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)

            Stack trace of thread 6675:
            #0 0x00007f9166db2893 __select (libc.so.6)
            #1 0x0000000000517903 _ZN10MadManager8listenerEv (oned)
            #2 0x000000000051732e mad_manager_listener (oned)
            #3 0x00007f91678a6064 start_thread (libpthread.so.0)
            #4 0x00007f9166db962d __clone (libc.so.6)

            Stack trace of thread 6681:
            #0 0x00007f91678aa3f8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acae5 _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000441b9d _ZN13ActionManager4loopEl (oned)
            #3 0x0000000000508de4 authm_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)

            Stack trace of thread 6674:
            #0 0x00007f91678aa04f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
            #1 0x00000000005acb3d _ZN13ActionManager4loopER8timespecRK13ActionRequest (oned)
            #2 0x0000000000458a99 _ZN13ActionManager4loopEv (oned)
            #3 0x000000000045822a lcm_action_loop (oned)
            #4 0x00007f91678a6064 start_thread (libpthread.so.0)
            #5 0x00007f9166db962d __clone (libc.so.6)

            Stack trace of thread 6686:
            #0 0x00007f9166db2893 __select (libc.so.6)
            #1 0x0000000000517903 _ZN10MadManager8listenerEv (oned)
            #2 0x000000000051732e mad_manager_listener (oned)
            #3 0x00007f91678a6064 start_thread (libpthread.so.0)
            #4 0x00007f9166db962d __clone (libc.so.6)

```

Refusing to dump core to tty.

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ruben/32/19_2.png) [@ruben](https://forum.opennebula.io/u/ruben)\
**Post date:** [August 1, 2017, 5:09pm UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700/8 "2017-08-01T17:09:58Z")

</div>

It seems that during the migration one of the VMs in node1 was not upgraded correctly, probably the one after 1273. Could you check the list of VMs running in node1 and send me the output of the following command (probably it is better to send me the output in a private message).

For VMs **not** in 1273, 1258, 1248, 1191, 1187, 1185 and 1172, execute the following SQL command ‘SELECT \* from history\_pool where vid = \<ID\_OF\_THE\_VM\>’ I’ll need this from the upgraded DB.

---

<div class="post-metadata">

**Author:** ![AndreiSamasiuk](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@AndreiSamasiuk](https://forum.opennebula.io/u/AndreiSamasiuk)\
**Post date:** [August 2, 2017, 7:26am UTC](https://forum.opennebula.io/t/opennebula-suddenly-dies-after-upgrade-to-5-4/4700/9 "2017-08-02T07:26:31Z")

</div>

Strange behavior … Why wrong action?  
Wed Aug 2 10:23:29 2017 [Z0][VMM][I]: Loading Virtual Machine Manager drivers.  
Wed Aug 2 10:23:29 2017 [Z0][VMM][I]: Loading driver: kvm (KVM)  
Wed Aug 2 10:23:29 2017 [Z0][VMM][E]: Wrong action: snap-create  
Wed Aug 2 10:23:29 2017 [Z0][VMM][E]: Wrong action: snap-delete