Opennebula Daemon Crashing

Community Support
1

2 sets of crashes happen so far.
I have the ugly timing crash of the daemon crashing due to the slow start of the monitord drivers. (Even with all mons but QEMU/KVM Disabled) Leading to segfaults.

Other crash results in exit-code 1 if it ever get’s past the monitord drivers loading but no reason is given even with oned log level set to 5 DDDebug.

Versions of the related components and OS (frontend, hypervisors, VMs):
Dell R720 etc
Ubuntu 22.04 with OpenNebula 6.6.0

Steps to reproduce:
Will provide anything at this point for evidence.

Current results:

Expected results:

2

Here is a GDB backtrace

#0  0x00005622b5998e0f in PoolSQL::list (table=0x5622b5bbe073 "datastore_pool", oids=std::vector of length 0, capacity 0, this=0x5622b6876290) at include/PoolSQL.h:171
#1  DatastorePool::list (oids=std::vector of length 0, capacity 0, this=0x5622b6876290) at include/DatastorePool.h:166
#2  ImageManager::timer_action (this=0x5622b6999a20) at src/image/ImageManager.cc:157
#3  ImageManager::ImageManager(long, long, ImagePool*, DatastorePool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)::{lambda()#1}::operator()() const (__closure=<optimized out>) at include/ImageManager.h:42
#4  std::__invoke_impl<void, ImageManager::ImageManager(long, long, ImagePool*, DatastorePool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)::{lambda()#1}&>(std::__invoke_other, ImageManager::ImageManager(long, long, ImagePool*, DatastorePool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)::{lambda()#1}&) (__f=...) at /usr/include/c++/11/bits/invoke.h:61
#5  std::__invoke_r<void, ImageManager::ImageManager(long, long, ImagePool*, DatastorePool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)::{lambda()#1}&>(ImageManager::ImageManager(long, long, ImagePool*, DatastorePool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)::{lambda()#1}&) (__fn=...) at /usr/include/c++/11/bits/invoke.h:154
#6  std::_Function_handler<void (), ImageManager::ImageManager(long, long, ImagePool*, DatastorePool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/11/bits/std_function.h:290
#7  0x00005622b598ddf8 in std::function<void ()>::operator()() const (this=0x5622b699a4b0) at /usr/include/c++/11/bits/std_function.h:590
#8  Timer::start(double, std::function<void ()>)::{lambda()#1}::operator()() const (__closure=<optimized out>) at include/Listener.h:94
#9  std::__invoke_impl<void, Timer::start(double, std::function<void ()>)::{lambda()#1}>(std::__invoke_other, Timer::start(double, std::function<void ()>)::{lambda()#1}&&) (__f=...) at /usr/include/c++/11/bits/invoke.h:61
#10 std::__invoke<Timer::start(double, std::function<void ()>)::{lambda()#1}>(Timer::start(double, std::function<void ()>)::{lambda()#1}&&) (__fn=...) at /usr/include/c++/11/bits/invoke.h:96
#11 std::thread::_Invoker<std::tuple<Timer::start(double, std::function<void ()>)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:253
#12 std::thread::_Invoker<std::tuple<Timer::start(double, std::function<void ()>)::{lambda()#1}> >::operator()() (this=<optimized out>) at /usr/include/c++/11/bits/std_thread.h:260
#13 std::thread::_State_impl<std::thread::_Invoker<std::tuple<Timer::start(double, std::function<void ()>)::{lambda()#1}> > >::_M_run() (this=0x5622b699a4a0) at /usr/include/c++/11/bits/std_thread.h:211
#14 0x00007f6b04d8a2b3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f6b04a13b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#16 0x00007f6b04aa5a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
3

Hi @Haywoodspartan,
at first look the crash at PoolSQL::list doesn’t make sense.
Can you please upload /var/log/one/monitord.log for the first crash and /var/log/one/oned.log for the second crash?