Frequent Nebula Service Hang Issue During VM Lifecycle Stage Changes

Product Support Operations
1

Dear Nebula Team,

We have observed that the Nebula service frequently gets stuck during VM lifecycle stage transitions.

For example, when more than 20+ actions occur simultaneously such as power-off to running, prolog to boot, or boot to running, the process often hangs. During this time, the logs stop updating in real time, as seen when running:

tail -f /var/log/one/<vm_id>.log

No further log entries are generated until we restart the Nebula service. Once restarted, the logs resume and the VM states begin changing normally again.

This suggests that the Nebula service becomes unresponsive during high-concurrency operations or multiple simultaneous state transitions. Could you please help us understand the root cause of this behavior?

Is this a known limitation or expected behavior of Nebula, or can any configuration parameters in oned.conf be adjusted to prevent such frequent hangs?

currently I need to restart the Nebula service often to recover from these situations and would appreciate your guidance on achieving a permanent or configuration-based fix.

Thanks & Regards,

Senthil Kumar M

2

Hello,

  • Which version of OpenNebula are you running? If it is under 7.0 can you check the scheduler log on /var/log/one/sched.log?
  • Is the frontend that you are running OpenNebula in also a host (thus, does it run any VM)?

I have not experienced OpenNebula frontend problems for that amount of operations except on environments with very tight resources

Thank you

3

Hello,

Thank you for your response.

  1. I am currently using OpenNebula version 6.4.5.

  2. The front-end server is not acting as a hypervisor; it does not run any virtual machines.

let me know why its happening for me? any suggestions?

4

Hello TEAM,

Any Suggestions?

5

Dear Team,

Your suggestion will help me lot in this issue.

6

Hello again,

Which database are you using? If it is mysql or mariadb can you check if there are slow queries on /var/log/one/oned.log?

In that case, you can execute the following in the OpenNebula database

ALTER TABLE vm_pool DROP INDEX ftidx;
ALTER TABLE vm_pool ADD FULLTEXT INDEX ftidx (search_token);

Cheers

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.