Vcenter script datastore monitoring spawning multiple executions

Community Support
1

Hi all,

Thank you for looking at my issue. At the moment, I’m trying to see what the issue is with my setup. I have an Opennebula frontend running 4.14.2. I’m seeing something weird with what I think is the datastore monitoring. Below is what I see when I run ps:

[root@opennebula lib]# ps -ef | grep CLUSTER01 | grep ruby
oneadmin   725   688 11 10:16 ?        00:01:15 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  1272  1242 11 09:50 ?        00:04:17 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  2145  2122 11 09:26 ?        00:07:05 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  2658  2635 11 10:22 ?        00:00:37 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  3452  3437 11 09:16 ?        00:08:17 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  3541  3511 11 08:59 ?        00:10:18 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  3636  3609 11 09:44 ?        00:05:05 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  4090  4050 11 09:45 ?        00:04:52 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  5124  5104 11 09:08 ?        00:09:18 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  6643  6606 11 08:54 ?        00:10:55 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  7124  7096 11 09:28 ?        00:06:51 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin  8741  8725 11 10:19 ?        00:01:00 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 11227 11202 11 09:25 ?        00:07:15 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 11977 11950 11 09:39 ?        00:05:38 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 14547 14520 11 09:34 ?        00:06:13 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 20937 20915 11 09:55 ?        00:03:46 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 21035 21017 11 09:12 ?        00:08:47 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 21343 21299 11 10:10 ?        00:01:58 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 22081 22050 11 09:02 ?        00:09:59 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 22730 22702 11 10:02 ?        00:02:57 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 23709 23678 11 09:21 ?        00:07:45 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 24135 24107 11 10:05 ?        00:02:32 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 24220 24185 11 10:27 ?        00:00:04 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 26969 26950 11 09:59 ?        00:03:20 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 30421 30392 11 10:09 ?        00:02:06 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01
oneadmin 31895 31876 11 10:01 ?        00:03:06 ruby ./vcenter.rb vcenter /var/lib/one//datastores 4124 20 0 CLUSTER01

I don’t understand why this is spawning so many instances. The processes take a lot of CPU resources, leaving the frontend completely drained. Is there something in particular I should be looking for? A misconfiguration of some sort?

Thanks in advance,
Jesus

2

Hi!
that’s certainly not be the usual behavior, and you’re right, those are processes related to datastore monitoring. It should launch one script and finish, and then after a few seconds later launch another one and finish…

Is there any error message in your /var/log/one/oned.log that can helps us know what’s happening?
e.g grep -e InM -e ImM /var/log/one/oned.log

And what’s the vCenter Information Driver Manager Configuration section in your /etc/one/oned.conf file?

Cheers!

3

Hi mcabrerizo,

Thank you for looking at this. I have ran your grep statement and the following section repeats over and over

Output:

Thank you for pointing out the oned.conf file as it seems that I had a misconfiguration based on the default configuration. It seems I had threads for the script set at 100 instead of 15 which is the default number. I have modifed this and restarted the services. Even with this though, the system is spawning multiple executions.

Thanks again for looking at this issue.
jamdev12

4

I will say that one of the things I do not see from the logs in this instance is this command:

/var/lib/one/remotes/datastore/vmfs/monitor

This command I see running on my staging box and I can’t seem to put my finger on the difference between these 2 systems in terms of the configuration.