vCenter monitoring stopped working after upgrade to 4.14.2

Community Support
1

Hi there,
I was running ONE 4.12.1 and had a vCenter 5.5 added for monitoring.
I just updated to ONE 4.14.2 and the vCenter instance created monitoring errors:

[Z0][InM][I]: Command execution fail: /var/lib/one/remotes/im/run_probes vcenter /var/lib/one//datastores 4124 20 21 Patna
[Z0][InM][I]: /usr/lib/one/ruby/vendors/rbvmomi/lib/rbvmomi/basic_types.rb:190:in `_get_property': NoPermission:  (RbVmomi::Fault)
[Z0][InM][I]: from /usr/lib/one/ruby/vendors/rbvmomi/lib/rbvmomi/basic_types.rb:35:in `block (2 levels) in init'
[Z0][InM][I]: from /usr/lib/one/ruby/vcenter_driver.rb:779:in `monitor_customizations'
[Z0][InM][I]: from ./vcenter.rb:49:in `<main>'
[Z0][InM][E]: Error executing vcenter.rb
[Z0][InM][I]: ExitCode: 1

Any idea what could cause this? Does my vCenter user lack certain permissions which are required after the update?

Cheers,
Christian

2

We were missing a new permission in the documentation indeed, thanks for the feedback, should be fixed now.

Try adding the following:

VirtualMachine.Provisioning.ReadCustSpecs

1 Like
3

Thanks! Will try that on monday.

4

Well sadly VirtualMachine.Provisioning.ReadCustSpecs is set for the corresponding vSphere user. Any other ideas?

5

Could you share the list of permissions of your user to try to reproduce the error?

6

Sadly we can’t export a list, but a screenshot in german language:

The last two options are “Read / Write CustomSpecs”.

So they are set for the user, but still vcenter.rb is throwing out the above error.

7

If roles are defined at the ESX level, the following will output a list of permissions associated to every role (run as root in the ESX shell):

vim-cmd vimsvc/auth/roles

8

I checked back with our vSphere admin. He says the user is not defined at the ESX level but at the cluster level instead. The user has all permissions, which are settable via the WebUI on the cluster level. He further states, that these settings “apply to the cluster and all subsequent items.” We currently do not have the ESX console activated.

So do we need to activate the console to set the needed permissions, or is there another way?

9

Hi,

What do you mean exactly by “all permissions”? Could you check with a full admin user to see if the error persists?

10

Well I’ve seen the UI, our admin showed me. There is the permission tab and all permissions were marked.
That’s the fullest admin we can setup via the vSphere UI for a cluster according to the administrator…

11

Just to clarify: We did exactly as stated in the latest stable documentation for OpenNebula 4.14.2:
"Define a vCenter user for OpenNebula. This vCenter user (let’s call her oneadmin) needs to have access to the ESX clusters that OpenNebula will manage. In order to avoid problems, the hassle free approach is to declare this oneadmin user as Administrator"
from: http://docs.opennebula.org/4.14/administration/virtualization/vcenterg.html

I double checked with the vCenter Administrator and showed him the documentation. He agrees with me, that this is precisely what we have.

We’re running on vCenter & ESX 5.5.0

Cheers,
Christian

12

Then is should not be a permissions problem.

What is the output of the following (as oneadmin in the frontend):

ruby -dw /var/lib/one/remotes/im/run_probes vcenter /var/lib/one//datastores 4124 20 21 Patna

13

Well when executing the command I get this:

> [root@molokai ~]# ruby -dw /var/lib/one/remotes/im/run_probes vcenter /var/lib/one//datastores 4124 20 21 Patna
> Exception `LoadError' at /usr/share/rubygems/rubygems.rb:1096 - cannot load such file -- rubygems/defaults/ruby
> Exception `LoadError' at /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55 - cannot load such file -- abrt
> Exception `LoadError' at /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:141 - cannot load such file -- abrt
> Exception `LoadError' at ruby - no Ruby script found in input
> ruby: no Ruby script found in input (LoadError)
14

That is a weird error indeed. Let’s see if we can find out what is going on:

  1. what OS are you in?

  2. what version of ruby (ruby -v)? (are you using rbenv?)

  3. did you run install_gems?

  4. does this file exists: “/var/lib/one/remotes/im/run_probes” ?

15
  1. CentOS 7
  2. ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
  3. yes
  4. yes exists
16

Hi,

Ok, let’s check the vcenter probe directly, could you please let us know the output of:

$ /var/lib/one/remotes/im/vcenter.d/vcenter.rb x x x x <HOST_ID>

where HOST_ID is the ID in OpenNebula of host Patna

17

Here you go:

[root@molokai vcenter.d]# ruby vcenter.rb x x x x 21
/usr/lib/one/ruby/vendors/rbvmomi/lib/rbvmomi/basic_types.rb:190:in _get_property': NoPermission: (RbVmomi::Fault) from /usr/lib/one/ruby/vendors/rbvmomi/lib/rbvmomi/basic_types.rb:35:in block (2 levels) in init’
from /usr/lib/one/ruby/vcenter_driver.rb:779:in monitor_customizations' from vcenter.rb:49:in ’

18

The only explanation I can come with is that this vCenter user somehow does not have permissions to access Customization profiles.

You can confirm this by editing out line 49 (with a #) in /var/lib/one/remotes/im/vcenter.d/vcenter.rb and repeat the test.

19

You were right. However we had to set the VirtualMachine.Provisioning.ReadCustSpecs permission on the highest possible hierarchy item, which in our case was “vCenter Server”. Our Hierarchy looks like this:

- vCenter Server
-- vCenter Datacenter
--- Patna Cluster
---- ESX Host 1
---- ESX Host 2

We had to set admin permissions on Patna Cluster and the VirtualMachine.Provisioning.ReadCustSpecs persmission on vCenter Server.
Now it works.

20

I see. Thanks a lot for your feedback!