Sunstone - Time out errors

Community Support
1

Hi Everyone,

We’re currently experiencing a weird issue on our OpenNebula management node.

The node is setup with the following configuration:

- CentOS 6.6
- Two interfaces in an balance-alb bond.
- MySQL backend
- /var/lib/one mounted on glusterfs and shared with all nodes.
- OpenNebula 4.12
- We're behind a proxy, so all proxy variables are set in the environment for all users.

When working on the commandline everything works as a charm, you can list all nodes/datastores, view the details, etc. All working smooth and fast!

The problem we face however is working inside the sunstone interface. It seems that listing resources (nodes and datastores) works normally but when we click a datastore/node/whatever it takes ages (minutes) before it shows the contents.

The sunstone.error logs only give the following info (in debug mode):

Errno::ETIMEDOUT - Connection timed out - connect(2):
 /usr/lib/ruby/1.8/net/http.rb:560:in `initialize'
 /usr/lib/ruby/1.8/net/http.rb:560:in `open'
 /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
 /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
 /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
 /usr/lib/ruby/1.8/net/http.rb:542:in `start'
 /usr/lib/ruby/1.8/net/http.rb:1035:in `request'
 /usr/lib/ruby/1.8/net/http.rb:772:in `get'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:80:in `perform_request'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:40:in `call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:32:in `call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/retry.rb:20:in `call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/encode_json.rb:21:in `call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/multipart.rb:14:in `call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/upload.rb:16:in `call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/etag_cache.rb:31:in `call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/authorization.rb:38:in `call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/parse_iso_dates.rb:11:in `call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/logger.rb:20:in `call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/callback.rb:14:in `call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/raise_error.rb:9:in `call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:140:in `get'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:104:in `find!'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:119:in `find'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in `send'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in `find'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/client.rb:56:in `current_user'
 /usr/lib/one/sunstone/routes/support.rb:66:in `zendesk_client'
 /usr/lib/one/sunstone/routes/support.rb:121:in `GET /support/request'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `call'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `route'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `instance_eval'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `route_eval'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `route!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `route!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `route!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in `dispatch!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `instance_eval'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/deflater.rb:13:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:63:in `context'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:58:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/showexceptions.rb:24:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in `call'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in `synchronize'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in `pre_process'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `catch'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `pre_process'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in `process'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in `receive_data'
 /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
 /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in `start'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in `run'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:946:in `run!'
 /usr/lib/one/sunstone/sunstone-server.rb:627

When I click a datastore in the interface, nothing will happen in the logs (only access logs give some info). Then, when this error appears, only then will we get information about the datastore.

So everything works eventually, but there’s a massive timeout causing this problem.

I’ve also noticed that I’m getting these connection time out errors without using the interface, so it might be something in the background doing some action?! If so, this action seems to block all other activities untill its finally done.

At first I thought this was IPv6 related, so we’ve disabled that, then I doubted the bonding interface, but that wasnt it either. Then DNS, but even that didnt budge. We even did traces on the OS but didn’t find anything suspicious.

Before we ran version 4.10, so after upgrading we hoped it would solve something, but that also didn’t happen.

So right now we’re kinda stuck… Does anyone have an idea on how to troubleshoot and solve this issue? What is causing this connection time out?

–
Bart G.

2

For the logs the problem seems to be related with the customer support plugin. It can be caused because the machine is behind a proxy and can not connect the server. You can try disabling that plugin. The instructions are here:

http://docs.opennebula.org/4.12/administration/sunstone_gui/sunstone.html#commercial-support-integration

3

Disabling support seems to have done the trick.

Is there any way on solving this by making sure that sunstone honors the proxy environment? (sunstone also cannot connect to the marketplace, seems the same issue)

4

Hi everyone,

maybe we have the same problem here. Actually we have two different OpenNebulas running, one of them in the lab and the other on is productive. Since we habe upgraded the productive to 4.10 we have problem accessing the hosts via sunstone. I just disabled the Support plugin in the admin view but it still get stucked loading the hosts tab.

In Sunstone logfile I just can find this:

Mon Mar 23 15:08:08 2015 [I]: xxx.xxx.xxx.xxx - - [23/Mar/2015 15:08:08] "GET /host/2?csrftoken=a5ac9be9cbf79d595112fa90a960334b " 200 - 0.1606
Mon Mar 23 15:10:13 2015 [I]: xxx.xxx.xxx.xxx - - [23/Mar/2015 15:10:13] "GET /support/request?timeout=false&csrftoken=50ec0166b2fb35aa5211132c75f00388 " 404 - 0.0024

Thanks in advance for your help.

Regards

5

One other thing to check is database consistency [1]

[1] http://docs.opennebula.org/4.12/administration/references/onedb.html

6

Hi! Did you find the solution? I have the same issue with many browsers (chrome, firefox, IE, Dolphin, Safari)
I think is not a browser issue.
I reinstalled opennebula and sunstone and still same.
Pls help me