Sunstone crashing on 5.12 while uploading ISO

Community Support GUI - Sunstone
1

Please, describe the problem here and provide additional information below (if applicable) …

While uploading an ISO file to a fresh install Opennebula 5.12 On Ubuntu 18.04, we see that the upload gets stuck and sunstone crashes. If we reload the webpage we are prompted to re-enter the username and password.

This issue is reproducible and happens on all the hosts. We are not using any proxy, just the Host LAN IP on port 9869

We have 6.8 GB available of space in /var/tmp while the ISO is 4.6 GB.

Perhaps there is a double copy that requires 2X free space for iso on the /var/tmp directory?

Sep 11 16:52:31 PS-Node3 ruby[1184015]: == Sinatra has ended his set (crowd applauds)
Sep 11 16:52:32 PS-Node3 ruby[1184015]: /usr/lib/ruby/2.5.0/delegate.rb:349:in write': closed stream (IOError) Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /usr/lib/ruby/2.5.0/delegate.rb:349:in <<’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /usr/lib/ruby/2.5.0/delegate.rb:349:in block in delegating_block' Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /var/lib/gems/2.5.0/gems/thin-1.7.2/lib/thin/request.rb:79:in parse’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /var/lib/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:39:in receive_data' Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /var/lib/gems/2.5.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run_machine’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /var/lib/gems/2.5.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run' Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /var/lib/gems/2.5.0/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in start’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /var/lib/gems/2.5.0/gems/thin-1.7.2/lib/thin/server.rb:162:in start' Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /var/lib/gems/2.5.0/gems/rack-2.2.2/lib/rack/handler/thin.rb:22:in run’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /var/lib/gems/2.5.0/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1526:in start_server' Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /var/lib/gems/2.5.0/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1460:in run!’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: from /usr/lib/one/sunstone/sunstone-server.rb:1047:in <main>' Sep 11 16:52:32 PS-Node3 ruby[1184015]: Invalid request: Invalid HTTP format, parsing fails. Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/thin-1.7.2/lib/thin/request.rb:84:in execute’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/thin-1.7.2/lib/thin/request.rb:84:in parse' Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:39:in receive_data’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run_machine' Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/eventmachine-1.2.7/lib/eventmachine.rb:195:in run’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/thin-1.7.2/lib/thin/backends/base.rb:73:in start' Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/thin-1.7.2/lib/thin/server.rb:162:in start’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/rack-2.2.2/lib/rack/handler/thin.rb:22:in run' Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1526:in start_server’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: /var/lib/gems/2.5.0/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1460:in run!' Sep 11 16:52:32 PS-Node3 ruby[1184015]: /usr/lib/one/sunstone/sunstone-server.rb:1047:in ’
Sep 11 16:52:32 PS-Node3 ruby[1184015]: Stopping …
Sep 11 16:52:32 PS-Node3 systemd[1]: opennebula-sunstone.service: Main process exited, code=exited, status=1/FAILURE
Sep 11 16:52:32 PS-Node3 systemd[1]: opennebula-sunstone.service: Failed with result ‘exit-code’.
Sep 11 16:52:37 PS-Node3 systemd[1]: opennebula-sunstone.service: Service hold-off time over, scheduling restart.
Sep 11 16:52:37 PS-Node3 systemd[1]: opennebula-sunstone.service: Scheduled restart job, restart counter is at 2.
Sep 11 16:52:37 PS-Node3 systemd[1]: Stopped OpenNebula Web UI Server.
Sep 11 16:52:37 PS-Node3 systemd[1]: Starting OpenNebula Web UI Server…
Sep 11 16:52:37 PS-Node3 systemd[1]: Started OpenNebula Web UI Server.

Versions of the related components and OS (frontend, hypervisors, VMs):
Opennebula 5.12

Steps to reproduce:
Upload a new ISO, we are using a 4.6GB Windows ISO but it happens on any ISO

Current results:
Upload gets stuck and Sunstone crashes

Expected results:
Upload should complete without error and Sunstone shouldn’t crash

2

Any assistance on this? the exact error in Ruby is

closed stream (IOError)

3

Using ceph datastore, I see that an uploaded iso is copied and converted before being uploaded to ceph. You will at least need 2X storage space where X is the size of your uploaded file.

4

We moved the :tmpdir: directory into a dedicated 200GB drive and restarted the host for changes to take effect.

Still Sunstone is crashing with Closed IO stream.

More info:
After the crash , we get 401 response in sunstone.log file, probably the browser still trying to upload the file.

We see that the tmpdir mountpoint is getting filled with the data and when it reaches the expected amount of used bytes, it goes back the initial used space value, seems like the file is getting deleted and stays on a loop.

5

Are you running Sunstone under Apache?

6

How to tell? The port 9869 is opened by ruby

tcp 0 0 0.0.0.0:9869 0.0.0.0:* LISTEN 423616/ruby