Slow operations in Sunstone

GabrielDias · October 30, 2017, 3:45pm

Every actions with VMs (attach disk, poweroff, resume, migrate & etc) takes from 30 to 60 seconds.
Host is not loaded.
Sometimes Sunstone stopes and process migrate to another host without errors.

Logs are clean. Maybe it is the problem with libvirt? But there are no problems with VMs and libvirtd log same clean.

How i can find problem?

Versions of the related components and OS (frontend, hypervisors, VMs):
CentOS 7, KVM, One 5.4, 33VMs, HA_Cluster

feldsam · October 31, 2017, 12:14pm

Hello, are you use mysql, right? Are you tune mysql setings? Like innodb buffer pool size etc…?

GabrielDias · October 31, 2017, 12:21pm

Hi, yes, I use galera cluster.
All as default. There are no another options.

innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
default_storage_engine=InnoDB
innodb_log_file_size=100M
innodb_file_per_table
innodb_flush_log_at_trx_commit=2

feldsam · October 31, 2017, 12:32pm

I experience simillar problem - sunstone is slow and problem was with small default innodb buffer, whic is just 128MB and our opennebula DB have about 1GB. You can use tool https://github.com/major/MySQLTuner-perl

GabrielDias · October 31, 2017, 12:34pm

Yes, I know about this utility. Thank you.
I shall try to tune mysql.

feldsam · November 1, 2017, 12:35pm

Hello, let us know if it helps. If it help, them it should be added to official documentation. Thank you

GabrielDias · November 1, 2017, 3:14pm

Now I set

[galera]
wsrep_on=ON
binlog_format=ROW
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
query_cache_size=0
query_cache_type=0
default_storage_engine=InnoDB
innodb_log_file_size=16M
innodb_file_per_table
innodb_flush_log_at_trx_commit=0
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
#wsrep_cluster_address="gcomm://"
wsrep_cluster_address="gcomm://10.0.10.1,10.0.10.2,10.0.10.3"
wsrep_cluster_name='galera_cluster'
wsrep_node_address='10.0.10.1'
wsrep_node_name='Server-1'
wsrep_sst_method=rsync
wsrep_sst_auth=USER:PASSWORD

tmp_table_size=32M
max_heap_table_size=64MB
thread_cache_size=4
performance_schema = ON
innodb_buffer_pool_instances=1
wsrep_slave_threads=32

Much better, but still not the same.
I will try to improve, and also I will tune NGINX.

GabrielDias · February 6, 2018, 8:32pm

The problem persists.
After entering sunstone menu loading 30-60s.
All actions with VMs too.

From the cli the same problem with error Net::ReadTimeout

my.cnf

innodb_log_file_size=1G
tmp_table_size=128M
max_heap_table_size=256MB
thread_cache_size=8
performance_schema = ON
innodb_buffer_pool_instances=4
innodb_buffer_pool_size = 4096M
wsrep_slave_threads=64

AbelC · February 7, 2018, 10:06am

Hello @GabrielDias,

We know this problem but not with those measured times or that amount of VMs (22).

We’ve tested load times in Sunstone with the old dashboard (yours).
In environments with more than 2000 VMs, the dashboard takes about 15 seconds to load. We have optimized this with a new dashboard in the next release (5.6.0).

it’s strange to me that with 22 machines, it takes almost a minute to load. I think this shouldn’t be for OpenNebula. The problem must reside in your environment probably.

Anyway, in the next release Sunstone’s charging time decreases substantially.

Regards,
Abel.

GabrielDias · February 8, 2018, 1:05pm

Hi,
Thank you for an answer.

I tried yo update all packages, update kernel, gems.
Try to improved performance of mariadb, but nothing helps me =/

I will be waiting for the new release.

feldsam · February 11, 2018, 7:36pm

Hi, new release is there. Do you analyse mysql process during dashboard loading?

Would you try to run this mysql cmd during slow loading? SHOW FULL PROCESSLIST You should execute it more times to see what happens. Also you should check CPU usage to see what process loads most.

GabrielDias · February 12, 2018, 5:50pm

Yes, I checked MySQL.

MariaDB> SHOW FULL PROCESSLIST;
+----+-------------+----------------+------+---------+--------+--------------------+-----------------------+----------+
| Id | User        | Host           | db   | Command | Time   | State              | Info                  | Progress |
+----+-------------+----------------+------+---------+--------+--------------------+-----------------------+----------+
| 1  | system user |                | NULL | Sleep   | 457081 | wsrep aborter idle | NULL                  | 0.000    |
| 2  | system user |                | NULL | Sleep   | 25     | committed 52391149 | NULL                  | 0.000    |
| 3  | system user |                | NULL | Sleep   | 6      | committed 52391189 | NULL                  | 0.000    |
| 4  | system user |                | NULL | Sleep   | 13     | committed 52391167 | NULL                  | 0.000    |
| 5  | system user |                | NULL | Sleep   | 25     | committed 52391138 | NULL                  | 0.000    |
| 6  | system user |                | NULL | Sleep   | 13     | committed 52391169 | NULL                  | 0.000    |
| 7  | system user |                | NULL | Sleep   | 25     | committed 52391162 | NULL                  | 0.000    |
| 8  | system user |                | NULL | Sleep   | 13     | committed 52391168 | NULL                  | 0.000    |
| 9  | system user |                | NULL | Sleep   | 13     | committed 52391171 | NULL                  | 0.000    |
| 10 | system user |                | NULL | Sleep   | 21     | committed 52391165 | NULL                  | 0.000    |
| 11 | system user |                | NULL | Sleep   | 25     | committed 52391142 | NULL                  | 0.000    |
| 12 | system user |                | NULL | Sleep   | 13     | committed 52391172 | NULL                  | 0.000    |
| 13 | system user |                | NULL | Sleep   | 25     | committed 52391152 | NULL                  | 0.000    |
| 14 | system user |                | NULL | Sleep   | 13     | committed 52391175 | NULL                  | 0.000    |
| 15 | system user |                | NULL | Sleep   | 12     | committed 52391180 | NULL                  | 0.000    |
| 16 | system user |                | NULL | Sleep   | 25     | committed 52391151 | NULL                  | 0.000    |
| 17 | system user |                | NULL | Sleep   | 13     | committed 52391173 | NULL                  | 0.000    |
| 18 | system user |                | NULL | Sleep   | 0      | committed 52391193 | NULL                  | 0.000    |
| 19 | system user |                | NULL | Sleep   | 25     | committed 52391156 | NULL                  | 0.000    |
| 20 | system user |                | NULL | Sleep   | 25     | committed 52391163 | NULL                  | 0.000    |
| 21 | system user |                | NULL | Sleep   | 6      | committed 52391187 | NULL                  | 0.000    |
| 22 | system user |                | NULL | Sleep   | 25     | committed 52391141 | NULL                  | 0.000    |
| 23 | system user |                | NULL | Sleep   | 25     | committed 52391160 | NULL                  | 0.000    |
| 24 | system user |                | NULL | Sleep   | 25     | committed 52391143 | NULL                  | 0.000    |
| 25 | system user |                | NULL | Sleep   | 25     | committed 52391134 | NULL                  | 0.000    |
| 26 | system user |                | NULL | Sleep   | 13     | committed 52391166 | NULL                  | 0.000    |
| 27 | system user |                | NULL | Sleep   | 25     | committed 52391136 | NULL                  | 0.000    |
| 28 | system user |                | NULL | Sleep   | 6      | committed 52391190 | NULL                  | 0.000    |
| 29 | system user |                | NULL | Sleep   | 25     | committed 52391133 | NULL                  | 0.000    |
| 30 | system user |                | NULL | Sleep   | 25     | committed 52391135 | NULL                  | 0.000    |
| 31 | system user |                | NULL | Sleep   | 25     | committed 52391154 | NULL                  | 0.000    |
| 32 | system user |                | NULL | Sleep   | 25     | committed 52391159 | NULL                  | 0.000    |
| 33 | system user |                | NULL | Sleep   | 11     | committed 52391184 | NULL                  | 0.000    |
| 34 | system user |                | NULL | Sleep   | 12     | committed 52391178 | NULL                  | 0.000    |
| 35 | system user |                | NULL | Sleep   | 25     | committed 52391145 | NULL                  | 0.000    |
| 36 | system user |                | NULL | Sleep   | 25     | committed 52391155 | NULL                  | 0.000    |
| 37 | system user |                | NULL | Sleep   | 13     | committed 52391174 | NULL                  | 0.000    |
| 38 | system user |                | NULL | Sleep   | 12     | committed 52391181 | NULL                  | 0.000    |
| 39 | system user |                | NULL | Sleep   | 25     | committed 52391139 | NULL                  | 0.000    |
| 40 | system user |                | NULL | Sleep   | 10     | committed 52391185 | NULL                  | 0.000    |
| 41 | system user |                | NULL | Sleep   | 25     | committed 52391146 | NULL                  | 0.000    |
| 42 | system user |                | NULL | Sleep   | 45     | committed 52391073 | NULL                  | 0.000    |
| 43 | system user |                | NULL | Sleep   | 0      | committed 52391194 | NULL                  | 0.000    |
| 44 | system user |                | NULL | Sleep   | 12     | committed 52391183 | NULL                  | 0.000    |
| 45 | system user |                | NULL | Sleep   | 12     | committed 52391182 | NULL                  | 0.000    |
| 46 | system user |                | NULL | Sleep   | 13     | committed 52391177 | NULL                  | 0.000    |
| 47 | system user |                | NULL | Sleep   | 6      | committed 52391188 | NULL                  | 0.000    |
| 48 | system user |                | NULL | Sleep   | 25     | committed 52391137 | NULL                  | 0.000    |
| 49 | system user |                | NULL | Sleep   | 25     | committed 52391148 | NULL                  | 0.000    |
| 50 | system user |                | NULL | Sleep   | 25     | committed 52391140 | NULL                  | 0.000    |
| 51 | system user |                | NULL | Sleep   | 25     | committed 52391157 | NULL                  | 0.000    |
| 52 | system user |                | NULL | Sleep   | 10     | committed 52391186 | NULL                  | 0.000    |
| 53 | system user |                | NULL | Sleep   | 21     | committed 52391164 | NULL                  | 0.000    |
| 54 | system user |                | NULL | Sleep   | 25     | committed 52391158 | NULL                  | 0.000    |
| 55 | system user |                | NULL | Sleep   | 25     | committed 52391150 | NULL                  | 0.000    |
| 56 | system user |                | NULL | Sleep   | 12     | committed 52391179 | NULL                  | 0.000    |
| 57 | system user |                | NULL | Sleep   | 25     | committed 52391147 | NULL                  | 0.000    |
| 58 | system user |                | NULL | Sleep   | 13     | committed 52391176 | NULL                  | 0.000    |
| 59 | system user |                | NULL | Sleep   | 25     | committed 52391144 | NULL                  | 0.000    |
| 60 | system user |                | NULL | Sleep   | 6      | committed 52391191 | NULL                  | 0.000    |
| 61 | system user |                | NULL | Sleep   | 6      | committed 52391192 | NULL                  | 0.000    |
| 62 | system user |                | NULL | Sleep   | 45     | committed 52391085 | NULL                  | 0.000    |
| 63 | system user |                | NULL | Sleep   | 13     | committed 52391170 | NULL                  | 0.000    |
| 64 | system user |                | NULL | Sleep   | 25     | committed 52391161 | NULL                  | 0.000    |
| 65 | system user |                | NULL | Sleep   | 25     | committed 52391153 | NULL                  | 0.000    |
| 72 | root        | localhost:9457 | NULL | Sleep   | 53     |                    | NULL                  | 0.000    |
| 73 | root        | localhost:9459 | NULL | Query   | 0      | init               | SHOW FULL PROCESSLIST | 0.000    |
+----+-------------+----------------+------+---------+--------+--------------------+-----------------------+----------+
67 rows in set (0.09 sec)

When VM or Dash loading - there are no any processes with high loading.

How it look like…
Dash
-12-2018%2020-43-07

VMs
-12-2018%2020-46-31

UPD:

Topic		Replies	Views
Slow query detected in OpenNebula Sunstone GUI - Sunstone	15	1867	June 18, 2019
URGENT - GET /VM? ERROR from sunstone portal Community Support	3	802	July 8, 2015
Opennebula memory high usage Community Support	4	2024	July 1, 2015
How can build HA for Sunstone? Community Support	9	772	June 5, 2020
CentOS 7 blockdevice performance problems with virtio Community Support	0	940	July 24, 2015

Slow operations in Sunstone

How i can find problem?

Related topics