# System datastore show 100% full

**URL:** <https://forum.opennebula.io/t/system-datastore-show-100-full/7710>\
**Category:** Operations\
**Created:** [October 4, 2019, 4:28pm UTC](https://forum.opennebula.io/t/system-datastore-show-100-full/7710 "2019-10-04T16:28:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![erivera](https://avatars.discourse-cdn.com/v4/letter/e/edb3f5/32.png) [@erivera](https://forum.opennebula.io/u/erivera)\
**Post date:** [October 4, 2019, 4:28pm UTC](https://forum.opennebula.io/t/system-datastore-show-100-full/7710/1 "2019-10-04T16:28:03Z")

</div>

Hello colleagues,

We have a cluster with 4 KVM nodes and 1 Sunstone server. 2 iSCSI LUNs (multipath) are presented to the kvm nodes and VG and LV have been created, but when looking at the GUI, SYSTEM datastore appears with 100% used

 ![on-gui1](https://canada1.discourse-cdn.com/flex031/uploads/opennebula/original/2X/8/8f4074c37aaca004bc58ede05b2fa1abce042162.png). The datstores configurations is as follows:

[root@opennebula-cp ~]# onedatastore show 110  
DATASTORE 110 INFORMATION  
ID : 110  
NAME : lvm\_images  
USER : oneadmin  
GROUP : oneadmin  
CLUSTERS : 0  
TYPE : IMAGE  
DS\_MAD : fs  
TM\_MAD : fs\_lvm  
BASE PATH : /var/lib/one/datastores/110  
DISK\_TYPE : BLOCK  
STATE : READY

DATASTORE CAPACITY  
TOTAL: : 492G  
FREE: : 466.9G  
USED: : 128M  
LIMIT: : -

PERMISSIONS  
OWNER : um-  
GROUP : u–  
OTHER : —

DATASTORE TEMPLATE  
ALLOW\_ORPHANS=“NO”  
CLONE\_TARGET=“SYSTEM”  
DISK\_TYPE=“BLOCK”  
DRIVER=“raw”  
DS\_MAD=“fs”  
LN\_TARGET=“SYSTEM”  
SAFE\_DIRS=“/var/tmp /tmp”  
TM\_MAD=“fs\_lvm”  
TYPE=“IMAGE\_DS”

IMAGES  
[root@opennebula-cp ~]# onedatastore show 111  
DATASTORE 111 INFORMATION  
ID : 111  
NAME : lvm\_system  
USER : oneadmin  
GROUP : oneadmin  
CLUSTERS : 0  
TYPE : SYSTEM  
DS\_MAD : -  
TM\_MAD : fs\_lvm  
BASE PATH : /var/lib/one/datastores/111  
DISK\_TYPE : FILE  
STATE : READY

DATASTORE CAPACITY  
TOTAL: : 500G  
FREE: : 0M  
USED: : 500G  
LIMIT: : -

PERMISSIONS  
OWNER : um-  
GROUP : u–  
OTHER : —

DATASTORE TEMPLATE  
ALLOW\_ORPHANS=“NO”  
BRIDGE\_LIST=“on-hvsc01 on-hvsc02 on-hvsc03 on-hvsc04”  
CLUSTER=“0”  
DISK\_TYPE=“FILE”  
DS\_MIGRATE=“YES”  
RESTRICTED\_DIRS=“/”  
SAFE\_DIRS=“/var/tmp”  
SHARED=“YES”  
TM\_MAD=“fs\_lvm”  
TYPE=“SYSTEM\_DS”

What could be happening?

Centos 7  
OpenNebula 5.8.5

Regards,  
Eduardo Rivera.

---

<div class="post-metadata">

**Author:** ![jorel](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/jorel/32/6552_2.png) [@jorel](https://forum.opennebula.io/u/jorel)\
**Post date:** [October 8, 2019, 1:42pm UTC](https://forum.opennebula.io/t/system-datastore-show-100-full/7710/2 "2019-10-08T13:42:01Z")

</div>

OpenNebula monitors usage of the LVM datastore using `/var/lib/one/remotes/tm/fs_lvm/monitor` which basicaly just ssh to some node from the `BRIDGE_LIST` and runs following command (as oneadmin via sudo)  
`vgdisplay --separator : --units m -o vg_size,vg_free --nosuffix --noheadings -C vg-one-111`

Eventually it fallbacks to `df` command if `vgdisplay` is not available.

Anyway, can you check the monitor script and spot what could be wrong?

---

<div class="post-metadata">

**Author:** ![wt75](https://avatars.discourse-cdn.com/v4/letter/w/ecc23a/32.png) [@wt75](https://forum.opennebula.io/u/wt75)\
**Post date:** [January 31, 2020, 2:51pm UTC](https://forum.opennebula.io/t/system-datastore-show-100-full/7710/3 "2020-01-31T14:51:59Z")

</div>

I have similar issue where BRIDGE\_LIST has 3 nodes (2x2TB and 1x4TB), all with vg-one-XXX volume group. Sometimes the capacity of lvm\_system is showing 2TB sometimes 4TB (like the host to be monitored chosen randomly?). It should show 8TB of total space, shouldn’t it? This is version 5.8.1  
If I knew with what parameters to call /var/lib/one/remotes/tm/fs\_lvm/monitor I could confirm further how it works.

EDIT: I managed to find the reason.  
monitor uses get\_destination\_host function from /var/lib/one/remotes/datastore/libfs.sh  
that randomly chooses host from BRIDGE\_LIST so the lvm\_system capacity is shown from a random node.

Capacity should be total from all nodes that offer vg-one-XXX volume group. Is it a bug?