# After updating from 4.10 to 4.12 ceph became 100% used

**URL:** <https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425>\
**Category:** Product Support\
**Created:** [March 16, 2015, 12:43pm UTC](https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425 "2015-03-16T12:43:11Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![keksior](https://avatars.discourse-cdn.com/v4/letter/k/e47774/32.png) [@keksior](https://forum.opennebula.io/u/keksior)\
**Post date:** [March 16, 2015, 12:43pm UTC](https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425/1 "2015-03-16T12:43:12Z")

</div>

After upgrading from 4.10 to 4.12 my ceph datastore is now  
100% full with 5.2/5.2 TB used. I don’t know from where opennebula gets  
this data if i have in my ceph This:

6485 GB data, 13704 GB used, 8022 GB / 21726 GB avail

Can anyone help solving it out ? I can’t create new VM’s, and that’s a big problem for me.

Please help.

---

<div class="post-metadata">

**Author:** ![RaveNoX](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ravenox/32/4290_2.png) [@RaveNoX](https://forum.opennebula.io/u/RaveNoX)\
**Post date:** [March 16, 2015, 3:19pm UTC](https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425/2 "2015-03-16T15:19:11Z")

</div>

Can you provide output of this commands:

```
ceph df detail

ceph osd pool get <opennebula_pool> size

```

Replace \<opennebula\_pool\> with pool’s name used by opennebula.

---

<div class="post-metadata">

**Author:** ![keksior](https://avatars.discourse-cdn.com/v4/letter/k/e47774/32.png) [@keksior](https://forum.opennebula.io/u/keksior)\
**Post date:** [March 16, 2015, 3:21pm UTC](https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425/3 "2015-03-16T15:21:13Z")

</div>

Here you are. Also i attached one onedatastore list output.

[root@master-01 ceph]# ceph df detail  
GLOBAL:  
SIZE AVAIL RAW USED %RAW USED OBJECTS  
21726G 8021G 13705G 63.08 1725k  
POOLS:  
NAME ID CATEGORY USED %USED OBJECTS DIRTY READ WRITE  
data 0 - 0 0 0 0 0 0  
metadata 1 - 0 0 0 0 0 0  
rbd 2 - 0 0 0 0 0 0  
test 3 - 640G 2.95 173344 164k 39850 791k  
one 4 - 5309G 24.44 1387308 1354k 782M 6959M  
backup 5 - 535G 2.46 205838 201k 1187k 6213k  
[root@master-01 ceph]# ceph osd pool get one size  
size: 2

[oneadmin@master-01 ceph]$ onedatastore list  
ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS TM STAT  
0 system 196.7G 63% - 0 sys - shared on  
1 default 196.7G 63% - 2 img fs shared on  
2 files 196.7G 63% - 2 fil fs ssh on  
100 ceph 5.2T 0% - 74 img ceph ceph on  
106 ssd-lvm-node- 196.7G 63% - 0 img fs shared on

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/ruben/32/19_2.png) [@ruben](https://forum.opennebula.io/u/ruben)\
**Post date:** [March 16, 2015, 4:14pm UTC](https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425/4 "2015-03-16T16:14:22Z")

</div>

Hi

It seems that there is a kind of mismatch between the output of ceph df and that expect by OpenNebula.

The script is located in remotes/datastore/ceph/monitor. It basically gets the output of ceph df and takes the 5th element (MAX AVAIL) which is %USE in your case:

```
MAX_AVAIL=\$($CEPH df | grep "$POOL_NAME" | awk '{print \$5}')                                                                                                                            
USED=\$($CEPH df | grep "$POOL_NAME" | awk '{print \$3}')   

```

In your case it may be better to restore the previous version of the script.

Cheers

---

<div class="post-metadata">

**Author:** ![keksior](https://avatars.discourse-cdn.com/v4/letter/k/e47774/32.png) [@keksior](https://forum.opennebula.io/u/keksior)\
**Post date:** [March 17, 2015, 7:23am UTC](https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425/5 "2015-03-17T07:23:14Z")

</div>

I didn’t do the backup of the /var/lib/one/remotes. Can anyone paste here code from the 4.10 version?

---

<div class="post-metadata">

**Author:** ![knawnd](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/knawnd/32/6119_2.png) [@knawnd](https://forum.opennebula.io/u/knawnd)\
**Post date:** [March 17, 2015, 8:22am UTC](https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425/6 "2015-03-17T08:22:27Z")

</div>

I guess [that](http://dev.opennebula.org/projects/opennebula/repository/changes/src/datastore_mad/remotes/ceph/monitor?rev=one-4.10) is what you are looking for (press “Download” link).

---

<div class="post-metadata">

**Author:** ![keksior](https://avatars.discourse-cdn.com/v4/letter/k/e47774/32.png) [@keksior](https://forum.opennebula.io/u/keksior)\
**Post date:** [March 17, 2015, 8:32am UTC](https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425/7 "2015-03-17T08:32:46Z")

</div>

Thank you very much. 🙂 Solved

---

<div class="post-metadata">

**Author:** ![apdibbo](https://avatars.discourse-cdn.com/v4/letter/a/0ea827/32.png) [@apdibbo](https://forum.opennebula.io/u/apdibbo)\
**Post date:** [July 21, 2015, 10:08am UTC](https://forum.opennebula.io/t/after-updating-from-4-10-to-4-12-ceph-became-100-used/425/8 "2015-07-21T10:08:36Z")

</div>

Hi,

I am having the same issue however replacing monitor with the one from 14.10 did not resolve the issue for me.

When I run the monitor command from the logs I get the following result:

```
/usr/lib/ruby/1.8/rexml/parsers/treeparser.rb:92:in `parse': #<REXML::ParseException: #<NoMethodError: undefined method `[]' for nil:NilClass> (REXML::ParseException)
/usr/lib/ruby/1.8/rexml/parsers/baseparser.rb:330:in `pull'
/usr/lib/ruby/1.8/rexml/parsers/treeparser.rb:22:in `parse'
/usr/lib/ruby/1.8/rexml/document.rb:245:in `build'
/usr/lib/ruby/1.8/rexml/document.rb:43:in `initialize'
/var/lib/one/remotes/datastore/ceph/../xpath.rb:58:in `new'
/var/lib/one/remotes/datastore/ceph/../xpath.rb:58
...
Exception parsing
Line:
Position:
Last 80 unconsumed characters:
</LN_TARG>
/usr/lib/ruby/1.8/rexml/parsers/baseparser.rb:418:in `pull'
/usr/lib/ruby/1.8/rexml/parsers/treeparser.rb:22:in `parse'
/usr/lib/ruby/1.8/rexml/document.rb:245:in `build'
/usr/lib/ruby/1.8/rexml/document.rb:43:in `initialize'
/var/lib/one/remotes/datastore/ceph/../xpath.rb:58:in `new'
/var/lib/one/remotes/datastore/ceph/../xpath.rb:58
...
#<NoMethodError: undefined method `[]' for nil:NilClass>
/usr/lib/ruby/1.8/rexml/parsers/baseparser.rb:330:in `pull'
/usr/lib/ruby/1.8/rexml/parsers/treeparser.rb:22:in `parse'
/usr/lib/ruby/1.8/rexml/document.rb:245:in `build'
/usr/lib/ruby/1.8/rexml/document.rb:43:in `initialize'
/var/lib/one/remotes/datastore/ceph/../xpath.rb:58:in `new'
/var/lib/one/remotes/datastore/ceph/../xpath.rb:58
...
Exception parsing
Line:
Position:
Last 80 unconsumed characters:
</LN_TARG
Line:
Position:
Last 80 unconsumed characters:
</LN_TARG
        from /usr/lib/ruby/1.8/rexml/document.rb:245:in `build'
        from /usr/lib/ruby/1.8/rexml/document.rb:43:in `initialize'
        from /var/lib/one/remotes/datastore/ceph/../xpath.rb:58:in `new'
        from /var/lib/one/remotes/datastore/ceph/../xpath.rb:58
/var/lib/one/remotes/datastore/ceph/../libfs.sh: line 238: RANDOM % 0: division by 0 (error token is "0")
ERROR MESSAGE --8<------
Datastore template missing 'BRIDGE_LIST' attribute.
ERROR MESSAGE ------>8--

```

Is there anything else you can advise?

Thanks

Alex