# Gluster system datastore

**URL:** <https://forum.opennebula.io/t/gluster-system-datastore/3382>\
**Category:** Product Support\
**Created:** [November 29, 2016, 10:44pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382 "2016-11-29T22:44:28Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [November 29, 2016, 10:44pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/1 "2016-11-29T22:44:28Z")

</div>

Hi,

As I can see on [http://docs.opennebula.org/5.2/operation/host\_cluster\_management/datastore\_guide.html](http://docs.opennebula.org/5.2/operation/host_cluster_management/datastore_guide.html), runing VMs may run on system datastores:

Looks that following the procedure on:  
[http://docs.opennebula.org/4.12/administration/storage/gluster\_ds.html](http://docs.opennebula.org/4.12/administration/storage/gluster_ds.html)

It should be possible to create a system datastore (this is the system datastore chapter) but as you can see in the example the dastore is an image datastore:

onedatastore list  
ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS TM  
0 system 9.9G 98% - 0 sys - shared  
1 default 9.9G 98% - 2 img fs shared  
2 files 12.3G 66% - 0 fil fs ssh  
**101 default 9.9G 98% - 0 img fs shared**

Is there any procedure to create a gluster system datastore?

Thanks a lot,

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 2, 2016, 5:33pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/2 "2016-12-02T17:33:27Z")

</div>

Hi, if it is not possible, is there any way to run VMs on an image datastore?

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![mcabrerizo](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/mcabrerizo/32/4829_2.png) [@mcabrerizo](https://forum.opennebula.io/u/mcabrerizo)\
**Post date:** [December 3, 2016, 8:30am UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/3 "2016-12-03T08:30:37Z")

</div>

Morning oscar!,  
I haven’t used Gluster in my OpenNebula setups so I can’t help you with that ☹ but as you mentioned that you want to use Gluster with OpenNebula 5.2, these recent conversations in the forum will provide you info about Gluster and OpenNebula 5:

- [GlusterFS support](https://forum.opennebula.io/t/glusterfs-support/2946/3)
- [Missing GlusterFS in the datastore?](https://forum.opennebula.io/t/missing-glusterfs-in-the-datastore/2415/3)

I’m sure that users in the forum that are working with Gluster will help you with your setup.

For you second question, you can’t run VMs on an image datastore, as OpenNebula transfers images from the image datastore to the system datastore in the node where the hypervisor runs the VM.

Cheers!

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 3, 2016, 9:45am UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/4 "2016-12-03T09:45:02Z")

</div>

Hi Miguel Angel,

I have took a look to that notes but there is no explanation about how to configure gluster as system dataster (it does for image datastore).

I will have to wait anybody in this forum with experience in glusterfs as system datastore.

Nevertheles, I’d like to ask if this configuration is suitable. In my farm, I’d like to have two system datastores:  
1.- GlusterFS system datastore to run servers (sunstone, sql, and so on…) that can migrate from one host to other (persistent)  
2.- Local FileSystem system datastore to run Windows 7 VDI (persistent or not persistent)

In this scenario, how can I instruct OpenNebula to, on vm creation, to use one system datastore or the other one?

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![Daryl\_Lee](https://avatars.discourse-cdn.com/v4/letter/d/9e8a1a/32.png) [@Daryl\_Lee](https://forum.opennebula.io/u/Daryl_Lee)\
**Post date:** [December 4, 2016, 12:06am UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/5 "2016-12-04T00:06:31Z")

</div>

I use OpenNebula with a GlusterFS datastore. Say you create your GlusterFS datastore, that one would be shared. The other one you would probably create as a SSH datastore, which means it will be copied to the local opennebula node when it’s created and then launched locally on that system.

When you create the “images” that your VMs will use as it’s hard drive, you can choose what datastore to use.

Example 1: I want to create a new server on my GlusterFS datastore.

1. Create a new image (the hard drive for the VM), when creating it you will choose the GlusterFS datastore.
2. This will create the file on that share. However, you need more than just a datastore to get this working. OpenNebula Front end needs to access the share using FUSE, so make sure it’s mounted to /var/lib/one/datastores/100 or whatever the ID of your Gluster datastore is.
3. Depending on if you can use GFAPI for native access with KVM, the VM will use the GlusterFS directly without using FUSE (which is slow) and then leave the FUSE mounted datastore directory for everything else.
4. If you don’t plan on using KVM with GFAPI you will need to simply create a shared datastore pointed to the FUSE Mounted gluster partition. If you are using CentOS/Redhat, GFAPI is already compiled and available for you to use.

To be honest, it’s hard to answer this without know your current level of experience with OpenNebula and what your setup is like. Maybe next week we can skype or something and I can answer your questions more fluidly and directly.

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 4, 2016, 12:23am UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/6 "2016-12-04T00:23:27Z")

</div>

Hi Daryl,

I don’t have so much experience with OpenNebula but I have working hard latest two weeks with it.

What I want is creating a farm with 3 hosts and create two gluster volumes:  
1.- Images datastore (GlusterFS)  
2.- System datastore (GlusterFS)

As both volumes are shared between 3 nodes I want to deploy on int two centos servers (one for mysql, and the other one for sunstone). That way I will provide HA to sunstone.

Regarding to the access, I’m using GFAPI and looks system is using port 24007 to acceed the system.

I have been able to create the images datastore (Gluster) and create a soft link against the mount point. But I’d like to create the gluster System Datastore.

Thanks a lot.

Afterwards, I will start working with my vdi infraestructure.

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 4, 2016, 12:35am UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/7 "2016-12-04T00:35:20Z")

</div>

Nevertheless, I have tried your suggested configuration, it is:

1.- Gluster datastore for images:  
2.- Local ssh system datastore as system datasatore.

But it looks does not work either: I have pasted the logs:

[http://paste.fedoraproject.org/497916/81153814/](http://paste.fedoraproject.org/497916/81153814/)

I will feel very glad if we can work together during some minutes on my configuration in order to make it work!

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 4, 2016, 7:09pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/8 "2016-12-04T19:09:16Z")

</div>

Hi,

I have added a third disk for gluster system datastore test. In this moments I have:

1.- Gluster datastore for images (vdic-core)  
2.- Local ssh system datastore as system datasatore (vdic-vm)  
3.- Gluster datastore for “system datastore” (vdic-core-vm)

I have tried to configure my new vdic-core-vm gluster datastore as follows:

```
NAME = "vdic-core-vm"
TM_MAD = shared

# the following line *must* be preset
DISK_TYPE = GLUSTER
TYPE = SYSTEM_DS

GLUSTER_HOST = localhost:24007
GLUSTER_VOLUME = vdic-core-vm-gv0

```

But I get the following error:

```
[oneadmin@vdicone01 ~]$ onedatastore create vdic-core-vm-ds.conf
[DatastoreAllocate] Invalid DISK_TYPE for a System Datastore.

```

Means glusterfs cannot be system datastore?

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 11, 2016, 4:39pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/9 "2016-12-11T16:39:11Z")

</div>

Hi,

Anybody has been able to make system work with gluster?

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![Snowman](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Snowman](https://forum.opennebula.io/u/Snowman)\
**Post date:** [December 12, 2016, 8:52am UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/10 "2016-12-12T08:52:42Z")

</div>

What exactly dont work in your setup with Gluster?

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 12, 2016, 11:49am UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/11 "2016-12-12T11:49:55Z")

</div>

I cannot create a gluster system datastore.

```
[oneadmin@vdicone01 ~]$ onedatastore create vdic-core-vm-ds.conf
[DatastoreAllocate] Invalid DISK_TYPE for a System Datastore.

```

I want to use gluster to have the live host migration feature.

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\
**Post date:** [December 12, 2016, 1:53pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/12 "2016-12-12T13:53:42Z")

</div>

Ho oscar,

You should use the GlusterFS as pre-mounted filesystem on the hosts([https://docs.opennebula.org/5.2/deployment/open\_cloud\_storage\_setup/fs\_ds.html#shared-qcow2-transfer-modes](https://docs.opennebula.org/5.2/deployment/open_cloud_storage_setup/fs_ds.html#shared-qcow2-transfer-modes)) and use `shared` or `qcow2` TM\_MAD drivers with _DISK\_TYPE=FILE_

For example, mount a gluster volume on all hosts (for example in _/gluster/vdic-core-vm/_)  
Create a system datastore with the following template

```
NAME = "vdic-core-vm"
TM_MAD = shared
DISK_TYPE = FILE
TYPE = SYSTEM_DS

```

Check the datastore ID and create symlink in /var/lib/one/datastores/ to /gluster/vdic-core-vm/ on all hosts. For example, if the system datastore id is `105`

```
ln -s /gluster/vdic-core-vm /var/lib/one/datastores/105

```

Other option is to use `qcow2` TM\_MAD to have the files in qcow2 format.

This will leave the context and the volatile disks as files on the mounted shared gluster volume. and the VM disks from the gluster image datastore it will use libgfapi.

AFAIK [Gluster pools](https://libvirt.org/storage.html#StorageBackendGluster) are not supported by OpenNebula

Kind Regards,  
Anton Todorov

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 12, 2016, 3:41pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/13 "2016-12-12T15:41:29Z")

</div>

Hi Anton.

Thanks a lot for your clarifications.

This means that if I’m using Non Persistent images all volatile data access (like windows pagefile) will be acceeded by mount point - FUSE access- (qcow2-shapshot or shared-copy). On the other hand, If I’m creating persistent images all data will be acceded by a link, and therefore again through the mountpoint (FUSE access).

Is this correct?

 ![](https://canada1.discourse-cdn.com/flex031/uploads/opennebula/original/2X/4/46d7a6468dfbeaa8835f7d59ba3dbfa80050436f.png)ed

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\
**Post date:** [December 12, 2016, 4:24pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/14 "2016-12-12T16:24:33Z")

</div>

I am not very familiar with gluster, but the image that you postt looks like not in sync with the current v5.2. I’ve take a quick look at the sources and the code in OpenNebula is in sort of “floating” state - there is support for libgfapi in the core(it can build XML configuration) but there are no drivers to manage gluster backed datastores.

IMO to use FUSE for as little as possible the best solution is to write a driver( DS\_MAD and TM\_MAD) to manage gluster backed image and system datastores. This way you could have libgfapi access to the perssitent, non-persistent and volatile disks on one side and the vm definition XML, the context iso and the checkpoint files on FUSE filesystem.  
Here is the documentation regarding the storage drivers development [http://docs.opennebula.org/5.2/integration/infrastructure\_integration/sd.html#](http://docs.opennebula.org/5.2/integration/infrastructure_integration/sd.html#)

Kind Regards,  
Anton Todorov

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 12, 2016, 5:40pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/15 "2016-12-12T17:40:55Z")

</div>

Hi Anton,

You are right, this screenshot comes from an old fashioned version and I have not been able to find a new version of this document:

[http://docs.opennebula.org/4.12/administration/storage/system\_ds.html](http://docs.opennebula.org/4.12/administration/storage/system_ds.html)

Regarding to the developement of the DS\_MAD and the TM\_MAD drivers, if code has not been removed, accees to the images myb be done throug libgfapi:

> **[Native GlusterFS Image Access for KVM Drivers - OpenNebula – Open Source...](https://opennebula.io/blog/experiences/native-glusterfs-image-access-for-kvm-drivers/)**
>
> GlusterFS is a distributed filesystem with replica and storage distribution features that come really handy for virtualization. This storage can be mounted as a filesystem using NFS or FUSE adapter for GlusterFS and is used as any other shared...

> [@GlusterFS support](https://forum.opennebula.io/t/glusterfs-support/2946/14):
>
> Here are some references that might help. [https://github.com/OpenNebula/one/blob/master/src/vmm/LibVirtDriverKVM.cc#L610-L639](https://github.com/OpenNebula/one/blob/master/src/vmm/LibVirtDriverKVM.cc#L610-L639) I tried to get behind the fuse setup and ran some benchmarks and it wasn’t going to happen. The KVM VM loads with the gluster:// parameter. I’m looking at the process on the host now and it’s running file=gluster://gluster1:24007/imageid Here is my original datastore.conf i created when setting this up: NAME = "glusterds" DS\_MAD = fs TM\_MAD = shared DISK\_TYPE …

I’m a simple system administrator and I dont feel confident to develope a stable driver for gluster. Do you? 😉

Thanks a lot for your help.

---

<div class="post-metadata">

**Author:** ![Snowman](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Snowman](https://forum.opennebula.io/u/Snowman)\
**Post date:** [December 13, 2016, 8:09am UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/16 "2016-12-13T08:09:56Z")

</div>

I can confirm that you can use libgfapi to connect OpenNebula to GlusterFS. But afaik you need to install qemu with libgfsapi compiled in, by default its not compiled.

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 13, 2016, 10:44am UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/17 "2016-12-13T10:44:18Z")

</div>

Hi Martin,

With centos quemu is compiled by default with libgfsapi.

Can you post your “system datastore” configuration?

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![oscar](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/oscar/32/5120_2.png) [@oscar](https://forum.opennebula.io/u/oscar)\
**Post date:** [December 20, 2016, 10:26pm UTC](https://forum.opennebula.io/t/gluster-system-datastore/3382/18 "2016-12-20T22:26:08Z")

</div>

Hi, at least I have configured NFS4 acces using nfs-ganesha and looks work fine.

Thanks a lot!
