# Shared LVM datastore

**URL:** <https://forum.opennebula.io/t/shared-lvm-datastore/4034>\
**Category:** Product Support\
**Created:** [March 26, 2017, 9:49am UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034 "2017-03-26T09:49:41Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![UAnton](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/uanton/32/6309_2.png) [@UAnton](https://forum.opennebula.io/u/UAnton)\
**Post date:** [March 26, 2017, 9:49am UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/1 "2017-03-26T09:49:41Z")

</div>

Hello,  
I try setup LVM datastore.

1. Create VG named vg-one-102 (images) and vg-one-103 (system) and datastores 102, 103.
2. System DS show me right LUN size 2Tb, Image DS show local disk size.
3. Try setup VM. VM create disk on /var/lib/one/datastores/103/… not in LVM
4. lvs, lvscan, lvdisplay output - File descriptor 7 (pipe:[375538]) leaked on lvs invocation. Parent PID 19006: bash  
What I do wrong or LVM in Opennebula did not start working properly.  
Thanks!

---

<div class="post-metadata">

**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\
**Post date:** [March 26, 2017, 1:22pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/2 "2017-03-26T13:22:31Z")

</div>

Hi Anton,

> [@UAnton](#):
>
> I try setup LVM datastore.

Usual questions 🙂… What TM\_MAD and DS\_MAD do you use for the datastores, what is their configuration in `/etc/one/oned.conf`, which configuration guid do you follow, etc.

> [@UAnton](#):
>
> System DS show me right LUN size 2Tb, Image DS show local disk size

Which local disk - on the front-end or on the kvm host?

Some more background is needed to figure out where is the issue.

Kind Regards,  
Anton Todorov

---

<div class="post-metadata">

**Author:** ![UAnton](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/uanton/32/6309_2.png) [@UAnton](https://forum.opennebula.io/u/UAnton)\
**Post date:** [March 26, 2017, 1:27pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/3 "2017-03-26T13:27:47Z")

</div>

Hello Anton,  
I use this docs - [http://docs.opennebula.org/5.2/deployment/open\_cloud\_storage\_setup/lvm\_drivers.html](http://docs.opennebula.org/5.2/deployment/open_cloud_storage_setup/lvm_drivers.html)

[root@node02 one]# onedatastore list  
ID NAME SIZE AVAIL CLUSTERS IMAGES TYPE DS TM STAT  
2 files 167.6G 97% 0 0 fil fs ssh on  
105 production 167.6G 97% 0 0 img fs fs\_lvm on  
106 lvm\_system 500G 100% 0 0 sys - fs\_lvm on

Now IMGAGE show local space.

Local Disc on node01 (Frontend + kvm node)

---

<div class="post-metadata">

**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\
**Post date:** [March 26, 2017, 2:07pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/4 "2017-03-26T14:07:24Z")

</div>

Hello Anton,

> [@UAnton](#):
>
> Now IMGAGE show local space.

I could bet you mean the SYSTEM DS 🙂  
The system DS is monitored with remotes/tm/fs\_lvm/monitor which returns LVM sizes if LVM VG is found otherwise the local filesystem space.  
The image DS is monitored with remotes/datastore/fs/monitor which looks in the local filesystem only so it is working as designed but in your case it is wrong. I think that some code from tm/fs\_lvm/monitor should be merged to datastore/fs/monitor to cover this use case.

The VM deployment is baffling me because same code is used to provision both persistent and non-persistent images by symlinking to the LVM LV and there is no way to create a file. Are you sure that there is a file not symlink to `/dev/$VG/$LV`? Could you provide the output of `ls -la /var/lib/one/datastores/$DS_ID/$VM_ID` ?

Kind Regards,  
Anton Todorov

---

<div class="post-metadata">

**Author:** ![UAnton](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/uanton/32/6309_2.png) [@UAnton](https://forum.opennebula.io/u/UAnton)\
**Post date:** [March 26, 2017, 2:14pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/5 "2017-03-26T14:14:06Z")

</div>

Hello Anton,  
[root@node02 one]# ls -la /var/lib/one/datastores/106/19  
total 368  
drwxrwxr-x 2 oneadmin oneadmin 51 Mar 26 17:09 .  
drwxrwxr-x 3 oneadmin oneadmin 15 Mar 26 17:09 …  
-rw-rw-r-- 1 oneadmin oneadmin 1035 Mar 26 17:09 deployment.0  
lrwxrwxrwx 1 oneadmin oneadmin 27 Mar 26 17:09 disk.0 -\> /dev/vg-one-106/lv-one-19-0  
-rw-r–r-- 1 oneadmin oneadmin 372736 Mar 26 17:09 disk.1

I can’t create VMs. every time get error:  
Sun Mar 26 17:09:17 2017 : Error deploying virtual machine: Could not create domain from /var/lib/one//datastores/106/19/deployment.0

---

<div class="post-metadata">

**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\
**Post date:** [March 26, 2017, 2:23pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/6 "2017-03-26T14:23:58Z")

</div>

> [@UAnton](#):
>
> Sun Mar 26 17:09:17 2017 : Error deploying virtual machine: Could not create domain from /var/lib/one//datastores/106/19/deployment.0

There should be some more lines around this one in /var/log/one/oned.log pointing to exact reason for the deployment failure of the VM…

Anton

---

<div class="post-metadata">

**Author:** ![UAnton](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/uanton/32/6309_2.png) [@UAnton](https://forum.opennebula.io/u/UAnton)\
**Post date:** [March 26, 2017, 2:29pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/7 "2017-03-26T14:29:56Z")

</div>

Hello Anton,  
Try new VM from Market template Ubuntu 16.

Sun Mar 26 17:26:15 2017 [Z0][VM][I]: New state is ACTIVE  
Sun Mar 26 17:26:15 2017 [Z0][VM][I]: New LCM state is PROLOG  
Sun Mar 26 17:26:54 2017 [Z0][VM][I]: New LCM state is BOOT  
Sun Mar 26 17:26:54 2017 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/20/deployment.0  
Sun Mar 26 17:26:54 2017 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm\_context.  
Sun Mar 26 17:26:55 2017 [Z0][VMM][I]: ExitCode: 0  
Sun Mar 26 17:26:55 2017 [Z0][VMM][I]: Successfully execute network driver operation: pre.  
Sun Mar 26 17:26:55 2017 [Z0][VMM][I]: Command execution fail: cat \<\< EOT | /var/tmp/one/vmm/kvm/deploy ‘/var/lib/one//datastores/106/20/deployment.0’ ‘172.16.10.212’ 20 172.16.10.212  
Sun Mar 26 17:26:55 2017 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/106/20/deployment.0  
Sun Mar 26 17:26:55 2017 [Z0][VMM][I]: error: internal error: process exited while connecting to monitor: 2017-03-26T14:26:54.889761Z qemu-kvm: -drive file=/var/lib/one//datastores/106/20/disk.0,format=qcow2,if=none,id=drive-virtio-disk0,cache=none: Image is not in qcow2 format  
Sun Mar 26 17:26:55 2017 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/106/20/deployment.0  
Sun Mar 26 17:26:55 2017 [Z0][VMM][I]: ExitCode: 255  
Sun Mar 26 17:26:55 2017 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.  
Sun Mar 26 17:26:55 2017 [Z0][VMM][E]: Error deploying virtual machine: Could not create domain from /var/lib/one//datastores/106/20/deployment.0  
Sun Mar 26 17:26:55 2017 [Z0][VM][I]: New LCM state is BOOT\_FAILURE

---

<div class="post-metadata">

**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\
**Post date:** [March 26, 2017, 2:38pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/8 "2017-03-26T14:38:31Z")

</div>

> [@UAnton](#):
>
> Sun Mar 26 17:26:55 2017 [Z0][VMM][I]: error: internal error: process exited while connecting to monitor: 2017-03-26T14:26:54.889761Z qemu-kvm: -drive file=/var/lib/one//datastores/106/20/disk.0,format=qcow2,if=none,id=drive-virtio-disk0,cache=none: Image is not in qcow2 format

The LVM LV is raw, but somewhere in the template (or in the image) you have “image mapping driver” set to qcow2…

check the image attributes regarding the mapping driver following this order the VM template, the “master” image attributes, the default values in /etc/one/oned.conf (or /etc/one/vmm\_exec/kvm not sure where was exactly)

As a workaround you can try setting the “image mapping driver” to `raw` in the VM template (VM template → update → storage → `Advanced options`)

Best,  
Anton

Best,  
Anton

---

<div class="post-metadata">

**Author:** ![UAnton](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/uanton/32/6309_2.png) [@UAnton](https://forum.opennebula.io/u/UAnton)\
**Post date:** [March 26, 2017, 2:46pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/9 "2017-03-26T14:46:07Z")

</div>

Thanks Anton!  
Works!  
I go to tests!

---

<div class="post-metadata">

**Author:** ![UAnton](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/uanton/32/6309_2.png) [@UAnton](https://forum.opennebula.io/u/UAnton)\
**Post date:** [March 26, 2017, 4:22pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/10 "2017-03-26T16:22:27Z")

</div>

Hello Anton,  
Maybe you can help me. With LVM DS migraton not working. I try live and cold.  
Thanks!

Sun Mar 26 18:39:37 2017 [Z0][VM][I]: New LCM state is MIGRATE  
Sun Mar 26 18:39:37 2017 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm\_premigrate.  
Sun Mar 26 18:39:37 2017 [Z0][VMM][I]: ExitCode: 0  
Sun Mar 26 18:39:37 2017 [Z0][VMM][I]: Successfully execute network driver operation: pre.  
Sun Mar 26 18:39:38 2017 [Z0][VMM][I]: Command execution fail: cat \<\< EOT | /var/tmp/one/vmm/kvm/migrate ‘one-22’ ‘172.16.10.211’ ‘172.16.10.212’ 22 172.16.10.212  
Sun Mar 26 18:39:38 2017 [Z0][VMM][E]: migrate: Command “virsh --connect qemu:///system migrate --live one-22 qemu+ssh://172.16.10.211/system” failed: error: Cannot access storage file ‘/var/lib/one//datastores/106/22/disk.1’ (as uid:9869, gid:9869): No such file or directory  
Sun Mar 26 18:39:38 2017 [Z0][VMM][E]: Could not migrate one-22 to 172.16.10.211  
Sun Mar 26 18:39:38 2017 [Z0][VMM][I]: ExitCode: 1  
Sun Mar 26 18:39:38 2017 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm\_failmigrate.  
Sun Mar 26 18:39:38 2017 [Z0][VMM][I]: Failed to execute virtualization driver operation: migrate.  
Sun Mar 26 18:39:38 2017 [Z0][VMM][E]: Error live migrating VM: Could not migrate one-22 to 172.16.10.211  
Sun Mar 26 18:39:38 2017 [Z0][VM][I]: New LCM state is RUNNING  
Sun Mar 26 18:39:38 2017 [Z0][LCM][I]: Fail to live migrate VM. Assuming that the VM is still RUNNING (will poll VM).

---

<div class="post-metadata">

**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\
**Post date:** [March 26, 2017, 4:49pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/11 "2017-03-26T16:49:37Z")

</div>

Hi Anton,

> [@UAnton](#):
>
> Sun Mar 26 18:39:37 2017 [Z0][VMM][I]: Successfully execute network driver operation: pre.

This line tell us that `/var/lib/one/remotes/tm/fs_lvm/premigrate` completed without errors, which means that on the destiration host(172.16.10.211) you should have a symlink `/var/lib/one//datastores/106/22/disk.0` to /dev/$VG/$LV and this VG/LV should be in activated state.

> [@UAnton](#):
>
> Sun Mar 26 18:39:38 2017 [Z0][VMM][E]: migrate: Command “virsh --connect qemu:///system migrate --live one-22 qemu+ssh://172.16.10.211/system” failed: error: Cannot access storage file ‘/var/lib/one//datastores/106/22/disk.1’ (as uid:9869, gid:9869): No such file or directory

But the contextualization CDROM ISO disk is missing or not accessible by the oneadmin user(or the user running libvirtd/qemu-kvm). You should check that the SYSTEM DS path `'/var/lib/one//datastores/106/` is shared, visible and accessible by oneadmin on both kvm hosts. Also, check is selinux/apparmor involved too.

Best,  
Anton

---

<div class="post-metadata">

**Author:** ![UAnton](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/uanton/32/6309_2.png) [@UAnton](https://forum.opennebula.io/u/UAnton)\
**Post date:** [March 26, 2017, 5:15pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/12 "2017-03-26T17:15:00Z")

</div>

Hello Anton,

On second node no VM forlder in datastore. I think LVM DS not work correctly.

How to symlink to lv can move to second node?

---

<div class="post-metadata">

**Author:** ![atodorov\_storpool](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/atodorov_storpool/32/5327_2.png) [@atodorov\_storpool](https://forum.opennebula.io/u/atodorov_storpool)\
**Post date:** [March 26, 2017, 5:24pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/13 "2017-03-26T17:24:07Z")

</div>

Hi Anton,

You should configure shared filesystem (NFS or other) between the nodes. The fs-lvm driver expects shared filesytem between the nodes. As alternative you could patch the premigrate script to copy the source VM folder to the destination host before activating the LVM LVs and the postmigrate script to remove the source VM directory _after_ activating the volumes.

Best,  
Anton Todorov

---

<div class="post-metadata">

**Author:** ![UAnton](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/uanton/32/6309_2.png) [@UAnton](https://forum.opennebula.io/u/UAnton)\
**Post date:** [March 26, 2017, 5:25pm UTC](https://forum.opennebula.io/t/shared-lvm-datastore/4034/14 "2017-03-26T17:25:07Z")

</div>

Hello Anton,  
Thanks, now I understand how it works!