Default Image Datastore + SAN System Datastore

Community Support Storage
1

I have a Dell PowerVault 3200i that I want to use as System Datastore.
I have 1 frontend server and a few nodes (all KVM) all using v6.0.0.2 release.

To get iSCSI working, I followed the instructions here.

On the frontend:
I created 2 datastores: 1 for System and 1 for Image (I don’t understand how the Image DS is used here but I just followed the doc)

$ cat iscsi_lvm_system.ds
NAME = iscsi_lvm_system
TM_MAD = fs_lvm
TYPE = SYSTEM_DS
DISK_TYPE = "BLOCK"
BRIDGE_LIST = "node1 node2 nodeN"
$ onedatastore create iscsi_lvm_system.ds
ID: 102
$ cat issci_lvm_image.ds
NAME = iscsi_lvm_image
DS_MAD = fs
TM_MAD = fs_lvm
DISK_TYPE = "BLOCK"
TYPE = IMAGE_DS
SAFE_DIRS=”/var/tmp /tmp"
BRIDGE_LIST = "node1 node2 nodeN"
$ onedatastore create iscsi_lvm_image.ds
ID: 103

On the nodes:
All nodes are able to access 2 LUNs on the iSCSI and I was able to setup the Volume Groups properly for each DS.

$ sudo vgs
  VG         #PV #LV #SN Attr   VSize     VFree    
  vg-one-102   1   0   0 wz--n-   <14.36t   <14.36t
  vg-one-103   1   0   0 wz--n- <1024.00g <1024.00g

Issues:

  • When I list the datastores on fe, the Image datastore doesn’t have the correct size.
  ID NAME                   SIZE AVA CLUSTERS IMAGES TYPE DS      TM      STAT
 103 iscsi_lvm_image        133G 86% 0             1 img  fs      fs_lvm  on  
 102 iscsi_lvm_system      14.4T 100 0             0 sys  -       fs_lvm  on  
   2 files                  1.1T 94% 0             0 fil  fs      ssh     on  
   1 default                1.1T 94% 0             2 img  fs      ssh     on  
   0 system                    - -   0             0 sys  -       ssh     on
  • I downloaded an alpine image to the Image Datastore from the marketplace, but when I tried to deploy it to a node and specified the iscsi_lvm_system as Datastore, it fails with the ff logs:
Sun Jun 6 12:14:45 2021 [Z0][VM][I]: New state is ACTIVE
Sun Jun 6 12:14:45 2021 [Z0][VM][I]: New LCM state is PROLOG
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: Command execution failed (exit code: 2): /var/lib/one/remotes/tm/fs_lvm/clone frontend:/var/lib/one//datastores/103/2754b8bdc3e009edc02be6c240fcceea node3:/var/lib/one//datastores/102/7/disk.0 7 103
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: /var/lib/one/remotes/tm/fs_lvm/clone: line 87: [: too many arguments
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: /var/lib/one/remotes/tm/fs_lvm/clone: line 99: [: File not found: integer expression expected
Sun Jun 6 12:14:49 2021 [Z0][TrM][E]: clone: Command " set -e -o pipefail
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: mkdir -p /var/lib/one//datastores/102/7
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]:
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: hostname -f >"/var/lib/one//datastores/102/7/.host" || :
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]:
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: # zero trailing space
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: if [ "yes" = "yes" ]; then
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: LVSIZE=$(sudo -n lvs --nosuffix --noheadings --units B -o lv_size "/dev/vg-one-102/lv-one-7-0" | tr -d '[:blank:]')
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: dd if=/dev/zero of="/dev/vg-one-102/lv-one-7-0" bs=64k oflag=seek_bytes iflag=count_bytes seek="267386880" count="$(( LVSIZE - 267386880 ))"
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: fi
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]:
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: if [[ "cannot open `/var/lib/one//datastores/103/2754b8bdc3e009edc02be6c240fcceea' (No such file or directory)" =~ "LUKS encrypted file" ]]; then
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: cat "/var/lib/one//datastores/103/2754b8bdc3e009edc02be6c240fcceea" > "/dev/vg-one-102/lv-one-7-0"
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: else
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: qemu-img convert -O raw "/var/lib/one//datastores/103/2754b8bdc3e009edc02be6c240fcceea" "/dev/vg-one-102/lv-one-7-0"
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: fi
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]:
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: rm -f "/var/lib/one//datastores/102/7/disk.0"
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: ln -s "/dev/vg-one-102/lv-one-7-0" "/var/lib/one//datastores/102/7/disk.0"" failed: 16+0 records in
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: 16+0 records out
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162278 s, 6.5 MB/s
Sun Jun 6 12:14:49 2021 [Z0][TrM][I]: bash: line 14: unexpected EOF while looking for matching ``'
Sun Jun 6 12:14:49 2021 [Z0][TrM][E]: Error cloning /var/lib/one//datastores/103/2754b8bdc3e009edc02be6c240fcceea to lv-one-7-0
Sun Jun 6 12:14:49 2021 [Z0][TrM][E]: Error executing image transfer script: Error cloning /var/lib/one//datastores/103/2754b8bdc3e009edc02be6c240fcceea to lv-one-7-0
Sun Jun 6 12:14:49 2021 [Z0][VM][I]: New LCM state is PROLOG_FAILURE

I guess this is relatetd to the Image Datastore with the storage size not reflecting correctly.

  • I am under the impression that the Local Image Datastore will work with SAN/LVM System Datastores on nodes because there’s no mention in the docs about it. NFS was mentioned but it was only for live migration (which I intend to do but not for now). But when deploying an image from the default Image datatstore to a SAN/LVM System datasotore, it throws an error on the scheduler:
Image from default (local) to LVM (SAN) System datastore will result in:
Sun Jun  6 11:09:37 2021 [Z0][VM][E]: Error deploying virtual machine 6 to HID: 3. Reason: [one.vm.deploy] Image Datastore does not support transfer mode: fs_lvm

Is NFS NECESSARY for SAN/LVM System Datastore to work?
What is the purpose of SAN/LVM Image Datatstore on the Documentation mentioned above?

If anyone can enlighten me, I would really appreciate the help. Keep building!

2

I cannot get the follwoing combo to work when changing type from FILE → BLOCK:
TYPE = SYSTEM_DS
TM_MAD = fs_lvm
DISK_TYPE = “BLOCK”

[one.datastore.update] Cannot update template. Invalid DISK_TYPE for a System Datastore.

3

But this is from the documentation on how to create the SYSTEM DS

> cat ds.conf
NAME   = lvm_system
TM_MAD = fs_lvm_ssh
TYPE   = SYSTEM_DS
BRIDGE_LIST = "node1.kvm.lvm node2.kvm.lvm"

> onedatastore create ds.conf
ID: 100

no DISK_TYPE mentioned and TM_MAD = fs_lvm_ssh

I thought images could be downloaded to the FE and then during instantiation copied via ssh over to the SAN/LVM mounted image DS. Or maybe they should be downloaded directly to that DS ?

4

I think I just started a copy of this thread but I’m using FC instead of iSCSI but I guess the solution will be the same. Documentation is thin on this topic. Following with much anticipation!

5

And sorry Brian, not meaning to highjack just put my thoughts into this.

When I download an image from apps/marketplace into the LVM IMAGE DS it’s the metadata (/var/lib/one/datastores// is only available on a single node. This does indicate that it is up us to get them distributed i.e having that part NFS mounted. But I don’t really like that. What is the point of OpenNebula then? But the actual image is under that directory so what’s even the point of using an LVM based IMAGE DS ?

6

Hi @timedout and @Fredrik_T ,

The SAN Datastore docs [1] mentions the following:

Disk images are stored in file format in the Image Datastore and then dumped into a LV when a Virtual Machine is created. The SAN Datastore can access the Image files in two different ways:

  • NFS mode : The image files are available directly in the Hosts through a distributed file system, e.g. NFS or GlusterFS ( fs_lvm ).
  • SSH mode : The image files are transferred to the Host through the SSH protocol ( fs_lvm_ssh ).

So, trying to answer to your questions:

Is NFS NECESSARY for SAN/LVM System Datastore to work?

No, if you use SSH mode (it wasn’t available for v6.0.0.2).

What is the purpose of SAN/LVM Image Datatstore on the Documentation mentioned above?

Image Datastores act as the image repository from where the images are taken to deploying VMs (into System Datastores).

I thought images could be downloaded to the FE and then during instantiation copied via ssh over to the SAN/LVM mounted image DS. Or maybe they should be downloaded directly to that DS ?

The images are downloaded to the FE and then during instantiation copied via ssh over to the SAN/LVM System DS. But as mentioned in the docs images are stored as file when they are not in use.

Hope this can help you to better understand SAN/LVM Datastores behavior.

[1] SAN Datastore — OpenNebula 6.2.2 documentation

1 Like
7

Hi @cgonzalez.
Thanks for responding! This is the page I have been looking at but cannot get to work. Not sure how to troubleshoot this further at the moment.

8

Hi @FredrikT,

Can you share the exact error you are seeing?