Iscsi datastore

Hi All,

Hope everybody is safe and doing good in this pandemic.

I was testing opennebula setup for self learning purpose.I tried to configure ISCSI datastore for libvirt but opennebula created the image_ds which is not desired state.
Can anybody help me to guide what is the best way to configure system datastore for vms.(For VM High Availability) and guide me how to configure that.

One more thing i checked the SAN as datastore but it seems this functionality is only available in ENTERPRISE version.Is this true please confirm?

Thanks in advance.

Hi @Amit_Setia,

You can find the documentation on how to create iSCSI datastores here: iSCSI - Libvirt Datastore — OpenNebula 6.2.0 documentation. Please note that iSCSI datastores are intended to provide the VM access to the entire iSCSI storage.

One more thing i checked the SAN as datastore but it seems this functionality is only available in ENTERPRISE version

No, every storage driver is available in both versions, CE and EE.

Hi @cgonzalez ,

Thanks for getting back to me and appreciate your efforts.

After seeing the SAN documentation i am putting my efforts to setup SAN/LVM System Datastore.
So following are the things i tried to accomplish this:

  1. Mount a LUN via ISCSI on my Nodes.
  2. created a VG with name as vg-one-ds-id and then create a system ds with below config:
> cat ds.conf
NAME   = lvm_system
TM_MAD = fs_lvm_ssh
TYPE   = SYSTEM_DS
BRIDGE_LIST = "node1.kvm.lvm node2.kvm.lvm"

> onedatastore create ds.conf
ID: 100

But when i create a VM in this datastore it didn’t create any LV in that volume group.It created vm in default DS.
Further i created a LVM image DS, downloaded a image in that and then create a VM, but VM didn’t get deployed and getting below error in logs:

Thu Nov 18 21:48:33 2021 [Z0][ReM][E]: Req:1744 UID:0 one.vm.update result FAILURE [one.vm.update] Cannot update template. Parse error: syntax error, unexpected VARIABLE, expecting EQUAL or EQUAL_EMPTY at line 0, columns 137:139

Also getting error like this image datastore doesn’t support ssh transfer mode while importing/downloading image in image_ds.
Can you please me know where i am making mistake or am i doing all wrong?
Thanks in advance.

Hello @Amit_Setia,

Further i created a LVM image DS, downloaded a image in that and then create a VM, but VM didn’t get deployed and getting below error in logs:

Yes, note that in order to use the fs_lvm_ssh system datastore you need to have a compatible image ds (as stated in the docs). Regarding the error you’re seeing, it doesn’t seem related with this.

Also getting error like this image datastore doesn’t support ssh transfer mode while importing/downloading image in image_ds.

Here it seems that you’re trying to use a wrong combination for your image and system datastores. Could you share the output of onedatastore show -x <id> for the image and system datastore you’re trying to use? Also could you attach the output of oned -v command?

Hi @cgonzalez ,

Here are the required outputs:
System_DS

[root@opennebula ~]# onedatastore show -x 100
<DATASTORE>
  <ID>100</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>lvm_system</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>1</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <DS_MAD><![CDATA[-]]></DS_MAD>
  <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
  <BASE_PATH><![CDATA[/var/lib/one//datastores/100]]></BASE_PATH>
  <TYPE>1</TYPE>
  <DISK_TYPE>0</DISK_TYPE>
  <STATE>0</STATE>
  <CLUSTERS>
    <ID>0</ID>
  </CLUSTERS>
  <TOTAL_MB>0</TOTAL_MB>
  <FREE_MB>0</FREE_MB>
  <USED_MB>0</USED_MB>
  <IMAGES/>
  <TEMPLATE>
    <ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS>
    <BRIDGE_LIST><![CDATA[opennebula01,opennebula02]]></BRIDGE_LIST>
    <DISK_TYPE><![CDATA[FILE]]></DISK_TYPE>
    <DS_MIGRATE><![CDATA[YES]]></DS_MIGRATE>
    <RESTRICTED_DIRS><![CDATA[/]]></RESTRICTED_DIRS>
    <SAFE_DIRS><![CDATA[/var/tmp]]></SAFE_DIRS>
    <SHARED><![CDATA[YES]]></SHARED>
    <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
    <TYPE><![CDATA[SYSTEM_DS]]></TYPE>
  </TEMPLATE>
</DATASTORE>

Image DS

[root@opennebula ~]# onedatastore show -x 101
<DATASTORE>
  <ID>101</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>production</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>1</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <DS_MAD><![CDATA[fs]]></DS_MAD>
  <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
  <BASE_PATH><![CDATA[/var/lib/one//datastores/101]]></BASE_PATH>
  <TYPE>0</TYPE>
  <DISK_TYPE>2</DISK_TYPE>
  <STATE>0</STATE>
  <CLUSTERS>
    <ID>0</ID>
  </CLUSTERS>
  <TOTAL_MB>51175</TOTAL_MB>
  <FREE_MB>42629</FREE_MB>
  <USED_MB>8547</USED_MB>
  <IMAGES>
    <ID>1</ID>
    <ID>2</ID>
  </IMAGES>
  <TEMPLATE>
    <ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS>
    <CLONE_TARGET><![CDATA[SYSTEM]]></CLONE_TARGET>
    <DISK_TYPE><![CDATA[BLOCK]]></DISK_TYPE>
    <DRIVER><![CDATA[raw]]></DRIVER>
    <DS_MAD><![CDATA[fs]]></DS_MAD>
    <LN_TARGET><![CDATA[SYSTEM]]></LN_TARGET>
    <SAFE_DIRS><![CDATA[/var/tmp /tmp]]></SAFE_DIRS>
    <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
    <TYPE><![CDATA[IMAGE_DS]]></TYPE>
  </TEMPLATE>
</DATASTORE>

oned -v

[root@opennebula ~]# oned -v
OpenNebula 6.2.0 (c84c3303)
Copyright 2002-2021, OpenNebula Project, OpenNebula Systems 

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy
of the License at http://www.apache.org/licenses/LICENSE-2.0

Yeah agreed must be wrong using wrong combination.Is there any commands or documents where can i cross verify these.

SAN LVM datastore created and showing the full disk.While creating VM getting below error:

Driver Error
Fri Nov 26 14:12:53 2021: Error executing image transfer script: /var/lib/one/remotes/tm/fs_lvm_ssh/clone: line 89: [: too many arguments /var/lib/one/remotes/tm/fs_lvm_ssh/clone: line 101: [: File not found: integer expression expected ERROR: clone: Command " set -e -o pipefail mkdir -p /var/lib/one//datastores/104/4 hostname -f >"/var/lib/one//datastores/104/4/.host" || : # zero trailing space if [ "no" = "yes" ]; then LVSIZE=$(sudo -n lvs --nosuffix --noheadings --units B -o lv_size "/dev/vg-one-104/lv-one-4-0" | tr -d '[:blank:]') dd if=/dev/zero of="/dev/vg-one-104/lv-one-4-0" bs=32M oflag=seek_bytes iflag=count_bytes seek="10736369664" count="$(( LVSIZE - 10736369664 ))" fi rm -f "/var/lib/one//datastores/104/4/disk.0" ln -s "/dev/vg-one-104/lv-one-4-0" "/var/lib/one//datastores/104/4/disk.0" ssh -n opennebula "tar -cSO /var/lib/one//datastores/105/7d1e565f97637962396538916b2326b2 2> /dev/null" | tar -xO 2> /dev/null > "/dev/vg-one-104/lv-one-4-0"" failed: Error cloning /var/lib/one//datastores/105/7d1e565f97637962396538916b2326b2 to lv-one-4-0 

Just FYI i am able to do ssh to other hosts and local machine also with oneadmin user

New image_DS details output:

[root@opennebula ~]# onedatastore show -x 105
<DATASTORE>
  <ID>105</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>production</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>1</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <DS_MAD><![CDATA[fs]]></DS_MAD>
  <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
  <BASE_PATH><![CDATA[/var/lib/one//datastores/105]]></BASE_PATH>
  <TYPE>0</TYPE>
  <DISK_TYPE>2</DISK_TYPE>
  <STATE>0</STATE>
  <CLUSTERS>
    <ID>0</ID>
  </CLUSTERS>
  <TOTAL_MB>51175</TOTAL_MB>
  <FREE_MB>49141</FREE_MB>
  <USED_MB>2035</USED_MB>
  <IMAGES>
    <ID>5</ID>
  </IMAGES>
  <TEMPLATE>
    <ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS>
    <BRIDGE_LIST><![CDATA[opennebula01 opennebula02]]></BRIDGE_LIST>
    <CLONE_TARGET><![CDATA[SYSTEM]]></CLONE_TARGET>
    <DISK_TYPE><![CDATA[BLOCK]]></DISK_TYPE>
    <DRIVER><![CDATA[raw]]></DRIVER>
    <DS_MAD><![CDATA[fs]]></DS_MAD>
    <LN_TARGET><![CDATA[SYSTEM]]></LN_TARGET>
    <SAFE_DIRS><![CDATA[/var/tmp /tmp]]></SAFE_DIRS>
    <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
    <TYPE><![CDATA[IMAGE_DS]]></TYPE>
  </TEMPLATE>
</DATASTORE>

Will keep this thread updated with my findings.

@cgonzalez Thanks for your inputs.Due to some misconfiguration i was getting the above error and it has been resolved now.

Found one type in below link:
https://docs.opennebula.io/6.0/management_and_operations/vm_management/vm_instances.html#searching-for-vm-instances

Instead of “onevm” command it is written “nvme”. Not sure this is the right place to submit this correction or not.