Iscsi datastore

Amit_Setia · November 15, 2021, 8:18am

Hi All,

Hope everybody is safe and doing good in this pandemic.

I was testing opennebula setup for self learning purpose.I tried to configure ISCSI datastore for libvirt but opennebula created the image_ds which is not desired state.
Can anybody help me to guide what is the best way to configure system datastore for vms.(For VM High Availability) and guide me how to configure that.

One more thing i checked the SAN as datastore but it seems this functionality is only available in ENTERPRISE version.Is this true please confirm?

Thanks in advance.

cgonzalez · November 16, 2021, 11:14am

Hi @Amit_Setia,

You can find the documentation on how to create iSCSI datastores here: iSCSI - Libvirt Datastore — OpenNebula 6.2.2 documentation. Please note that iSCSI datastores are intended to provide the VM access to the entire iSCSI storage.

One more thing i checked the SAN as datastore but it seems this functionality is only available in ENTERPRISE version

No, every storage driver is available in both versions, CE and EE.

Amit_Setia · November 18, 2021, 4:26pm

Hi @cgonzalez ,

Thanks for getting back to me and appreciate your efforts.

After seeing the SAN documentation i am putting my efforts to setup SAN/LVM System Datastore.
So following are the things i tried to accomplish this:

Mount a LUN via ISCSI on my Nodes.
created a VG with name as vg-one-ds-id and then create a system ds with below config:

> cat ds.conf
NAME   = lvm_system
TM_MAD = fs_lvm_ssh
TYPE   = SYSTEM_DS
BRIDGE_LIST = "node1.kvm.lvm node2.kvm.lvm"

> onedatastore create ds.conf
ID: 100

But when i create a VM in this datastore it didn’t create any LV in that volume group.It created vm in default DS.
Further i created a LVM image DS, downloaded a image in that and then create a VM, but VM didn’t get deployed and getting below error in logs:

Thu Nov 18 21:48:33 2021 [Z0][ReM][E]: Req:1744 UID:0 one.vm.update result FAILURE [one.vm.update] Cannot update template. Parse error: syntax error, unexpected VARIABLE, expecting EQUAL or EQUAL_EMPTY at line 0, columns 137:139

Also getting error like this image datastore doesn’t support ssh transfer mode while importing/downloading image in image_ds.
Can you please me know where i am making mistake or am i doing all wrong?
Thanks in advance.

cgonzalez · November 25, 2021, 12:36pm

Hello @Amit_Setia,

Further i created a LVM image DS, downloaded a image in that and then create a VM, but VM didn’t get deployed and getting below error in logs:

Yes, note that in order to use the fs_lvm_ssh system datastore you need to have a compatible image ds (as stated in the docs). Regarding the error you’re seeing, it doesn’t seem related with this.

Also getting error like this image datastore doesn’t support ssh transfer mode while importing/downloading image in image_ds.

Here it seems that you’re trying to use a wrong combination for your image and system datastores. Could you share the output of onedatastore show -x <id> for the image and system datastore you’re trying to use? Also could you attach the output of oned -v command?

Amit_Setia · November 26, 2021, 5:21am

Hi @cgonzalez ,

Here are the required outputs:
System_DS

[root@opennebula ~]# onedatastore show -x 100
<DATASTORE>
  <ID>100</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>lvm_system</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>1</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <DS_MAD><![CDATA[-]]></DS_MAD>
  <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
  <BASE_PATH><![CDATA[/var/lib/one//datastores/100]]></BASE_PATH>
  <TYPE>1</TYPE>
  <DISK_TYPE>0</DISK_TYPE>
  <STATE>0</STATE>
  <CLUSTERS>
    <ID>0</ID>
  </CLUSTERS>
  <TOTAL_MB>0</TOTAL_MB>
  <FREE_MB>0</FREE_MB>
  <USED_MB>0</USED_MB>
  <IMAGES/>
  <TEMPLATE>
    <ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS>
    <BRIDGE_LIST><![CDATA[opennebula01,opennebula02]]></BRIDGE_LIST>
    <DISK_TYPE><![CDATA[FILE]]></DISK_TYPE>
    <DS_MIGRATE><![CDATA[YES]]></DS_MIGRATE>
    <RESTRICTED_DIRS><![CDATA[/]]></RESTRICTED_DIRS>
    <SAFE_DIRS><![CDATA[/var/tmp]]></SAFE_DIRS>
    <SHARED><![CDATA[YES]]></SHARED>
    <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
    <TYPE><![CDATA[SYSTEM_DS]]></TYPE>
  </TEMPLATE>
</DATASTORE>

Image DS

[root@opennebula ~]# onedatastore show -x 101
<DATASTORE>
  <ID>101</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>production</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>1</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <DS_MAD><![CDATA[fs]]></DS_MAD>
  <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
  <BASE_PATH><![CDATA[/var/lib/one//datastores/101]]></BASE_PATH>
  <TYPE>0</TYPE>
  <DISK_TYPE>2</DISK_TYPE>
  <STATE>0</STATE>
  <CLUSTERS>
    <ID>0</ID>
  </CLUSTERS>
  <TOTAL_MB>51175</TOTAL_MB>
  <FREE_MB>42629</FREE_MB>
  <USED_MB>8547</USED_MB>
  <IMAGES>
    <ID>1</ID>
    <ID>2</ID>
  </IMAGES>
  <TEMPLATE>
    <ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS>
    <CLONE_TARGET><![CDATA[SYSTEM]]></CLONE_TARGET>
    <DISK_TYPE><![CDATA[BLOCK]]></DISK_TYPE>
    <DRIVER><![CDATA[raw]]></DRIVER>
    <DS_MAD><![CDATA[fs]]></DS_MAD>
    <LN_TARGET><![CDATA[SYSTEM]]></LN_TARGET>
    <SAFE_DIRS><![CDATA[/var/tmp /tmp]]></SAFE_DIRS>
    <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
    <TYPE><![CDATA[IMAGE_DS]]></TYPE>
  </TEMPLATE>
</DATASTORE>

oned -v

[root@opennebula ~]# oned -v
OpenNebula 6.2.0 (c84c3303)
Copyright 2002-2021, OpenNebula Project, OpenNebula Systems 

Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy
of the License at http://www.apache.org/licenses/LICENSE-2.0

Yeah agreed must be wrong using wrong combination.Is there any commands or documents where can i cross verify these.

Amit_Setia · November 26, 2021, 8:47am

SAN LVM datastore created and showing the full disk.While creating VM getting below error:

Driver Error
Fri Nov 26 14:12:53 2021: Error executing image transfer script: /var/lib/one/remotes/tm/fs_lvm_ssh/clone: line 89: [: too many arguments /var/lib/one/remotes/tm/fs_lvm_ssh/clone: line 101: [: File not found: integer expression expected ERROR: clone: Command " set -e -o pipefail mkdir -p /var/lib/one//datastores/104/4 hostname -f >"/var/lib/one//datastores/104/4/.host" || : # zero trailing space if [ "no" = "yes" ]; then LVSIZE=$(sudo -n lvs --nosuffix --noheadings --units B -o lv_size "/dev/vg-one-104/lv-one-4-0" | tr -d '[:blank:]') dd if=/dev/zero of="/dev/vg-one-104/lv-one-4-0" bs=32M oflag=seek_bytes iflag=count_bytes seek="10736369664" count="$(( LVSIZE - 10736369664 ))" fi rm -f "/var/lib/one//datastores/104/4/disk.0" ln -s "/dev/vg-one-104/lv-one-4-0" "/var/lib/one//datastores/104/4/disk.0" ssh -n opennebula "tar -cSO /var/lib/one//datastores/105/7d1e565f97637962396538916b2326b2 2> /dev/null" | tar -xO 2> /dev/null > "/dev/vg-one-104/lv-one-4-0"" failed: Error cloning /var/lib/one//datastores/105/7d1e565f97637962396538916b2326b2 to lv-one-4-0

Just FYI i am able to do ssh to other hosts and local machine also with oneadmin user

New image_DS details output:

[root@opennebula ~]# onedatastore show -x 105
<DATASTORE>
  <ID>105</ID>
  <UID>0</UID>
  <GID>0</GID>
  <UNAME>oneadmin</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>production</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>1</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <DS_MAD><![CDATA[fs]]></DS_MAD>
  <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
  <BASE_PATH><![CDATA[/var/lib/one//datastores/105]]></BASE_PATH>
  <TYPE>0</TYPE>
  <DISK_TYPE>2</DISK_TYPE>
  <STATE>0</STATE>
  <CLUSTERS>
    <ID>0</ID>
  </CLUSTERS>
  <TOTAL_MB>51175</TOTAL_MB>
  <FREE_MB>49141</FREE_MB>
  <USED_MB>2035</USED_MB>
  <IMAGES>
    <ID>5</ID>
  </IMAGES>
  <TEMPLATE>
    <ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS>
    <BRIDGE_LIST><![CDATA[opennebula01 opennebula02]]></BRIDGE_LIST>
    <CLONE_TARGET><![CDATA[SYSTEM]]></CLONE_TARGET>
    <DISK_TYPE><![CDATA[BLOCK]]></DISK_TYPE>
    <DRIVER><![CDATA[raw]]></DRIVER>
    <DS_MAD><![CDATA[fs]]></DS_MAD>
    <LN_TARGET><![CDATA[SYSTEM]]></LN_TARGET>
    <SAFE_DIRS><![CDATA[/var/tmp /tmp]]></SAFE_DIRS>
    <TM_MAD><![CDATA[fs_lvm_ssh]]></TM_MAD>
    <TYPE><![CDATA[IMAGE_DS]]></TYPE>
  </TEMPLATE>
</DATASTORE>

Will keep this thread updated with my findings.

Amit_Setia · November 27, 2021, 7:05am

@cgonzalez Thanks for your inputs.Due to some misconfiguration i was getting the above error and it has been resolved now.

Found one type in below link:
https://docs.opennebula.io/6.0/management_and_operations/vm_management/vm_instances.html#searching-for-vm-instances

Instead of “onevm” command it is written “nvme”. Not sure this is the right place to submit this correction or not.

cgonzalez · November 30, 2021, 3:48pm

Hi @Amit_Setia, nice to hear that! It would be nice if you could share the error here in case somebody else fall in the same one.

Regarding the typo in the docs, if you want you can just open a PR at GitHub with the fix.

Amit_Setia · December 28, 2021, 11:38am

Hi @cgonzalez

Oops My bad , I was experimenting on the cluster and i have recreated it multiple times and now i don’t have error/logs.
Next time will collect all details and in the end summarize it and put it on the forum,so that other can also learn from it.
Sincere Apologies again.

Appreciate your help once again.

Topic		Replies	Views
Help with iscsi_libvirt datastores Community Support	6	1643	June 3, 2021
iSCSI libvirt datastore not working as expected Community Support	0	480	May 17, 2019
Opennebula iSCSi Datastore Capacity at 1MB Storage	3	458	June 28, 2021
Best driver with iSCSI shared storage Storage	20	2051	November 21, 2019
Default Image Datastore + SAN System Datastore Storage	7	1164	April 7, 2022

Iscsi datastore

Related topics