Hi,
We are working on new private OpenNebula 5.8 deployment. All our compute nodes are connected to same HighSpeed storage over iSCSI. All of our VMs are using persistent images. What would be the recommended way of Datastore deployment in this case?
Thank you.
Hi, what storage system is it? The best way is to write own driver. For example, we have HPE 3PAR storage and we wrote own driver, which communicates with storage over API and for each OpenNebula image it creates volume on the storage system. So basically each VM has its own volume(s) on 3PAR. We also leverage 3PAR QoS functionality, deduplication, and compression. We also have separate stats for each volume. This deployment is abbreviated as “volume centric” and these days replaces old “LUN centric” approach.
It is an Dell/EqualLogic HW controller based storage in m1000E chassis.
this? https://www.dell.com/support/home/si/en/sibsdt1/product-support/product/equallogic-ps-m4110/docs
Yes. 3 of those.
There is no existing driver for this. Of course, you can export one big lun to each node and use clustered FS on top of it with qcow2 images or use cLVM. Both is complicated and complex. Best way is to create driver for you storage system and automate volume creation using ssh cli.
We are able to do this, if you are interested contact us via email. https://www.feldhost.net/products/opennebula
Hello @Morpheus
You can use the LVM driver in OpenNebula. This is the recommended driver for high-end SAN as your case. To configure the nodes, you can take a look at this guide. Then in OpenNebula you need to create the datastores with fs_lvm as TM_MAD. You can find more information about that here.
Finally, to have a complete view of how it works, you can check the full documentation page here.
Hi,
I have followed to instructions, but I am getting following error
oneadmin@amscn03:~ onedatastore create ./ds.cfg
[one.datastore.allocate] Parse error: syntax error, unexpected VARIABLE, expecting EQUAL or EQUAL_EMPTY at line 3, columns 88:95
oneadmin@amscn03:~ cat ./ds.cfg
NAME = AmsMainDataStore
TM_MAD = fs_lvm
TYPE = SYSTEM_DS
BRIDGE_LIST = amscn01 amscn02 amscn03 amscn04 amscn05 amscn06 amscn07 amscn08
oneadmin@amscn03:~$
Hello @Morpheus
Try with the following template:
NAME = AmsMainDataStore
TM_MAD = fs_lvm
TYPE = SYSTEM_DS
BRIDGE_LIST = "amscn01,amscn02,amscn03,amscn04,amscn05,amscn06,amscn07,amscn08"
FS_LVM or BLOCK_LVM uses the LUN centric approach; you export one big LUN, and VMs sit on it. You lost per VM statistic direct on the storage system, lost QoS (bandwidth and IOPS limiting) per VM, and so on… It works, but it is worse than develop a storage driver directly for your storage system.
Hi,
@feldsam, Thank you for your input. Our implementation is very application specific and we are not looking for limiting the access to storage. Those are very specific VMs, which we are very much familiar with.
@ahuertas, Thank you for response, I was able to figure that part out, but I am having other issues. I have shared 20 TB VG ( currently VG_NAME = vg-one-107) storage. To build it I have followed the document you have mentioned(use_lvmetad = 0). I am looking to have one Datastore with some amount of persistent images and VMs using those images from that storage without cloning on initiation. I have build following template for Datastore:
NAME = AmsMainDataStore
DISK_TYPE = BLOCK
DS_MAD = lvm
TM_MAD = fs_lvm
TYPE = IMAGE_DS
VG_NAME = vg-one-107
LN_TARGET = SELF
CLONE_TARGET = SELF
BRIDGE_LIST = “amscn01 amscn02 amscn03 amscn04 amscn05 amscn06 amscn07 amscn08”
I think that I need to use DS_MAD as lvm, but I do not have lvm scripts in ~/remotes/datastore/
I have also changed in /etc/one/oned.conf
TM_MAD_CONF = [
NAME = “fs_lvm”, LN_TARGET = “SELF”, CLONE_TARGET = “SELF”, SHARED=“YES”,
DRIVER = “raw”
]
Any advice will be highly appreciated.
Best regards
Garry
Hi, in past I was using LVM block storage, somethign simillar to your setup and I used this block lvm storage driver.
It was part of OpenNebula in past, but dev teams removes it and replaced by fs_lvm.
If you want use block lvm, use that driver and look on this https://www.systutorials.com/docs/linux/man/8-lvmlockd/
it is shared lvm with use of sanlock and you dont need to run corosync/pacemaker and cLVM.
EDIT: lvmlockd also supports lvm thin provisioning and snapshots.
@feldsam Thank you. It is very “painful” topic, we are using Debian 9, for hosts. lvmlockd is not supported in “stretch” only in “buster/testing”. cman also not in “strech”. Also corosync, clvm and pcs are in conflict with opennebula 5.8 packages. I have few instances deployed with opennebula 4.14 and clvm/cman for few years now.
I can recommend you Oracle Linux 7, it is free and is better that CentOS 7. In past years, we use Centos, fedora, also tried debian stuff, but OL7 is best.
EDIT: my offer is still valid, we can help you with your setup 
Hello @Morpheus
Your DS_MAD as you said should be inside /var/lib/one/remotes/datastore. There is no lvm for DS_MAD, so you can leave it blank and it should work.
Hi @ahuertas,
Thank you for response. Unfortunately it requires the DS_MAD. I am getting following error:
[one.datastore.allocate] No DS_MAD in template.
Template:
NAME = AmsMainDataStore
DISK_TYPE = BLOCK
TM_MAD = fs_lvm
TYPE = IMAGE_DS
VG_NAME = vg-one-108
BRIDGE_LIST = “amscn01 amscn02 amscn03 amscn04 amscn05 amscn06 amscn07 amscn08”
Hello @Morpheus
Sorry I meant using - as blank value.
@ahuertas, sorry, but it doesn’t work. I have tried different variations without success.
Try using DS_MAD=fs
Unfortunately it does not work.