First of all, I have a lot of experience, but not with the storage. This is my biggest pain point while doing OpenNebula test. It seems to me that the manual is written for the storage guys, I found it hard to just plainly make a simple setup. Any help is appreciated.
The setup is simple. One front-end and one KVM host, both running centos 7. Installation went just fine until I reached the point of setting up system datastore. My kvm host has SAN based storage, connected via multipath and this is all fine. I have a “device” mpathb which is ready for mounting. OK.
Currently I have three automatically created datastores. All residing on the front-end. ID=0 is for the system and it is displaying capacity of ZERO. Strange since is is on the same place as OpenNebula. We don’t need this anyway since this is a boot disk for the OS. ID=1 is for the image ds, and ID=2 for the files. OK.
This is my kvm host as seen in the front-end:
ID NAME CLUSTER RVM ALLOCATED_CPU ALLOCATED_MEM STAT
1 csesx5 default 1 10 / 3200 (0%) 128M / 755.6G (0%) on
(don’t be confused with the hostname, it is our lab environment)
Now I wanted to add another system ds which is connected to my kvm host:
I have mounted my 2TB disk with mount /dev/mapper/mpathb /var/lib/one/datastores/101
Then I have checked the situation with /var/tmp/one/im/kvm-probes.d/monitor_ds.sh
As you can see ID=101 is properly seen, but I am a bit confused with the first part “DS_LOCATION” as it only references ID=0, like my newly added 101 is not parsed at all.
After that I have created a new datastore on the front-end with the same ID and executed onedatastore list
ID NAME SIZE AVAIL CLUSTERS IMAGES TYPE DS TM STAT
0 system - - 0 0 sys - ssh on
1 default 50G 91% 0 1 img fs ssh on
2 files 50G 91% 0 0 fil fs ssh on
101 kvm-ds - - 0 0 sys - ssh on
As you can see OpenNebula still cannot use it. I tried restarting the process but nothing new happened. Size/Avail is always at 0…
I guess I am doing something wrong and it’s because I do not understand the whole process. How is OpenNebula supposed to notice what is going on at the kvm host? How does it see the creation or deletion of the datastores on the host? What piece of the puzzle am I missing?
Replying to myself… One addition. I can see that the front-end can see the datastore through the host: onehost show 1
HOST 1 INFORMATION
ID : 1
NAME : csesx5
CLUSTER : default
STATE : MONITORED
IM_MAD : kvm
VM_MAD : kvm
LAST MONITORING TIME : 10/03 15:45:48
HOST SHARES
TOTAL MEM : 755.6G
USED MEM (REAL) : 4.9G
USED MEM (ALLOCATED) : 128M
TOTAL CPU : 3200
USED CPU (REAL) : 0
USED CPU (ALLOCATED) : 10
RUNNING VMS : 1
LOCAL SYSTEM DATASTORE #0 CAPACITY
TOTAL: : 17.5G
USED: : 1.2G
FREE: : 16.3G
LOCAL SYSTEM DATASTORE #101 CAPACITY
TOTAL: : 2T
USED: : 81M
FREE: : 1.9T
But still I cannot find a way to use it as a system datastore. This is my main question now…
Hi!
maybe I’m wrong but I think all is ok. My system datastore 0, in a fresh installation shows 0 as well. Have you tried to run a VM?
The system datastore as you may already know it’s the place where the VM image is moved from the image datastore so the hypervisor can run the VM. Though the system datastores may look 0 or - size that would be because you haven’t still run a VM. Once a VM is instantiated in datastore 0 or datastore 101 the real SIZE is known and it will be modified so it won’t be - anymore. As the onedatastore list says both system datastore are ON in the status column so they are ready to use.
Also as both system datastore are configured as TYPE SSH, scp will be used to transfer images from the image datastore to the system datastores.
Again, maybe I’m wrong but I’d try to instantiate a VM and have a look again.
OK, but how can I instruct OpenNebula to use 101 datastore only? I have created a system datastore with ID=101. Is that the proper way?
In other words, is there a correlation between datastore ID on the host, and the datastore ID on the front-end?
Interesting, I have tried to instantiate a VM as you suggested and it did chose the 101 datastore. It failed though because of the permission problems. But I am still not at ease because I don’t see a clear mechanism on how does ONE chooses the appropriate system datastore.
Error executing image transfer script: Error creating directory
/var/lib/one/datastores/101/1 at csesx5: mkdir: cannot create directory
’/var/lib/one/datastores/101/1’: Permission denied
For example with SCHED_DS_REQUIREMENTS=“ID=101” declared in your VM template you can tell OpenNebula’s scheduler what datastore to use. By default all datastores you create are members of the default cluster (ID=0)
Currently my template does not have this set, but the system uses 101 anyway for some reason. Anyhow, it’s great to know I can force it!
BTW the permission problem was because oneadmin user was not allowed to write… But now I have transfer problems…
Error executing image transfer script: Error copying
nebula.localdomain:/var/lib/one//datastores/1/eb2ea40361ad180152f884e41644b34c
to csesx5:/var/lib/one//datastores/101/2/disk.0
Hi!
As I’ve no previous experience with SAN storage in my environments I apologize if my answers are not accurate. I’m sure you’ll get it running.
About your copy error, as you mentioned that you’re using CentOS 7, is SELinux enabled? If not I guess you’ve checked that you can write flawlessly in the /var/lib/one/datastores/101 folder… sorry those are my usual troubleshooting steps
And I’d like to correct myself on my first answer, as Carlos answered here (Ssh system datastore capacity) when using SSH datastores: “the capacity is supposed to be inside each one of the host information (host tab, or onehost show). This is because each host will have a different local storage.” So you were right using onehost to show datastores capacity.
Yep, selinux is disabled. I have also put oneadmin public key in there, as per user guide. After I have changed ownership for datastores/101 directory, I can write there no problems. So the first error dissapeared, no I have this transfer error instead.
Never mind, thanks a lot for your help, you have jumpstarted me when you suggested it is fine now.
I will try to see why the transfer failed. Should be something with ssh but again I have put ssh key in there, hmmm…
you have solved my problem while you were working to help someone else.
Just wanted to start a new thread when I saw your answer about scp problem. When I understood that scp requires local host key as well, I did ssh to the local hostname, stored the key in the cache (known_hosts) and from that point my VM cloning to kvm host works fine. Phew.
I know this is scp basics, but when someone is not using linux on a daily basis it’s not that obvious. Thanks man, you’ve been of great help!
a lab is a lab!