Front-end cannot see system datastore mounted on KVM host

First of all, I have a lot of experience, but not with the storage. This is my biggest pain point while doing OpenNebula test. It seems to me that the manual is written for the storage guys, I found it hard to just plainly make a simple setup. Any help is appreciated.
The setup is simple. One front-end and one KVM host, both running centos 7. Installation went just fine until I reached the point of setting up system datastore. My kvm host has SAN based storage, connected via multipath and this is all fine. I have a “device” mpathb which is ready for mounting. OK.
Currently I have three automatically created datastores. All residing on the front-end. ID=0 is for the system and it is displaying capacity of ZERO. Strange since is is on the same place as OpenNebula. We don’t need this anyway since this is a boot disk for the OS. ID=1 is for the image ds, and ID=2 for the files. OK.

This is my kvm host as seen in the front-end:
ID NAME CLUSTER RVM ALLOCATED_CPU ALLOCATED_MEM STAT
1 csesx5 default 1 10 / 3200 (0%) 128M / 755.6G (0%) on

(don’t be confused with the hostname, it is our lab environment)

Now I wanted to add another system ds which is connected to my kvm host:

  • I have mounted my 2TB disk with
    mount /dev/mapper/mpathb /var/lib/one/datastores/101
  • Then I have checked the situation with
    /var/tmp/one/im/kvm-probes.d/monitor_ds.sh

DS_LOCATION_USED_MB=1199
DS_LOCATION_TOTAL_MB=17878
DS_LOCATION_FREE_MB=16680
DS = [
ID = 0,
USED_MB = 1199,
TOTAL_MB = 17878,
FREE_MB = 16680
]
DS = [
ID = 101,
USED_MB = 81,
TOTAL_MB = 2064108,
FREE_MB = 1959155
]

As you can see ID=101 is properly seen, but I am a bit confused with the first part “DS_LOCATION” as it only references ID=0, like my newly added 101 is not parsed at all.

  • After that I have created a new datastore on the front-end with the same ID and executed
    onedatastore list
    ID NAME SIZE AVAIL CLUSTERS IMAGES TYPE DS TM STAT
    0 system - - 0 0 sys - ssh on
    1 default 50G 91% 0 1 img fs ssh on
    2 files 50G 91% 0 0 fil fs ssh on
    101 kvm-ds - - 0 0 sys - ssh on

As you can see OpenNebula still cannot use it. I tried restarting the process but nothing new happened. Size/Avail is always at 0…

I guess I am doing something wrong and it’s because I do not understand the whole process. How is OpenNebula supposed to notice what is going on at the kvm host? How does it see the creation or deletion of the datastores on the host? What piece of the puzzle am I missing?

THANKS

Replying to myself… One addition. I can see that the front-end can see the datastore through the host:
onehost show 1
HOST 1 INFORMATION
ID : 1
NAME : csesx5
CLUSTER : default
STATE : MONITORED
IM_MAD : kvm
VM_MAD : kvm
LAST MONITORING TIME : 10/03 15:45:48

HOST SHARES
TOTAL MEM : 755.6G
USED MEM (REAL) : 4.9G
USED MEM (ALLOCATED) : 128M
TOTAL CPU : 3200
USED CPU (REAL) : 0
USED CPU (ALLOCATED) : 10
RUNNING VMS : 1

LOCAL SYSTEM DATASTORE #0 CAPACITY
TOTAL: : 17.5G
USED: : 1.2G
FREE: : 16.3G

LOCAL SYSTEM DATASTORE #101 CAPACITY
TOTAL: : 2T
USED: : 81M
FREE: : 1.9T

But still I cannot find a way to use it as a system datastore. This is my main question now…

Hi!
maybe I’m wrong but I think all is ok. My system datastore 0, in a fresh installation shows 0 as well. Have you tried to run a VM?

The system datastore as you may already know it’s the place where the VM image is moved from the image datastore so the hypervisor can run the VM. Though the system datastores may look 0 or - size that would be because you haven’t still run a VM. Once a VM is instantiated in datastore 0 or datastore 101 the real SIZE is known and it will be modified so it won’t be - anymore. As the onedatastore list says both system datastore are ON in the status column so they are ready to use.

Also as both system datastore are configured as TYPE SSH, scp will be used to transfer images from the image datastore to the system datastores.

Again, maybe I’m wrong but I’d try to instantiate a VM and have a look again.

Cheers!

OK, but how can I instruct OpenNebula to use 101 datastore only? I have created a system datastore with ID=101. Is that the proper way?
In other words, is there a correlation between datastore ID on the host, and the datastore ID on the front-end?

Interesting, I have tried to instantiate a VM as you suggested and it did chose the 101 datastore. It failed though because of the permission problems. But I am still not at ease because I don’t see a clear mechanism on how does ONE chooses the appropriate system datastore.

Error executing image transfer script: Error creating directory
/var/lib/one/datastores/101/1 at csesx5: mkdir: cannot create directory
’/var/lib/one/datastores/101/1’: Permission denied

Hi again.
I think the answer you need is here: http://docs.opennebula.org/5.0/operation/host_cluster_management/datastore_guide.html

For example with SCHED_DS_REQUIREMENTS=“ID=101” declared in your VM template you can tell OpenNebula’s scheduler what datastore to use. By default all datastores you create are members of the default cluster (ID=0)

Cheers

Thanks a lot.

Currently my template does not have this set, but the system uses 101 anyway for some reason. Anyhow, it’s great to know I can force it!

BTW the permission problem was because oneadmin user was not allowed to write… But now I have transfer problems… :slight_smile:
Error executing image transfer script: Error copying
nebula.localdomain:/var/lib/one//datastores/1/eb2ea40361ad180152f884e41644b34c
to csesx5:/var/lib/one//datastores/101/2/disk.0

Bit by bit I will instantiate…

Hi!
As I’ve no previous experience with SAN storage in my environments I apologize if my answers are not accurate. I’m sure you’ll get it running.

About your copy error, as you mentioned that you’re using CentOS 7, is SELinux enabled? If not I guess you’ve checked that you can write flawlessly in the /var/lib/one/datastores/101 folder… sorry those are my usual troubleshooting steps :slight_smile:

And I’d like to correct myself on my first answer, as Carlos answered here (Ssh system datastore capacity) when using SSH datastores: “the capacity is supposed to be inside each one of the host information (host tab, or onehost show). This is because each host will have a different local storage.” So you were right using onehost to show datastores capacity.

Cheers!

Yep, selinux is disabled. I have also put oneadmin public key in there, as per user guide. After I have changed ownership for datastores/101 directory, I can write there no problems. So the first error dissapeared, no I have this transfer error instead.
Never mind, thanks a lot for your help, you have jumpstarted me when you suggested it is fine now. :slight_smile:

I will try to see why the transfer failed. Should be something with ssh but again I have put ssh key in there, hmmm…

DNS name not resolved for scp copying…
It’s a lab environment, many little things to be taken care of. :slight_smile:

Yes I know :smile: a lab is a lab!
I’m sure you’ll get it working, if not here’s a community to give you support.

Cheers!

Yep, sure,

now I have “failed: Host key verification failed” LOL… Saga continues. :smiley:

Well that’s quite common, don’t worry, you have to check ssh works well without password (check http://docs.opennebula.org/5.0/deployment/node_installation/kvm_node_installation.html#step-4-configure-passwordless-ssh).

As you’ve already copied the ssh pub keys, from the opennebula frontend try to:

  1. su - oneadmin
  2. ssh oneadmin@name_of_your_frontend, check if no password is needed and exit
  3. ssh oneadmin@name_of_your_kvm_node, check if no password is needed and exit

Cheers!

Thanks. Continuing with another post, I have recreated known_hosts and still the same message appears. DOH

Miguel,

you have solved my problem while you were working to help someone else. :slight_smile:

Just wanted to start a new thread when I saw your answer about scp problem. When I understood that scp requires local host key as well, I did ssh to the local hostname, stored the key in the cache (known_hosts) and from that point my VM cloning to kvm host works fine. Phew. :slight_smile:

I know this is scp basics, but when someone is not using linux on a daily basis it’s not that obvious. Thanks man, you’ve been of great help!

Awesome! I was answering you right now and I’ve seen you’ve solved it.

The community is here to help so I’m really happy… Have a lot of fun with your lab!