Front-end cannot see system datastore mounted on KVM host

First of all, I have a lot of experience, but not with the storage. This is my biggest pain point while doing OpenNebula test. It seems to me that the manual is written for the storage guys, I found it hard to just plainly make a simple setup. Any help is appreciated.
The setup is simple. One front-end and one KVM host, both running centos 7. Installation went just fine until I reached the point of setting up system datastore. My kvm host has SAN based storage, connected via multipath and this is all fine. I have a “device” mpathb which is ready for mounting. OK.
Currently I have three automatically created datastores. All residing on the front-end. ID=0 is for the system and it is displaying capacity of ZERO. Strange since is is on the same place as OpenNebula. We don’t need this anyway since this is a boot disk for the OS. ID=1 is for the image ds, and ID=2 for the files. OK.

This is my kvm host as seen in the front-end:
1 csesx5 default 1 10 / 3200 (0%) 128M / 755.6G (0%) on

(don’t be confused with the hostname, it is our lab environment)

Now I wanted to add another system ds which is connected to my kvm host:

  • I have mounted my 2TB disk with
    mount /dev/mapper/mpathb /var/lib/one/datastores/101
  • Then I have checked the situation with

DS = [
ID = 0,
USED_MB = 1199,
TOTAL_MB = 17878,
FREE_MB = 16680
DS = [
ID = 101,
USED_MB = 81,
TOTAL_MB = 2064108,
FREE_MB = 1959155

As you can see ID=101 is properly seen, but I am a bit confused with the first part “DS_LOCATION” as it only references ID=0, like my newly added 101 is not parsed at all.

  • After that I have created a new datastore on the front-end with the same ID and executed
    onedatastore list
    0 system - - 0 0 sys - ssh on
    1 default 50G 91% 0 1 img fs ssh on
    2 files 50G 91% 0 0 fil fs ssh on
    101 kvm-ds - - 0 0 sys - ssh on

As you can see OpenNebula still cannot use it. I tried restarting the process but nothing new happened. Size/Avail is always at 0…

I guess I am doing something wrong and it’s because I do not understand the whole process. How is OpenNebula supposed to notice what is going on at the kvm host? How does it see the creation or deletion of the datastores on the host? What piece of the puzzle am I missing?


Replying to myself… One addition. I can see that the front-end can see the datastore through the host:
onehost show 1
ID : 1
NAME : csesx5
CLUSTER : default
IM_MAD : kvm
VM_MAD : kvm
LAST MONITORING TIME : 10/03 15:45:48

TOTAL MEM : 755.6G
TOTAL CPU : 3200

TOTAL: : 17.5G
USED: : 1.2G
FREE: : 16.3G

USED: : 81M
FREE: : 1.9T

But still I cannot find a way to use it as a system datastore. This is my main question now…

maybe I’m wrong but I think all is ok. My system datastore 0, in a fresh installation shows 0 as well. Have you tried to run a VM?

The system datastore as you may already know it’s the place where the VM image is moved from the image datastore so the hypervisor can run the VM. Though the system datastores may look 0 or - size that would be because you haven’t still run a VM. Once a VM is instantiated in datastore 0 or datastore 101 the real SIZE is known and it will be modified so it won’t be - anymore. As the onedatastore list says both system datastore are ON in the status column so they are ready to use.

Also as both system datastore are configured as TYPE SSH, scp will be used to transfer images from the image datastore to the system datastores.

Again, maybe I’m wrong but I’d try to instantiate a VM and have a look again.


OK, but how can I instruct OpenNebula to use 101 datastore only? I have created a system datastore with ID=101. Is that the proper way?
In other words, is there a correlation between datastore ID on the host, and the datastore ID on the front-end?

Interesting, I have tried to instantiate a VM as you suggested and it did chose the 101 datastore. It failed though because of the permission problems. But I am still not at ease because I don’t see a clear mechanism on how does ONE chooses the appropriate system datastore.

Error executing image transfer script: Error creating directory
/var/lib/one/datastores/101/1 at csesx5: mkdir: cannot create directory
’/var/lib/one/datastores/101/1’: Permission denied

Hi again.
I think the answer you need is here:

For example with SCHED_DS_REQUIREMENTS=“ID=101” declared in your VM template you can tell OpenNebula’s scheduler what datastore to use. By default all datastores you create are members of the default cluster (ID=0)


Thanks a lot.

Currently my template does not have this set, but the system uses 101 anyway for some reason. Anyhow, it’s great to know I can force it!

BTW the permission problem was because oneadmin user was not allowed to write… But now I have transfer problems… :slight_smile:
Error executing image transfer script: Error copying
to csesx5:/var/lib/one//datastores/101/2/disk.0

Bit by bit I will instantiate…

As I’ve no previous experience with SAN storage in my environments I apologize if my answers are not accurate. I’m sure you’ll get it running.

About your copy error, as you mentioned that you’re using CentOS 7, is SELinux enabled? If not I guess you’ve checked that you can write flawlessly in the /var/lib/one/datastores/101 folder… sorry those are my usual troubleshooting steps :slight_smile:

And I’d like to correct myself on my first answer, as Carlos answered here (Ssh system datastore capacity) when using SSH datastores: “the capacity is supposed to be inside each one of the host information (host tab, or onehost show). This is because each host will have a different local storage.” So you were right using onehost to show datastores capacity.


Yep, selinux is disabled. I have also put oneadmin public key in there, as per user guide. After I have changed ownership for datastores/101 directory, I can write there no problems. So the first error dissapeared, no I have this transfer error instead.
Never mind, thanks a lot for your help, you have jumpstarted me when you suggested it is fine now. :slight_smile:

I will try to see why the transfer failed. Should be something with ssh but again I have put ssh key in there, hmmm…

DNS name not resolved for scp copying…
It’s a lab environment, many little things to be taken care of. :slight_smile:

Yes I know :smile: a lab is a lab!
I’m sure you’ll get it working, if not here’s a community to give you support.


Yep, sure,

now I have “failed: Host key verification failed” LOL… Saga continues. :smiley:

Well that’s quite common, don’t worry, you have to check ssh works well without password (check

As you’ve already copied the ssh pub keys, from the opennebula frontend try to:

  1. su - oneadmin
  2. ssh oneadmin@name_of_your_frontend, check if no password is needed and exit
  3. ssh oneadmin@name_of_your_kvm_node, check if no password is needed and exit


Thanks. Continuing with another post, I have recreated known_hosts and still the same message appears. DOH


you have solved my problem while you were working to help someone else. :slight_smile:

Just wanted to start a new thread when I saw your answer about scp problem. When I understood that scp requires local host key as well, I did ssh to the local hostname, stored the key in the cache (known_hosts) and from that point my VM cloning to kvm host works fine. Phew. :slight_smile:

I know this is scp basics, but when someone is not using linux on a daily basis it’s not that obvious. Thanks man, you’ve been of great help!

Awesome! I was answering you right now and I’ve seen you’ve solved it.

The community is here to help so I’m really happy… Have a lot of fun with your lab!