Unable to create VMs ("failed to execute virtualization driver operation: deploy")

Community Support
1

I have searched for hours for many (many) different variations of the errors I am seeing in /var/log/one/oned.log. I have found many solutions pertaining to each issue I have read, but none seem to address my issue specifically.

Here is my setup:

  • Control Node: Has sunstone, mysql, etc
  • Compute Node: Has libvirt, opennebula-node-kvm
  • Storage: GlusterFS 2 node cluster

(I moved away from an openstack installation and into opennebula, hence “compute” and “control”).

It was an uphill battle getting gluster to work. I ended up following a series of docs and forum posts to get mine to work “properly” (not entirely sure if it is or not yet). For the sake of anyone asking for my storage config, here they are:

  ID NAME                SIZE AVAIL CLUSTERS     IMAGES TYPE DS      TM      STAT
 110 system              5.5T 100%  0                 0 sys  -       shared  on  
 111 images              5.5T 100%  0                 2 img  fs      shared  on  
 112 files               5.5T 100%  0                 0 fil  fs      shared  on  

DATASTORE 110 INFORMATION                                                       
ID             : 110                 
NAME           : system              
USER           : oneadmin            
GROUP          : oneadmin            
CLUSTERS       : 0                   
TYPE           : SYSTEM              
DS_MAD         : -                   
TM_MAD         : shared              
BASE PATH      : /var/lib/one//datastores/110
DISK_TYPE      : FILE                
STATE          : READY               

PERMISSIONS                                                                     
OWNER          : um-                 
GROUP          : u--                 
OTHER          : ---                 

DATASTORE TEMPLATE                                                              
DISK_TYPE="FILE"
DS_MIGRATE="YES"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
SHARED="YES"
TM_MAD="shared"
TYPE="SYSTEM_DS"

DATASTORE 111 INFORMATION                                                       
ID             : 111                 
NAME           : images              
USER           : oneadmin            
GROUP          : oneadmin            
CLUSTERS       : 0                   
TYPE           : IMAGE               
DS_MAD         : fs                  
TM_MAD         : shared              
BASE PATH      : /var/lib/one//datastores/111
DISK_TYPE      :                     
STATE          : READY                               

PERMISSIONS                                                                     
OWNER          : um-                 
GROUP          : u--                 
OTHER          : ---                 

DATASTORE TEMPLATE                                                              
CLONE_TARGET="SYSTEM"
DISK_TYPE="GLUSTER"
DS_MAD="fs"
GLUSTER_HOST="gluster1:24007"
GLUSTER_VOLUME="volumename"
LN_TARGET="NONE"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
TM_MAD="shared"
TYPE="IMAGE_DS"

DATASTORE 112 INFORMATION                                                       
ID             : 112                 
NAME           : files               
USER           : oneadmin            
GROUP          : oneadmin            
CLUSTERS       : 0                   
TYPE           : FILE                
DS_MAD         : fs                  
TM_MAD         : shared              
BASE PATH      : /var/lib/one//datastores/112
DISK_TYPE      : FILE                
STATE          : READY                            

PERMISSIONS                                                                     
OWNER          : um-                 
GROUP          : u--                 
OTHER          : ---                 

DATASTORE TEMPLATE                                                              
CLONE_TARGET="SYSTEM"
DS_MAD="fs"
GLUSTER_HOST="gluster1:24007"
GLUSTER_VOLUME="volumename"
LN_TARGET="NONE"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
TM_MAD="shared"
TYPE="FILE_DS"

So now we’ve got that out of the way, let’s take a look at my logs. This is an excerpt from a VM creation with a CentOS x86_64 template from OpenNebula “apps” area. I also tested with “ttylinux” and qcow2c image from CentOS repos.

Thu Jan 26 19:43:19 2017 [Z0][VMM][D]: Message received: LOG I 16 Successfully execute transfer manager driver operation: tm_context.
Thu Jan 26 19:43:19 2017 [Z0][VMM][D]: Message received: LOG I 16 Successfully execute network driver operation: pre.
Thu Jan 26 19:43:20 2017 [Z0][VMM][D]: Message received: LOG I 16 Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy '/var/lib/one//datastores/110/16/deployment.0' 'compute1' 16 compute1
Thu Jan 26 19:43:20 2017 [Z0][VMM][D]: Message received: LOG I 16 error: Failed to create domain from /var/lib/one//datastores/110/16/deployment.0
Thu Jan 26 19:43:20 2017 [Z0][VMM][D]: Message received: LOG I 16 error: Cannot access storage file '/622fda14e7c896865adf8175b3ab1b01' (as uid:9869, gid:9869): No such file or directory
Thu Jan 26 19:43:20 2017 [Z0][VMM][D]: Message received: LOG E 16 Could not create domain from /var/lib/one//datastores/110/16/deployment.0
Thu Jan 26 19:43:20 2017 [Z0][VMM][D]: Message received: LOG I 16 ExitCode: 255
Thu Jan 26 19:43:20 2017 [Z0][VMM][D]: Message received: LOG I 16 Failed to execute virtualization driver operation: deploy.
Thu Jan 26 19:43:20 2017 [Z0][VMM][D]: Message received: DEPLOY FAILURE 16 Could not create domain from /var/lib/one//datastores/110/16/deployment.0

Where I believe the problem is occurring is with the images themselves (not a bad image, but an issue with one of the servers/protocols handling the image). The reason I say this is because I can create a “blank” VM (no image just an attached disk, gives “no bootable disk found” on VNC) and it deploys fine. I believe this to be enough proof that the compute (kvm) node is setup correctly. But of course, I could be wrong.

I have verified that oneadmin can reach the datastores, and has access to the datastores (read and write). I have torn down my gluster installation, removed all bricks/volumes, and recreated it. I have torn down my compute installation, removed libvirtd and opennebula, and reinstalled them.

Any ideas anyone? Please help :disappointed_relieved:

2

it look like that issue in image pass '/622fda14e7c896865adf8175b3ab1b01 it should be something like /var/lib/one/datastores/111
try to modify from web sunstone

3

I don’t think so. I am first trying to tackle this problem:

Thu Jan 26 19:43:20 2017 [Z0][VMM][D]: Message received: LOG I 16 Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy '/var/lib/one//datastores/110/16/deployment.0' 'compute1' 16 compute1

I believe that the inability to deploy the VM consequently causes the rest of the process to fail. Also, this error has appeared in many different situations for many other people. My specific issue is different, and cannot be solved using the solutions to the other problems (I have tried).

4

if your sure you can touch and access your storage from fronend than it could be apparmor in your node aa-status it find if apparmor make the problem

ls -l /var/lib/one/datastores/110/622fda14e7c896865adf8175b3ab1b01
qemu-img info /var/lib/one/datastores/110/622fda14e7c896865adf8175b3ab1b01

5

I do not have apparmor installed.

6

and wt about the ls and qemu-img command

7

I would appreciate more in depth solutions on how to fix this error

 Cannot access storage file '/622fda14e7c896865adf8175b3ab1b01' (as uid:9869, gid:9869): No such file or directory

Again:

  • I do not have apparmor.
  • I do not have vmware.
  • My datastores show no sign that there is an incorrectly configured path.
  • I am running Centos7
  • I am running a gluster storage backend.

Please keep replies helpful, and not nonsensical. Thanks.

8

no one say anything about vmware
and your datastores has this

9

What?

(requires me to put at least 10 characters, but you have made 0 sense).

10

oh good u know about sense
that count as 10 characters

11

It is clear you do not have any understanding of the issue at hand, nor are you interested in providing constructive solutions. I kindly ask that you leave this question alone so that someone more experienced may come along.

12

before looking for experience learn how to answer

13

I believe it is you who must first learn how to answer :wink:

14

i will answer with this (sorry)

15

Perpetually unhelpful. Thanks!

16

all smart like you talk like you

17

Calm down guys. I think we all want to help here and I appreciate it.

The problem you’re experiencing is caused because you’re using gluster drivers:

Gluster is not supported since release 5.0. We decided to leave the functionality in deployment file generation so it could be used with drivers written by the users.

You can still use glusterfs as backed but with shared or qcow2 drivers. I recommend you to use qcow2 drivers:

http://docs.opennebula.org/5.2/deployment/open_cloud_storage_setup/fs_ds.html

Delete this lines from images datastore:

DISK_TYPE="GLUSTER"
GLUSTER_HOST="gluster1:24007"
GLUSTER_VOLUME="volumename"

If that works then change TM_MAD in both datastores to qcow2. Also make sure that the images have DRIVER=qcow2 or the default driver is set to qcow2:

http://docs.opennebula.org/5.2/deployment/open_cloud_host_setup/kvm_driver.html?highlight=vmm_exec_kvm#driver-defaults

2 Likes
18

Thank you very much. I will try this shortly when I get a chance.

There should certainly be guidelines for writing answers (much like stack exchange has) to avoid situations such as this.

Will update this with new information when I get it.

19

i change the stack but u start

20

I was able to deploy a virtual machine using this information. Thank you again. As an end user, might I suggest placing this note somewhere in the official documentation? This is the first time I have actually seen these instructions.

Thank you.