Can't create VM from "old" images

girg · April 12, 2021, 3:41pm

Hi,
we recently upgraded from 5.12 to 6.0 and as a result we can’t create VMs from our images.

Mon Apr 12 17:20:21 2021 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy ‘/var/lib/one//datastores/103/1208/deployment.0’ ‘virt02’ 1208 virt02
Mon Apr 12 17:20:21 2021 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/103/1208/deployment.0
Mon Apr 12 17:20:21 2021 [Z0][VMM][I]: error: internal error: qemu unexpectedly closed the monitor: 2021-04-12T15:20:21.327420Z qemu-kvm-one: -drive file=rbd:one/one-554-1208-0:id=libvirt:auth_supported=cephx;none:mon_host=mon01:6789;mon02:6789;mon03:6789,file.password-secret=scsi0-0-0-0-secret0,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=none,discard=unmap: Image is not in qcow2 format
Mon Apr 12 17:20:21 2021 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/103/1208/deployment.0
Mon Apr 12 17:20:21 2021 [Z0][VMM][I]: ExitCode: 255

oneimage show tells me the format is qcow2.

I can create VMs from a fresh downloaded image from the marketplace… Also curious is that the format shown in the marketplace is qcow2 but when I do oneimage show on that newly createt image, it says that it raw…

I am grateful for any advice.

girg · April 13, 2021, 8:34am

here is the deployment.0 file:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>one-1212</name>
<title>test123123</title>
<uuid>035c63be-51ef-4e11-bf30-7e8f3ba930f6</uuid>
<cputune>
	<shares>1024</shares>
</cputune>
<memory>4194304</memory>
<os>
	<type arch='x86_64'>hvm</type>
</os>
<cpu mode='host-passthrough'>
</cpu>
<devices>
	<emulator><![CDATA[/usr/bin/qemu-kvm-one]]></emulator>
	<disk type='network' device='disk'>
		<source protocol='rbd' name='one/one-554-1212-0'>
			<host name='mon01' port='6789'/>
			<host name='mon02' port='6789'/>
			<host name='mon03' port='6789'/>
		</source>
		<auth username='libvirt'>
			<secret type='ceph' uuid='blablabla-325b-5c10-a349-blabla'/>
		</auth>
		<target dev='sda'/>
		<driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
		<address type='drive' controller='0' bus='0' target='0' unit='0'/>
	</disk>
	<disk type='file' device='cdrom'>
		<source file='/var/lib/one//datastores/103/1212/disk.1'/>
		<target dev='hda' bus='ide'/>
		<readonly/>
		<driver name='qemu' type='raw'/>
	</disk>
	<interface type='bridge'>
		<source bridge='onebr.56'/>
		<mac address='02:00:3e:71:c5:18'/>
		<target dev='one-1212-0'/>
		<model type='virtio'/>
		<bandwidth>
			<inbound average='100000' peak='125000'/>
			<outbound average='100000' peak='125000'/>
		</bandwidth>
	</interface>
	<graphics type='vnc' listen='0.0.0.0' port='7112'/>
</devices>
<features>
	<acpi/>
</features>
<clock offset='localtime'/>
<devices>
	<channel type='unix'>
		<source mode='bind'/><target type='virtio' name='org.qemu.guest_agent.0'/>
	</channel>
</devices>
<devices>
	<controller type='scsi' index='0' model='virtio-scsi'>
		<driver queues='1'/>
	</controller>
</devices>
<metadata>
	<one:vm xmlns:one="http://opennebula.org/xmlns/libvirt/1.0">
		<one:system_datastore><![CDATA[/var/lib/one//datastores/103/1212]]></one:system_datastore>
		<one:name><![CDATA[test123123]]></one:name>
		<one:uname><![CDATA[itwasme]]></one:uname>
		<one:uid>56</one:uid>
		<one:gname><![CDATA[oneadmin]]></one:gname>
		<one:gid>0</one:gid>
		<one:opennebula_version>6.0.0</one:opennebula_version>
		<one:stime>1618302143</one:stime>
		<one:deployment_time>1618302156</one:deployment_time>
	</one:vm>
</metadata>

atodorov_storpool · April 13, 2021, 8:51am

Hi Fabian,

Could you provide the OpenNebula’s VM XML too? It is the output of onevm show 1212 --xml

cgonzalez · April 13, 2021, 9:12am

Hello @girg,

We did rework that part of the code for 6.0 in order to add some improvements. In order to help us figure out what’s exactly the problem affecting you it would be nice if, apart from what @atodorov_storpool asked you, you could send us:

The XML of the image DS where the image is stored (onedatastore show -x <img_ds_id>)
The XML of the image itself (oneimage show -x <img-id>)
The output of file command for the image file. You can get the image path by running oneimage show <img_id> | grep -i source.
The output of file command for the image in the system DS (i.e file /var/lib/one/datastores/<sys_ds_id>/<vm_id>/disk.<disk_id>).

girg · April 13, 2021, 9:24am

Hi Anton,
here the output of onevm show 1212 --xml

<VM>
      <ID>1212</ID>
      <UID>56</UID>
      <GID>0</GID>
      <UNAME>itwasme</UNAME>
      <GNAME>oneadmin</GNAME>
      <NAME>test123123</NAME>
      <PERMISSIONS>
        <OWNER_U>1</OWNER_U>
        <OWNER_M>1</OWNER_M>
        <OWNER_A>0</OWNER_A>
        <GROUP_U>0</GROUP_U>
        <GROUP_M>0</GROUP_M>
        <GROUP_A>0</GROUP_A>
        <OTHER_U>0</OTHER_U>
        <OTHER_M>0</OTHER_M>
        <OTHER_A>0</OTHER_A>
      </PERMISSIONS>
      <LAST_POLL>0</LAST_POLL>
      <STATE>3</STATE>
      <LCM_STATE>36</LCM_STATE>
      <PREV_STATE>3</PREV_STATE>
      <PREV_LCM_STATE>36</PREV_LCM_STATE>
      <RESCHED>0</RESCHED>
      <STIME>1618302143</STIME>
      <ETIME>0</ETIME>
      <DEPLOY_ID/>
      <MONITORING/>
      <TEMPLATE>
        <AUTOMATIC_DS_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_DS_REQUIREMENTS>
        <AUTOMATIC_NIC_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_NIC_REQUIREMENTS>
        <AUTOMATIC_REQUIREMENTS><![CDATA[(CLUSTER_ID = 0) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED)]]></AUTOMATIC_REQUIREMENTS>
        <CONTEXT>
          <DISK_ID><![CDATA[1]]></DISK_ID>
          <ETH0_CONTEXT_FORCE_IPV4><![CDATA[]]></ETH0_CONTEXT_FORCE_IPV4>
          <ETH0_DNS><![CDATA[x.x.x.x]]></ETH0_DNS>
          <ETH0_EXTERNAL><![CDATA[]]></ETH0_EXTERNAL>
          <ETH0_GATEWAY><![CDATA[x.x.x.x]]></ETH0_GATEWAY>
          <ETH0_GATEWAY6><![CDATA[x:x:x:x:x..]]></ETH0_GATEWAY6>
          <ETH0_IP><![CDATA[x.x.x.x]]></ETH0_IP>
          <ETH0_IP6><![CDATA[x:x:x:x:x..]]></ETH0_IP6>
          <ETH0_IP6_PREFIX_LENGTH><![CDATA[120]]></ETH0_IP6_PREFIX_LENGTH>
          <ETH0_IP6_ULA><![CDATA[]]></ETH0_IP6_ULA>
          <ETH0_MAC><![CDATA[02:00:3e:x:x:x]]></ETH0_MAC>
          <ETH0_MASK><![CDATA[255.255.255.0]]></ETH0_MASK>
          <ETH0_METRIC><![CDATA[]]></ETH0_METRIC>
          <ETH0_METRIC6><![CDATA[]]></ETH0_METRIC6>
          <ETH0_MTU><![CDATA[]]></ETH0_MTU>
          <ETH0_NETWORK><![CDATA[x.x.x.x]]></ETH0_NETWORK>
          <ETH0_SEARCH_DOMAIN><![CDATA[]]></ETH0_SEARCH_DOMAIN>
          <ETH0_VLAN_ID><![CDATA[56]]></ETH0_VLAN_ID>
          <ETH0_VROUTER_IP><![CDATA[]]></ETH0_VROUTER_IP>
          <ETH0_VROUTER_IP6><![CDATA[]]></ETH0_VROUTER_IP6>
          <ETH0_VROUTER_MANAGEMENT><![CDATA[]]></ETH0_VROUTER_MANAGEMENT>
          <NETWORK><![CDATA[YES]]></NETWORK>
          <SSH_PUBLIC_KEY><![CDATA[ssh-rsa blablabla]]></SSH_PUBLIC_KEY>
          <TARGET><![CDATA[hda]]></TARGET>
        </CONTEXT>
        <CPU><![CDATA[1]]></CPU>
        <CPU_MODEL>
          <MODEL><![CDATA[host-passthrough]]></MODEL>
        </CPU_MODEL>
        <DISK>
          <ALLOW_ORPHANS><![CDATA[YES]]></ALLOW_ORPHANS>
          <CEPH_HOST><![CDATA[mon01 mon02 mon03]]></CEPH_HOST>
          <CEPH_SECRET><![CDATA[asd-asd-asd-asd]]></CEPH_SECRET>
          <CEPH_USER><![CDATA[libvirt]]></CEPH_USER>
          <CLONE><![CDATA[YES]]></CLONE>
          <CLONE_TARGET><![CDATA[SYSTEM]]></CLONE_TARGET>
          <CLUSTER_ID><![CDATA[0]]></CLUSTER_ID>
          <DATASTORE><![CDATA[IMAGE]]></DATASTORE>
          <DATASTORE_ID><![CDATA[102]]></DATASTORE_ID>
          <DEV_PREFIX><![CDATA[sd]]></DEV_PREFIX>
          <DISK_ID><![CDATA[0]]></DISK_ID>
          <DISK_SNAPSHOT_TOTAL_SIZE><![CDATA[0]]></DISK_SNAPSHOT_TOTAL_SIZE>
          <DISK_TYPE><![CDATA[RBD]]></DISK_TYPE>
          <DRIVER><![CDATA[qcow2]]></DRIVER>
          <FORMAT><![CDATA[qcow2]]></FORMAT>
          <IMAGE><![CDATA[Copy of CentOS 7]]></IMAGE>
          <IMAGE_ID><![CDATA[554]]></IMAGE_ID>
          <IMAGE_STATE><![CDATA[2]]></IMAGE_STATE>
          <IMAGE_UNAME><![CDATA[fvonlojewski]]></IMAGE_UNAME>
          <LN_TARGET><![CDATA[NONE]]></LN_TARGET>
          <ORIGINAL_SIZE><![CDATA[8192]]></ORIGINAL_SIZE>
          <POOL_NAME><![CDATA[one]]></POOL_NAME>
          <READONLY><![CDATA[NO]]></READONLY>
          <SAVE><![CDATA[NO]]></SAVE>
          <SIZE><![CDATA[8192]]></SIZE>
          <SOURCE><![CDATA[one/one-554]]></SOURCE>
          <TARGET><![CDATA[sda]]></TARGET>
          <TM_MAD><![CDATA[ceph]]></TM_MAD>
          <TYPE><![CDATA[RBD]]></TYPE>
        </DISK>
        <FEATURES>
          <ACPI><![CDATA[yes]]></ACPI>
          <LOCALTIME><![CDATA[yes]]></LOCALTIME>
        </FEATURES>
        <GRAPHICS>
          <LISTEN><![CDATA[0.0.0.0]]></LISTEN>
          <PORT><![CDATA[7112]]></PORT>
          <TYPE><![CDATA[VNC]]></TYPE>
        </GRAPHICS>
        <MEMORY><![CDATA[4096]]></MEMORY>
        <NIC>
          <AR_ID><![CDATA[0]]></AR_ID>
          <BRIDGE><![CDATA[onebr.56]]></BRIDGE>
          <BRIDGE_TYPE><![CDATA[linux]]></BRIDGE_TYPE>
          <CLUSTER_ID><![CDATA[0]]></CLUSTER_ID>
          <FILTER_IP_SPOOFING><![CDATA[YES]]></FILTER_IP_SPOOFING>
          <FILTER_MAC_SPOOFING><![CDATA[YES]]></FILTER_MAC_SPOOFING>
          <GATEWAY><![CDATA[x.x.x.x]]></GATEWAY>
          <INBOUND_AVG_BW><![CDATA[100000]]></INBOUND_AVG_BW>
          <INBOUND_PEAK_BW><![CDATA[125000]]></INBOUND_PEAK_BW>
          <IP><![CDATA[x.x.x.x]]></IP>
          <IP6><![CDATA[x:x:x:x:x:x...]]></IP6>
          <MAC><![CDATA[02:00:3e:x:x:x]]></MAC>
          <MODEL><![CDATA[virtio]]></MODEL>
          <NAME><![CDATA[NIC0]]></NAME>
          <NETWORK><![CDATA[Public-x.x.x.x]]></NETWORK>
          <NETWORK_ID><![CDATA[6]]></NETWORK_ID>
          <NETWORK_UNAME><![CDATA[itwasme]]></NETWORK_UNAME>
          <NIC_ID><![CDATA[0]]></NIC_ID>
          <OUTBOUND_AVG_BW><![CDATA[100000]]></OUTBOUND_AVG_BW>
          <OUTBOUND_PEAK_BW><![CDATA[125000]]></OUTBOUND_PEAK_BW>
          <PHYDEV><![CDATA[bond1]]></PHYDEV>
          <SECURITY_GROUPS><![CDATA[0]]></SECURITY_GROUPS>
          <TARGET><![CDATA[one-1212-0]]></TARGET>
          <VLAN_ID><![CDATA[56]]></VLAN_ID>
          <VN_MAD><![CDATA[802.1Q]]></VN_MAD>
        </NIC>
        <NIC_DEFAULT>
          <MODEL><![CDATA[virtio]]></MODEL>
        </NIC_DEFAULT>
        <OS>
          <ARCH><![CDATA[x86_64]]></ARCH>
          <BOOT><![CDATA[]]></BOOT>
          <UUID><![CDATA[035c63be-51ef-4e11-bf30-7e8f3ba930f6]]></UUID>
        </OS>
        <SECURITY_GROUP_RULE>
          <PROTOCOL><![CDATA[ALL]]></PROTOCOL>
          <RULE_TYPE><![CDATA[OUTBOUND]]></RULE_TYPE>
          <SECURITY_GROUP_ID><![CDATA[0]]></SECURITY_GROUP_ID>
          <SECURITY_GROUP_NAME><![CDATA[default]]></SECURITY_GROUP_NAME>
        </SECURITY_GROUP_RULE>
        <SECURITY_GROUP_RULE>
          <PROTOCOL><![CDATA[ALL]]></PROTOCOL>
          <RULE_TYPE><![CDATA[INBOUND]]></RULE_TYPE>
          <SECURITY_GROUP_ID><![CDATA[0]]></SECURITY_GROUP_ID>
          <SECURITY_GROUP_NAME><![CDATA[default]]></SECURITY_GROUP_NAME>
        </SECURITY_GROUP_RULE>
        <TEMPLATE_ID><![CDATA[441]]></TEMPLATE_ID>
        <TM_MAD_SYSTEM><![CDATA[ceph]]></TM_MAD_SYSTEM>
        <VMID><![CDATA[1212]]></VMID>
      </TEMPLATE>
      <USER_TEMPLATE>
        <DESCRIPTION><![CDATA[test]]></DESCRIPTION>
        <ERROR><![CDATA[Tue Apr 13 10:22:38 2021: DEPLOY: Could not create domain from /var/lib/one//datastores/103/1212/deployment.0]]></ERROR>
        <HOT_RESIZE>
          <CPU_HOT_ADD_ENABLED><![CDATA[NO]]></CPU_HOT_ADD_ENABLED>
          <MEMORY_HOT_ADD_ENABLED><![CDATA[NO]]></MEMORY_HOT_ADD_ENABLED>
        </HOT_RESIZE>
        <INPUTS_ORDER><![CDATA[]]></INPUTS_ORDER>
        <MEMORY_UNIT_COST><![CDATA[MB]]></MEMORY_UNIT_COST>
      </USER_TEMPLATE>
      <HISTORY_RECORDS>
        <HISTORY>
          <OID>1212</OID>
          <SEQ>0</SEQ>
          <HOSTNAME>virt04</HOSTNAME>
          <HID>5</HID>
          <CID>0</CID>
          <STIME>1618302155</STIME>
          <ETIME>0</ETIME>
          <VM_MAD><![CDATA[kvm]]></VM_MAD>
          <TM_MAD><![CDATA[ceph]]></TM_MAD>
          <DS_ID>103</DS_ID>
          <PSTIME>1618302155</PSTIME>
          <PETIME>1618302156</PETIME>
          <RSTIME>1618302156</RSTIME>
          <RETIME>0</RETIME>
          <ESTIME>0</ESTIME>
          <EETIME>0</EETIME>
          <ACTION>0</ACTION>
          <UID>-1</UID>
          <GID>-1</GID>
          <REQUEST_ID>-1</REQUEST_ID>
        </HISTORY>
      </HISTORY_RECORDS>
    </VM>

girg · April 13, 2021, 9:54am

onedatastore show -x 103

<DATASTORE>
      <ID>103</ID>
      <UID>0</UID>
      <GID>0</GID>
      <UNAME>oneadmin</UNAME>
      <GNAME>oneadmin</GNAME>
      <NAME>SYSTEM</NAME>
      <PERMISSIONS>
        <OWNER_U>1</OWNER_U>
        <OWNER_M>1</OWNER_M>
        <OWNER_A>0</OWNER_A>
        <GROUP_U>1</GROUP_U>
        <GROUP_M>0</GROUP_M>
        <GROUP_A>0</GROUP_A>
        <OTHER_U>0</OTHER_U>
        <OTHER_M>0</OTHER_M>
        <OTHER_A>0</OTHER_A>
      </PERMISSIONS>
      <DS_MAD><![CDATA[-]]></DS_MAD>
      <TM_MAD><![CDATA[ceph]]></TM_MAD>
      <BASE_PATH><![CDATA[/var/lib/one//datastores/103]]></BASE_PATH>
      <TYPE>1</TYPE>
      <DISK_TYPE>3</DISK_TYPE>
      <STATE>0</STATE>
      <CLUSTERS>
        <ID>0</ID>
      </CLUSTERS>
      <TOTAL_MB>16798966</TOTAL_MB>
      <FREE_MB>4804834</FREE_MB>
      <USED_MB>11994132</USED_MB>
      <IMAGES/>
      <TEMPLATE>
        <ALLOW_ORPHANS><![CDATA[mixed]]></ALLOW_ORPHANS>
        <BRIDGE_LIST><![CDATA[virt01 virt02 virt03]]></BRIDGE_LIST>
        <CEPH_HOST><![CDATA[mon01 mon02 mon03]]></CEPH_HOST>
        <CEPH_SECRET><![CDATA[bla-bla-bla]]></CEPH_SECRET>
        <CEPH_USER><![CDATA[libvirt]]></CEPH_USER>
        <DISK_TYPE><![CDATA[RBD]]></DISK_TYPE>
        <DRIVER><![CDATA[raw]]></DRIVER>
        <DS_MIGRATE><![CDATA[NO]]></DS_MIGRATE>
        <POOL_NAME><![CDATA[one]]></POOL_NAME>
        <RESTRICTED_DIRS><![CDATA[/]]></RESTRICTED_DIRS>
        <SAFE_DIRS><![CDATA[/var/tmp /home/one]]></SAFE_DIRS>
        <SHARED><![CDATA[YES]]></SHARED>
        <STAGING_DIR><![CDATA[/home/one]]></STAGING_DIR>
        <TM_MAD><![CDATA[ceph]]></TM_MAD>
        <TYPE><![CDATA[SYSTEM_DS]]></TYPE>
      </TEMPLATE>
    </DATASTORE>

oneimage show -x 471

<IMAGE>
      <ID>471</ID>
      <UID>3</UID>
      <GID>0</GID>
      <UNAME>itwasme</UNAME>
      <GNAME>oneadmin</GNAME>
      <NAME>CentOS 7</NAME>
      <PERMISSIONS>
        <OWNER_U>1</OWNER_U>
        <OWNER_M>1</OWNER_M>
        <OWNER_A>0</OWNER_A>
        <GROUP_U>0</GROUP_U>
        <GROUP_M>0</GROUP_M>
        <GROUP_A>0</GROUP_A>
        <OTHER_U>0</OTHER_U>
        <OTHER_M>0</OTHER_M>
        <OTHER_A>0</OTHER_A>
      </PERMISSIONS>
      <TYPE>0</TYPE>
      <DISK_TYPE>3</DISK_TYPE>
      <PERSISTENT>0</PERSISTENT>
      <REGTIME>1588596202</REGTIME>
      <SOURCE><![CDATA[one/one-471]]></SOURCE>
      <PATH><![CDATA[https://marketplace.opennebula.systems//appliance/a093aa48-1113-11ea-8c65-f0def1753696/download/0]]></PATH>
      <FORMAT><![CDATA[qcow2]]></FORMAT>
      <FS><![CDATA[]]></FS>
      <SIZE>8192</SIZE>
      <STATE>2</STATE>
      <PREV_STATE>2</PREV_STATE>
      <RUNNING_VMS>2</RUNNING_VMS>
      <CLONING_OPS>0</CLONING_OPS>
      <CLONING_ID>-1</CLONING_ID>
      <TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT>
      <DATASTORE_ID>102</DATASTORE_ID>
      <DATASTORE>IMAGE</DATASTORE>
      <VMS>
        <ID>965</ID>
        <ID>1194</ID>
      </VMS>
      <CLONES/>
      <APP_CLONES/>
      <TEMPLATE>
        <DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX>
        <DRIVER><![CDATA[qcow2]]></DRIVER>
        <FROM_APP><![CDATA[69]]></FROM_APP>
        <FROM_APP_MD5><![CDATA[e1baa0b43ab17bd798c2fdc0ece38fa3]]></FROM_APP_MD5>
        <FROM_APP_NAME><![CDATA[CentOS 7]]></FROM_APP_NAME>
      </TEMPLATE>
      <SNAPSHOTS>
        <ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS>
        <CURRENT_BASE><![CDATA[-1]]></CURRENT_BASE>
        <NEXT_SNAPSHOT><![CDATA[0]]></NEXT_SNAPSHOT>
      </SNAPSHOTS>
    </IMAGE>

oneimage show 471 | grep -i source
SOURCE : one/one-471

I don’t know how to access the ceph storage where the images is

root@virt04: /var/lib/one/datastores/103/1212 # ls -la
insgesamt 384
drwxrwxr-x  2 oneadmin oneadmin     88 13. Apr 10:22 .
drwxrwxr-x 26 oneadmin oneadmin   4096 13. Apr 11:28 ..
-rw-rw-r--  1 oneadmin oneadmin   2320 13. Apr 10:22 deployment.0
-rw-r--r--  1 oneadmin oneadmin 372736 13. Apr 10:22 disk.1
-rw-rw-r--  1 oneadmin oneadmin   1292 13. Apr 10:22 ds.xml
-rw-rw-r--  1 oneadmin oneadmin   6933 13. Apr 10:22 vm.xml
root@virt04: /var/lib/one/datastores/103/1212 # file disk.1
disk.1: # ISO 9660 CD-ROM filesystem data 'CONTEXT'

atodorov_storpool · April 13, 2021, 10:06am

So, the libvirt error point that the error is because the disk is defined with type qcow2 but it is not (wild guess it should be raw)

In the VM XML the disk had the following elements as qcow2

          <DRIVER><![CDATA[qcow2]]></DRIVER>
          <FORMAT><![CDATA[qcow2]]></FORMAT>

In the sources, the DRIVER value is used to determine the disk driver type in the domain XML:

github.com

OpenNebula/one/blob/master/src/vmm/LibVirtDriverKVM.cc#L1216


      
          {
              ostringstream dev;
          
              dev << vm->get_system_dir() << "/disk." << disk_id;
          
              if (type == "BLOCK_CDROM")
              {
                  file << "\t\t<disk type='block' device='cdrom'>" << endl;
              }
              else
              {
                  file << "\t\t<disk type='block' device='disk'>" << endl;
              }
          
              file << "\t\t\t<source dev=" << one_util::escape_xml_attr(dev.str())
                   << "/>" << endl;
          }
          else if ( type == "ISCSI" )
          {
              file << "\t\t<disk type='network' device='disk'>" << endl;

Which lead to the incompatible configuration.

The DRIVER element is in the Image Template too so if not overridden in the VM Template it is inherited from there:

<IMAGE>
...
      <TEMPLATE>
        <DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX>
        <DRIVER><![CDATA[qcow2]]></DRIVER>
        <FROM_APP><![CDATA[69]]></FROM_APP>
        <FROM_APP_MD5><![CDATA[e1baa0b43ab17bd798c2fdc0ece38fa3]]></FROM_APP_MD5>
        <FROM_APP_NAME><![CDATA[CentOS 7]]></FROM_APP_NAME>
      </TEMPLATE>

IMO in this case the DRIVER element should be set to raw.

Best Regards,
Anton

cgonzalez · April 13, 2021, 11:10am

Hello @girg and @atodorov_storpool,

Anton is in the right way, to fix the problem, update the image FORMAT and DRIVER attributes and set them to raw so they will match the real format of the image.

The real reason for the error, In the current OpenNebula 6.0 version, the FORMAT attribute is automatically set by the drivers, and from it the rest of attributes (like DRIVER) are inferred. The upgrade process may leave some images with the old value for FORMAT.

Long term fix, We’ll improve the upgrade process so it is able to fix wrong FORMAT values. For the current version we’ve added this as known issues in the documentation.

Thank you very much for the feedback guys!

References
[1] Qcow2 & ceph: QEMU and Block Devices — Ceph Documentation
[2] OpenNebula Driver & Format setting: https://github.com/OpenNebula/one-ee/blob/5a7fed9122dad418050aba256c16ef56c76adb1f/src/image/Image.cc#L576

atodorov_storpool · April 13, 2021, 12:14pm

Fabian,

If you have production VMs with this issue you could fix them online by carefully using the onedb change-body command:

onedb change-body vm --id $VM_ID "/VM/TEMPLATE/DISK[DISK_ID=$DISK_ID]/DRIVER" 'raw'
onedb change-body vm --id $VM_ID "/VM/TEMPLATE/DISK[DISK_ID=$DISK_ID]/FORMAT" 'raw'

Where $VM_ID is the ID of the VM and $DISK_ID is the VM disk id that should be fixed.

I think that the DRIVER in the Image Template could be edited via the Sunstone or oneimage shell tool.

Hope this helps.

Best Regards,
Anton Todorov

tobx · April 13, 2021, 12:55pm

Hello @cgonzalez

isn’t this a major issue? I think you should explain some official workarounds in your documentation.

Probably, several users will hit this issue…

girg · April 13, 2021, 1:02pm

Hi Anton and Christian,

thank you for your help and explenations!!

I got it to work by editing the body field in the table image_pool. I changed the FORMAT to raw and now I am able to deploy VMs again from that image.

Thanks!
Fabian

cgonzalez · April 15, 2021, 6:38am

Hello @tobx,

As mentioned before, we have already added this as a known issue in the documentation so anyone who is about to upgrade can decide if they prefer to wait until next release or upgrade and follow the workaround.

tobx · April 15, 2021, 7:17am

Hey @cgonzalez

it’s about the workaround not even being documented - not the fact, that there is a known issue.

cgonzalez · April 15, 2021, 12:58pm

Ok @tobx it makes sense.vIt’s mentioned in the known issues, but we can add a more detailed procedure. Also, I’ll add the workaround explained by @atodorov_storpool in case it might affect any running VM.

Thanks for the feedback!

girg · June 24, 2021, 12:19pm

Hi @cgonzalez
I also get this error when cloning images and setting them as persistent (or deploying new vm with “instantiate as persistent”). The source disk has DRIVER and FORMAT set to raw. The new disk has DRIVER and FORMAT set to save_as. I am using version 6.0.0.2.

Thu Jun 24 12:44:20 2021 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/kvm/deploy ‘/var/lib/one//datastores/103/1297/deployment.0’ ‘virt02’ 1297 virt02
Thu Jun 24 12:44:20 2021 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/103/1297/deployment.0
Thu Jun 24 12:44:20 2021 [Z0][VMM][I]: error: unsupported configuration: unknown driver format value ‘save_as’
Thu Jun 24 12:44:20 2021 [Z0][VMM][E]: Could not create domain from /var/lib/one//datastores/103/1297/deployment.0
Thu Jun 24 12:44:20 2021 [Z0][VMM][I]: ExitCode: 255

Will there be a fix for this?

Best regards
Fabian

cgonzalez · June 25, 2021, 7:22am

Hello @girg,

I’m not able to reproduce the issue, it would be useful if you can share:

The XML of the source image.
The XML of the image created after instantiating as persistent.
The exact steps you do for reproducing the issue.

Also, it would be useful to know if this is happening with every image in your cloud or just some of them.

girg · June 28, 2021, 1:42pm

Hi @cgonzalez

oneimage show 394 --xml

<IMAGE>
  <ID>394</ID>
  <UID>11</UID>
  <GID>107</GID>
  <UNAME>myuser</UNAME>
  <GNAME>mygroup</GNAME>
  <NAME>myvm</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>0</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <TYPE>0</TYPE>
  <DISK_TYPE>3</DISK_TYPE>
  <PERSISTENT>0</PERSISTENT>
  <REGTIME>1573825729</REGTIME>
  <SOURCE><![CDATA[one/one-394]]></SOURCE>
  <PATH><![CDATA[]]></PATH>
  <FORMAT><![CDATA[raw]]></FORMAT>
  <FS><![CDATA[]]></FS>
  <SIZE>40960</SIZE>
  <STATE>2</STATE>
  <PREV_STATE>2</PREV_STATE>
  <RUNNING_VMS>1</RUNNING_VMS>
  <CLONING_OPS>0</CLONING_OPS>
  <CLONING_ID>-1</CLONING_ID>
  <TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT>
  <DATASTORE_ID>102</DATASTORE_ID>
  <DATASTORE>IMAGE</DATASTORE>
  <VMS>
    <ID>979</ID>
  </VMS>
  <CLONES/>
  <APP_CLONES/>
  <TEMPLATE>
    <DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX>
    <DRIVER><![CDATA[raw]]></DRIVER>
    <FORMAT><![CDATA[raw]]></FORMAT>
    <SAVED_DISK_ID><![CDATA[0]]></SAVED_DISK_ID>
    <SAVED_IMAGE_ID><![CDATA[310]]></SAVED_IMAGE_ID>
    <SAVED_VM_ID><![CDATA[710]]></SAVED_VM_ID>
    <SAVE_AS_HOT><![CDATA[YES]]></SAVE_AS_HOT>
  </TEMPLATE>
  <SNAPSHOTS/>
</IMAGE>

When I create a new VM from template, I instantiate as persistent. The new image looks like this:
oneimage show 625 --xml

<IMAGE>
  <ID>625</ID>
  <UID>56</UID>
  <GID>0</GID>
  <UNAME>myuser</UNAME>
  <GNAME>mygroup</GNAME>
  <NAME>myvmclone</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>0</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <TYPE>0</TYPE>
  <DISK_TYPE>3</DISK_TYPE>
  <PERSISTENT>0</PERSISTENT>
  <REGTIME>1624880449</REGTIME>
  <SOURCE><![CDATA[one/one-625]]></SOURCE>
  <PATH><![CDATA[one/one-394]]></PATH>
  <FORMAT><![CDATA[save_as]]></FORMAT>
  <FS><![CDATA[]]></FS>
  <SIZE>40960</SIZE>
  <STATE>2</STATE>
  <PREV_STATE>2</PREV_STATE>
  <RUNNING_VMS>1</RUNNING_VMS>
  <CLONING_OPS>0</CLONING_OPS>
  <CLONING_ID>-1</CLONING_ID>
  <TARGET_SNAPSHOT>-1</TARGET_SNAPSHOT>
  <DATASTORE_ID>102</DATASTORE_ID>
  <DATASTORE>IMAGE</DATASTORE>
  <VMS>
    <ID>1304</ID>
  </VMS>
  <CLONES/>
  <APP_CLONES/>
  <TEMPLATE>
    <DEV_PREFIX><![CDATA[vd]]></DEV_PREFIX>
    <SAVED_DISK_ID><![CDATA[0]]></SAVED_DISK_ID>
    <SAVED_IMAGE_ID><![CDATA[310]]></SAVED_IMAGE_ID>
    <SAVED_VM_ID><![CDATA[710]]></SAVED_VM_ID>
    <SAVE_AS_HOT><![CDATA[YES]]></SAVE_AS_HOT>
  </TEMPLATE>
  <SNAPSHOTS>
    <ALLOW_ORPHANS><![CDATA[NO]]></ALLOW_ORPHANS>
    <CURRENT_BASE><![CDATA[-1]]></CURRENT_BASE>
    <NEXT_SNAPSHOT><![CDATA[0]]></NEXT_SNAPSHOT>
  </SNAPSHOTS>
</IMAGE>

I can deploy the vm when I change

<FORMAT><![CDATA[save_as]]></FORMAT>

to raw in the database.

Actually this is the only image I found where this problem accurs at the moment. This procedure worked until before the update to 6.0.

Best regards
Fabian

Topic		Replies	Views
Could not create domain from /var/lib/one//datastores/0/3/deployment.0 General	3	1996	August 23, 2021
Cannot create VM General	0	426	May 7, 2022
Can't create VMs Community Support	4	1008	May 18, 2022
Random problems when deploying a Virtual Machine with Raw Image format Community Support	2	852	February 23, 2016
Problems creating VM General	1	256	April 16, 2021

Can't create VM from "old" images

Related topics