CEPH cannot attach CDROM

Community Support
1

Hello,

I am not able to hot-attach a CD-ROM image to the VM in Sunstone. When I add the image to the template as a second disk and then instantiate the template, everything works as expected, and I am able to install the OS from the CD-ROM. When I try to attach the very same CD-ROM image to the running VM, which does not have the CD-ROM defined in the template, it fails with the following messages in the log file:

Wed Mar 30 11:38:36 2016 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_attach.
Wed Mar 30 11:38:37 2016 [Z0][VMM][I]: Command execution fail: /var/lib/one/tmp/vmm/kvm/attach_disk 'one-66' '/var/lib/one//datastores/0/66/disk.2' 'hdb' '1' 'P...4=' 66 physcalhost10
Wed Mar 30 11:38:37 2016 [Z0][VMM][E]: attach_disk: Command "virsh --connect qemu:///system attach-device one-66 /var/lib/one//datastores/0/66/disk.2.attach" failed: error: Failed to attach device from /var/lib/one//datastores/0/66/disk.2.attach
Wed Mar 30 11:38:37 2016 [Z0][VMM][I]: error: Cannot access storage file '/var/lib/one//datastores/0/66/disk.2' (as uid:9869, gid:9869): No such file or directory
Wed Mar 30 11:38:37 2016 [Z0][VMM][E]: Could not attach /var/lib/one//datastores/0/66/disk.2 (hdb) to one-66
Wed Mar 30 11:38:37 2016 [Z0][VMM][I]: ExitCode: 1
Wed Mar 30 11:38:37 2016 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_detach.
Wed Mar 30 11:38:37 2016 [Z0][VMM][I]: Failed to execute virtualization driver operation: attach_disk.
Wed Mar 30 11:38:37 2016 [Z0][VMM][E]: Error attaching new VM Disk: Could not attach /var/lib/one//datastores/0/66/disk.2 (hdb) to one-66

I am not sure about the .../datastores/0/... path, because my CEPH datastore (on which the CD-ROM image resides) has ID=100, not 0.

The workaround of destroying the VM and instantiating it again from a modified template is not pretty, for example the newly instantiated VM gets a new MAC/IP addresses.

Is it possible to hot-attach a CD-ROM image from the CEPH datastore to the running VM?
Also, is it possible to change the boot device in the running VM without instantiating it again from the template?

Thanks!

2

For the changing the boot device, there is no other way without reinstantiate the VM. For the attaching the CD-ROM, i guess you need to shutdown the VM before attach the CD-ROM type image since the device type is “hd”

3

Is there a way of reinstantiating VM without changing its MAC address?

If the hot-attaching and removing the CD worked, I could leave the first boot device as CD-ROM, and the second one as HDD. But it does not work for me.

Also, attaching the CD-ROM after the VM shutdown does not work for me - the green “Attach disk” button is greyed out (disabled) in Sunstone for the VM which is shut down.

You suggest that the problem might be with the image being of type “hd”. So I have changed DEV_PREFIX of the CD-ROM image to “vd” and started over. The result is more-or-less the same:

Fri Apr 1 12:21:51 2016 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_attach.
Fri Apr 1 12:21:51 2016 [Z0][VMM][I]: Command execution fail: /var/lib/one/tmp/vmm/kvm/attach_disk 'one-52' '/var/lib/one//datastores/0/52/disk.2' 'vdb' '1' 'P...E+' 52 physicalhost12
Fri Apr 1 12:21:51 2016 [Z0][VMM][E]: attach_disk: Command "virsh --connect qemu:///system attach-device one-52 /var/lib/one//datastores/0/52/disk.2.attach" failed: error: Failed to attach device from /var/lib/one//datastores/0/52/disk.2.attach
Fri Apr 1 12:21:51 2016 [Z0][VMM][I]: error: Cannot access storage file '/var/lib/one//datastores/0/52/disk.2' (as uid:9869, gid:9869): No such file or directory
Fri Apr 1 12:21:51 2016 [Z0][VMM][E]: Could not attach /var/lib/one//datastores/0/52/disk.2 (vdb) to one-52
Fri Apr 1 12:21:51 2016 [Z0][VMM][I]: ExitCode: 1
Fri Apr 1 12:21:52 2016 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_detach.
Fri Apr 1 12:21:52 2016 [Z0][VMM][I]: Failed to execute virtualization driver operation: attach_disk.
Fri Apr 1 12:21:52 2016 [Z0][VMM][E]: Error attaching new VM Disk: Could not attach /var/lib/one//datastores/0/52/disk.2 (vdb) to one-52

The problem is that /var/lib/one//datastores/0/52/disk.2 indeed does not exist - the image itself is from the CEPH datastore 100, not from the system datastore 0 (FWIW, I use non-shared SYSTEM_DS with TM_MAD=ssh). The disk.2.attach file references the local image instead of a CEPH-based one:

<disk type='file' device='disk'>
  <driver name='qemu' type='raw' />
  <source file='/var/lib/one//datastores/0/52/disk.2' > 
  </source>
  <target dev='vdb'/>
  <readonly/>
</disk>
4

Is hot-plugging CDROM images supported at all?

I discovered that when I change the type of the image from CDROM to either DATABLOCK or OS, hot-attaching the image works as expected. Why it is not possible to attach images of type CDROM from the CEPH RBD datastore? From the above post it seems that the disk.2.attach XML file is simply incorrect - it does not reference CEPH at all.

There are other problems with ISO images of type DATABLOCk or OS - they are seen as disks with partitions by the guest kernel, not as CD/DVD drives. Also, they are unusable for the VM recovery by booting from CD - I did not find how to boot the VM from the second disk without reinstantiating it (and getting a new MAC address, etc.).

What is the supported way of booting the VM from a rescue image without changing its MAC address (i.e. without reinstantiating it)?

5

Hi Jan,

Yes this is indeed a bug, the attach description is not properly generated for rbd_cdrom images. I’ll post a patch asap.

Thanks for your feedback

6

Could you try the following patch?

0001-bug-4398-Add-rbd_cdrom-to-attach-generation-script.patch (932 Bytes)

probably, the easier way to apply this is

  1. Copy the above file to /var/lib/one/remotes/vmm/kvm
  2. Execute patch -p5 < 0001-bug-4398-Add-rbd_cdrom-to-attach-generation-script.patch
  3. Sync your hosts with onehost sync -f <host>

and try again, if it does not work I’ll be interested in disk.2.attach file and log messages, as you did in your previous posts.

Sorry but I cannot access our ceph cluster right now to test it

THANKS

7

OK, I have tried the above patch (thanks for the reply!). The disk.2.attach file looks different now:

<disk type='network' device='cdrom'>
  <driver name='qemu' type='raw' />
  <source protocol='rbd' name='one/one-19'>
  <host name='br1'/><host name='br2'/><host name='br3'/>
  </source>
  <auth username='libvirt'>                <secret type='ceph' uuid='...'/>              </auth>
  <target dev='hdb'/>
  <readonly/>
</disk>

(where br1…br3 are ceph bridge and monitor hosts and one-19 is the correct name of the CDROM image). However, attaching the CDROM itself fails:

Sat Apr 9 00:35:37 2016 [Z0][VMM][E]: attach_disk: Command "virsh --connect qemu:///system attach-device one-89 /var/lib/one//datastores/0/89/disk.2.attach" failed: error: Failed to attach device from /var/lib/one//datastores/0/89/disk.2.attach
Sat Apr 9 00:35:37 2016 [Z0][VMM][I]: error: internal error: No device with bus 'ide' and target 'hdb'. cdrom and floppy device hotplug isn't supported by libvirt
Sat Apr 9 00:35:37 2016 [Z0][VMM][E]: Could not attach rbd (hdb) to one-89
Sat Apr 9 00:35:37 2016 [Z0][VMM][I]: ExitCode: 1
Sat Apr 9 00:35:37 2016 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_detach.
Sat Apr 9 00:35:37 2016 [Z0][VMM][I]: Failed to execute virtualization driver operation: attach_disk.
Sat Apr 9 00:35:37 2016 [Z0][VMM][E]: Error attaching new VM Disk: Could not attach rbd (hdb) to one-89

The physical hosts are running up-to-date CentOS 7, FWIW. Maybe the virtual device (the CD-ROM “drive”) should be present in the VM from the beginning, and only the media change should be done by libvirt? (see virsh change-media command).

8

Can you try to use DEV_PREFIX=sd for the CDROM DISK?

9

It failed also with DEV_PREFIX=sd (and also with DEV_PREFIX=vd as well as no DEV_PREFIX at all):

Mon Apr 11 08:58:38 2016 [Z0][VMM][E]: attach_disk: Command "virsh --connect qemu:///system attach-device one-91 /var/lib/one//datastores/0/91/disk.2.attach" failed: error: Failed to attach device from /var/lib/one//datastores/0/91/disk.2.attach
Mon Apr 11 08:58:38 2016 [Z0][VMM][I]: error: internal error: No device with bus 'scsi' and target 'sda'. cdrom and floppy device hotplug isn't supported by libvirt
Mon Apr 11 08:58:38 2016 [Z0][VMM][E]: Could not attach rbd (sda) to one-91
Mon Apr 11 08:58:38 2016 [Z0][VMM][I]: ExitCode: 1
Mon Apr 11 08:58:38 2016 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_detach.
Mon Apr 11 08:58:38 2016 [Z0][VMM][I]: Failed to execute virtualization driver operation: attach_disk.
Mon Apr 11 08:58:38 2016 [Z0][VMM][E]: Error attaching new VM Disk: Could not attach rbd (sda) to one-91

With DEV_PREFIX=vd the log looks similar except that it mentions vdb instead of sda and a different bus:

error: internal error: No device with bus 'virtio' and target 'vdb'. cdrom and floppy device hotplug isn't supported by libvirt

10

Sorry I over look the error output:

: error: internal error: No device with bus ‘scsi’ and target ‘sda’. cdrom
and floppy device hotplug isn’t supported by libvirt

11

Digging into this further, I managed to insert the image to the existing CD-ROM device. I took the contextualization CD-ROM drive (hda in my case), edited the disk.2.attach file generated by OpenNebula - changed <target dev=sda> to <target dev=hda>, and ran “virsh update-device one-91 disk.2.attach”. This correctly plugged the image from the CEPH RBD to the existing CD-ROM drive (instead of the contextualiztion image), and I have successfully mounted it from the guest VM.

So maybe there should be an option to create VM with an empty CD-ROM drive for further hotplugging the media?

12

Probably this would help:

http://dev.opennebula.org/issues/4320

13

Anyway, I don’t stritcly need CD-ROM hotplug per se. What am I trying to find out is how to run long-living VMs in OpenNebula.

One of the problems is that I want to be able to fix the unbootable VM by booting it from the recovery media (either CD-ROM or HDD). Of course without having to delete it and instantiate it back, because this means MAC address change, which in turn leads into various other problems (arpwatch, DHCP, etc.).

Also, it would be nice if the hot-plug of CD-ROMs can be disabled in SunStone, because being able to initiate the hot-plug operation only to find out it later failed because it is not supported in libvirt and having to dig into the VM log is kind of frustrating :slight_smile:

14

Yes, this is probably what I am looking for. It would be nice to have.