Question about VMM actions and libvirt options

Community Support
1

Hi all

We are configuring a new cloud infrastructure and we have enabled kerberos authentication in our hypervisors. The good thing is that libvirtd also supports kerberos auth through tcp sasl.

That means that we can connect and migrate VMs using virsh and kerberos authentication to increase our internal security. The point is that kerberos requires a valid ticket, if your system is already configured is quite easy to generate one, you only need to execute in the hyp as oneadmin:

kinit -k -l 2h -t /etc/libvirt/krb5.tab

to generate a ticket valid for 2 hours, enough to migrate any VM to another hyp. The question is, how we can do this automatically from OpenNebula before the live migration process starts? (we only need it for VM live migrations).

It seems that you can set your own VMM drivers and set oned.conf to use them like this to be used for migrate action (please correct me if I’m wrong):

VM_MAD = [
name = “kvm”,
executable = “one_vmm_exec”,
arguments = “-t 15 -r 0 -l migrate=migrate_krb5 kvm”,
default = “vmm_exec/vmm_exec_kvm.conf”,
type = “kvm” ]

So we have modified /var/lib/one/remotes/vmm/kvm/kvmrc to include a new variable to set our keytab for libvirtd:

export LANG=C
export LIBVIRT_KEYTAB=/etc/libvirt/krb5.tab
export LIBVIRT_URI=qemu:///system
export QEMU_PROTOCOL=qemu+tcp
export SHUTDOWN_TIMEOUT=300

and our adhoc migrate script in /var/lib/one/remotes/vmm/kvm/migrate_krb5 looks like:

source $(dirname $0)/kvmrc
source $(dirname $0)/…/…/scripts_common.sh

deploy_id=$1
dest_host=$2

klist
exit=$?

if [ -n $LIBVIRT_KEYTAB ] && [ $exit -eq 1 ]
then
kinit -k -l 1h -t $LIBVIRT_KEYTAB libvirt/$(hostname -f)
fi

exec_and_log “virsh --connect $LIBVIRT_URI migrate --live $MIGRATE_OPTIONS $deploy_id $QEMU_PROTOCOL://$dest_host/system”
“Could not migrate $deploy_id to $dest_host”

The migrate script was modified to detect if is still available a valid ticket, if not is created a new one…

The problem is that this script is executed locally in the ONE server not in hypA to migrate a VM to hypB, the other option is to change directly the migrate script provided by OpenNebula, but it will be overwritten in the nex upgrade and we don’t want that.

Do you know which options do we have and which is the best? It will be great to have the possibility to execute a external script before any official VMM script but in the hypervisors not only locally, any suggestion? (I don’t know if this option is already implemented…)

Thanks in advance!
Alvaro

2

I can think of two possible options, none of them perfect, though.

The first possibility is to use the “premigrate” script. The problem is that this script exists within the context of the TM drivers. Every time a live-migration takes place, the “premigrate” script of the system datastore is called. I assume in your case you are using shared? In that case you can simply add that logic to that script and you’re done. However, if you are using other hypervisors you need to check in the “premigrate” whether you are indeed in a libvirt hypervisor, and if you are using other system datastores, you need to copy that logic to all the system datastores.

The other option is to customize this driver: one_vmm_exec.rb:def migrate. You can add steps and define your own vmm pre-migrate step. Of course the problem is that you will need to re-patch that while on every upgrade…

3

Hi @jmelis

Thanks a lot for your reply! about the two options that you have commented I think that we will try the first one. I assume that /var/lib/one/remotes/tm/shared/premigrate will also overwritten after a package upgrade right?.

For shared datastores it only contains an exit 0. I will include our script here.

probably you have already thought about that, to include something like a prolog and epilog scripts defined by the user not only for the migration, also for cp and mv VMM actions to be executed in the hypervisors, that would be great! I don’t know if someone else will be interested on this…

But now I will try the premigrate script :smile: , we can also set this file from Quattor toolkit

Cheers and thanks!
Alvaro

4

I assume that /var/lib/one/remotes/tm/shared/premigrate will also overwritten after a package upgrade right?.

Yes, but the files under /var/lib/one/remotes are declared as configuration files, therefore, you will see your oldfile as premigrate.rpmsave, and in debian environments it will show you a prompt.

probably you have already thought about that, to include something like a prolog and epilog scripts defined by the user not only for the migration, also for cp and mv VMM actions to be executed in the hypervisors, that would be great! I don’t know if someone else will be interested on this…

No, we hadn’t thought of that :slight_smile: Feel free to open a feature request elaborating a bit more on the use case and the details!

5

Hi @jmelis

Just one more question, about /var/lib/one/remotes/tm/shared/premigrate script, it seems that the script is not executed in the hypervisor, it’s executed in the OpenNebula server, that’s right?

About the feature request, sure! I will open a new one :smile:

Cheers
Alvaro

6

You are right, it’s executed in the OpenNebula frontend. You could however use ssh_exec_and_log and a heredoc to send the script. Would that work for you?

7

Hi,
In our driver we are modifying pre/post migrate scripts with(removed comments):

#!/bin/bash
[ -d "${0}.d" ] && for hook in "${0}.d"/* ; do source "$hook"; done
exit 0

And our code placed in …/{pre,post}migrate.d/

This way anyone can drop their script there. It must only do the work if it detect that it is responsible for the action.

I am sure that @alvaro_simongarcia is thinkig for something like this to be placed in the beginning and ending of each script. like:

#!/bin/bash

#some initialisation stuff

[ -d "${0}-pre.d" ] && for hook in "${0}-pre.d"/* ; do source "$hook"; done

# main work

[ -d "${0}-post.d" ] && for hook in "${0}-post.d"/* ; do source "$hook"; done

exit 0

Cheers
Anton Todorov

8

@jmelis yes that works thanks!

@atodorov_storpool yes, I was thinking about something like that! Is quite easy to implement and the prolog and epilog scripts are only used in the hypervisor if the user includes the directory and the script and it is valid for any VMM. @atodorov_storpool are you using this in your setup right now? If so, maybe you should open a PR :smile:

Cheers
Alvaro

9

btw I have opened a new request to include this feature:

http://dev.opennebula.org/issues/4136