How to set number of VM cores (use cores instead of sockets)?

I have the following issue. I have a host with CPU 4000. 40 cores.
So I want to create a WIN10 VM with 20 cores. But the VM is started with this in the process
-smp 20,sockets=20,cores=1,threads=1
Of course this does not work, because windows 10 has 2 sockets limit.

How can i change that to sockets=1, cores=10, threads=10 or something else with 1 socket.
I see that generated deployment file does not contain such info so is this a libvirt issue or something else?

I also tried adding to the RAW DATA tab

<cpu mode='host-passthrough'>
<topology sockets='1' cores='10' threads='10'/>
</cpu>

It’s added to the deployment file, but it doesn’t work.

Opennebula version is 5.8.1 running on debian stretch 9.8

Setting a fine grained topology for the VMs is a feature that was added to the master branch somewhat recently. You could wait for release 5.10 or build the master branch yourself and install it.

Thank you.
Yes I’ve noticed something related about numa and pinning on github which would be great. So until 5.10, one can’t can’t run win 10/7 with more than two cores?
I’m asking, because could not find somebody else asked the same question. Which is odd, because, a lot of people are using on.
And also I think that this is so significant, that should be mentioned in the docs.

Hi Branislav,

I’ve developed a sort of framework to extend the default domain XML generated by OpenNebula. There is an option to override the default VMM_MAD scripts with local ones.

Here is a short guide how to configure it if you’d like to give it a try.

# checkout addon-storpool
git clone https://github.com/OpenNebula/addon-storpool
# copy the relevant files
cp -a addon-storpool/vmm/kvm/deploy* /var/lib/one/remotes/vmm/kvm/
# create the placeholder for the extra scripts
mkdir /var/lib/one/remotes/vmm/kvm/deploy-tweaks.d
# and "enable" some of the scripts
cp /var/lib/one/remotes/vmm/kvm/deploy-tweaks.d{.example,}/cpu.py
# optionally if you want better windows performace add hyperv-clock.py too
cp /var/lib/one/remotes/vmm/kvm/deploy-tweaks.d{.example,}/hyperv-clock.py.py
# fix the ownership...
chown -R oneadmin.oneadmin  /var/lib/one/remotes/vmm/kvm

After preparing the files you’ll need to enable them in oned.conf, so edit /etc/one/oned.conf and add to the VM_MAD’s ARGUMENTS:

VM_MAD = [
    ...
    ARGUMENTS      = "-t 15 -r 0 kvm -l deploy=deploy-tweaks",
    ...
]

The topology is configured via variables in the VM’s “USER TEMPLATE”. To protect the variables to be editable only by the oneadmin user you should append the following lines to /etc/one/oned.conf:

cat >>/etc/one/oned.conf <<EOF

VM_RESTRICTED_ATTR = "T_CPU_THREADS"
VM_RESTRICTED_ATTR = "T_CPU_SOCKETS"
VM_RESTRICTED_ATTR = "T_CPU_FEATURES"
VM_RESTRICTED_ATTR = "T_CPU_MODE"
VM_RESTRICTED_ATTR = "T_CPU_MODEL"
VM_RESTRICTED_ATTR = "T_CPU_VENDOR"
VM_RESTRICTED_ATTR = "T_CPU_CHECK"
VM_RESTRICTED_ATTR = "T_CPU_MATCH"
EOF

After the configuration changes you must restart the opennebula service and update the hosts with su - oneadmin -c 'onehost sync --force'

Here is how it works. With this configuration OpenNebula will call the deply-tweaks on the front-end instead of calling /var/tmp/one/vmm/kvm/deploy on the KVM host via ssh. The deploy-tweaks script cache the domain XML and will query opennebula for the VM details (a.k.a. the VM’s metadata). Then it will pass both the domain XML and the VM’s metadata to each of the *.py scripts in deploy-tweaks.d/ folder. Each called script will tweak the domain XML. Finally deploy-tweaks will ssh to the KVM host and pass the tweaked domain XML to the OpenNebula’s native deploy script that will start the VM.

Regarding the configuration variables for your case, you’ll need to define in the VM’s USER TEMPLATE T_CPU_THREADS = 2 and T_CPU_SOCKETS = 2 (and have even number of VCPUs :wink:). The other variables are mostly for internal use to define weird testing VM configurations.
Here is an example of a configuration - these are the VM Attributes:

image

Finally you should restart the reconfigured VM so the tweaked deploy script will kick in.

Hope this helps.

Edit: Please note that you’ll probably need to add some python library packages on the front-end, though.

Best Regards,
Anton Todorov

1 Like

Thanks Anton great job. Looks like exactly what I need. Will give this a try tomorrow.

I can’t get it work and I can’t find anything in the logs that it’s failing. By looking into the script source it seems that it should add lines to the log. Instead nothing new and everything is as before.

On front-end there is this process

19556 ? SNl 0:01 ruby /usr/lib/one/mads/one_vmm_exec.rb -t 15 -r 0 kvm -l deploy=deploy-tweaks

so oned.conf changes should be in effect. I also checked /var/tmp on node, it has the new files.

And /var/lib/one/remotes/vmm/kvm/ looks like this

-rwxr-xr-x  1 oneadmin oneadmin 5.9K Apr  7 15:21 attach_disk
-rwxr-xr-x  1 oneadmin oneadmin 3.9K Apr  7 15:21 attach_nic
-rwxr-xr-x  1 oneadmin oneadmin 1.7K Apr  7 15:21 cancel
-rwxr-xr-x  1 oneadmin oneadmin 2.2K Apr  7 15:21 deploy
-rwxr-xr-x  1 oneadmin oneadmin 4.1K Aug 29 12:15 deploy-tweaks
drwxr-xr-x  2 oneadmin oneadmin 4.0K Aug 29 13:03 deploy-tweaks.d
drwxr-xr-x  2 oneadmin oneadmin 4.0K Aug 29 12:15 deploy-tweaks.d.example
-rwxr-xr-x  1 oneadmin oneadmin 1.7K Apr  7 15:21 detach_disk
-rwxr-xr-x  1 oneadmin oneadmin 1.7K Apr  7 15:21 detach_nic
-rwxr-xr-x  1 oneadmin oneadmin 1.9K Apr  7 15:21 migrate
-rwxr-xr-x  1 oneadmin oneadmin 1.9K Apr  7 15:21 migrate_local
-rwxr-xr-x  1 oneadmin oneadmin  15K Apr  7 15:21 poll
-rwxr-xr-x  1 oneadmin oneadmin 1.6K Apr  7 15:21 prereconfigure
-rwxr-xr-x  1 oneadmin oneadmin 1.4K Apr  7 15:21 reboot
-rwxr-xr-x  1 oneadmin oneadmin 1.6K Apr  7 15:21 reconfigure
-rwxr-xr-x  1 oneadmin oneadmin 1.4K Apr  7 15:21 reset
-rwxr-xr-x  1 oneadmin oneadmin 1.5K Apr  7 15:21 resize_disk
-rwxr-xr-x  1 oneadmin oneadmin 3.4K Apr  7 15:21 restore
-rwxr-xr-x  1 oneadmin oneadmin 2.4K Apr  7 15:21 restore.ceph
-rwxr-xr-x  1 oneadmin oneadmin 2.4K Apr  7 15:21 save
-rwxr-xr-x  1 oneadmin oneadmin 2.6K Apr  7 15:21 save.ceph
-rwxr-xr-x  1 oneadmin oneadmin 2.3K Apr  7 15:21 shutdown
-rwxr-xr-x  1 oneadmin oneadmin 2.4K Apr  7 15:21 snapshot_create
-rwxr-xr-x  1 oneadmin oneadmin 2.1K Apr  7 15:21 snapshot_delete
-rwxr-xr-x  1 oneadmin oneadmin 1.5K Apr  7 15:21 snapshot_revert

I’ve also tested the python script by manually executing it with args deployment.xml and vm.xml and it works. The deployment file is changed as expected.

So everything is in place but it’s not executing the main script and I’m wondering what else to check.

EDIT: Got it working. Front-end was missing libxml2-utils which was causing the script to error. Had to comment out trap to see the actual error in /tmp/

Great. I am working on CentOS mostly, that’s the reason to hint for missing packages :slight_smile:

Best,
Anton

This is awesome nice work !!

Hi,

Thansk @atodorov_storpool for this work, i also faced the same issue on our cluster.

It works very well to set a correct topolgy for the cpu. However it also adds “numa” configuration but if we deploy several VMs on the same node “cpus” ranges are always the same.
Moreover it does not take into account cgroup configuration on hosts

here is an exemple, I create a VM with 4 vCPUS, T_CPU_THREADs=2 and T_CPU_SOCKETS=2:

<cpu mode="host-passthrough">
<topology cores="1" sockets="2" threads="2" />
        <numa><cell cpus="0-1" id="0" memory="4194304" />
                <cell cpus="2-3" id="1" memory="4194304" />
        </numa>
</cpu>

If I create antother VM on the same Host, it will also use same cpu ranges as the python script under deploy-tweaks.d always start at 0.

Regards

Hi @jpfoures,

It works very well to set a correct topolgy for the cpu. However it also adds “numa” configuration but if we deploy several VMs on the same node “cpus” ranges are always the same.

That is expected. Each VM should have cpus starting from 0… The topology element is what is represent as VCPU in the guest VM it is not related to the HOST cpus. Each VCPU has all available in the cpuset/machine.slice/cpuset.cpus. The tool is to configure what is “visible” within the VM so the entire topology is “virtual”. Without setting the “numa” definition there are no “sockets” in the VM’s cpu topology.

Moreover it does not take into account cgroup configuration on hosts

Correct. Pinning the VCPUs to the physical host CPUs is possible but then the live-migrating is complicated(but not impossible). I have no time to check the latest development in one-5.10 but I think the live migration for VMs with pinned cpus is not available at lest for the initial release.

But OpenNebula Conf is soon so I hope to have a chat with the developers how to resolve this limitation.
(and write a resolution via a deploy tweak :wink: )

Best Regards,
Anton Todorov

Hi @atodorov_storpool

thank you for the update. So it means it match exactly our need. We do not want doing a full pinning because, as you said, migration are too complex.

So we will use it as it is now for all VMs.

Waiting for the 5.10 which is implement natively this yes :).

Regards