Error when starting a new VM on specific memory size with NUMA

Community Support
#1

So I have this issue where i can’t start a new VM. The error is

Wed Jan 15 10:40:21 2020 [Z0][VMM][I]: error: Failed to create domain from /var/lib/one//datastores/103/413/deployment.8
Wed Jan 15 10:40:21 2020 [Z0][VMM][I]: error: invalid argument: Failed to parse bitmap ‘’

This occurs only when I set 63GB RAM or above. If I’m under 63GB RAM it works.
It also works when set to 64GB, BUT if i don’t use NUMA and pinning. What I’m thinking is that there is something wrong with detection/calculation of total RAM. This is how it looks when set to 62GB.

image
image931×679 26.5 KB

And this is how it looks when resized to 64GB. Things are empty.

image
image966×612 16.3 KB

So I believe that sunstone submits/calculates something wrong which leads to wrong template.
Host is detected with 66.8 GB Memory total which is correct. I’ve also tried increasing the overcommit limit, but it didn’t help.

Version is 5.10.1

EDIT: sorry, but can’t figure how to remove the bold of that last paragraph :slight_smile:

#2

Can you paste the deployment file for the error situation? You may still have the file (/var/lib/one//datastores/103/413/deployment.8)

Cheers

#3

Just to note that there is a second place where the file could be found - on the frontend in /var/lib/one/vms/413/deployment.8 (in case the VM is terminated).

#4

@atodorov_storpool thanks. It was indeed terminated hard

  <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
        <name>one-413</name>
        <title>test</title>
        <vcpu><![CDATA[40]]></vcpu>
        <cputune>
                <shares>40960</shares>
                <emulatorpin cpuset=''/>
        </cputune>
        <memory>67108864</memory>
        <os>
                <type arch='x86_64'>hvm</type>
        </os>
        <cpu mode='host-passthrough'>
                <topology sockets='2' cores='10' threads='2'/>
                <numa>
                        <cell id='0' memory='33554432' cpus='0-19'/>
                        <cell id='1' memory='33554432' cpus='20-39'/>
                </numa>
        </cpu>
        <devices>
                <emulator><![CDATA[/usr/bin/qemu-system-x86_64]]></emulator>
                <disk type='file' device='cdrom'>
                        <source file='/var/lib/one//datastores/103/413/disk.0'/>
                        <target dev='hda' bus='ide'/>
                        <readonly/>
                        <driver name='qemu' type='raw'/>
                </disk>
                <interface type='bridge'>
                        <source bridge='vmbr5'/>
                        <mac address='02:00:0a:0a:04:01'/>
                        <target dev='one-413-0'/>
                        <model type='virtio'/>
                </interface>
                <graphics type='vnc' listen='0.0.0.0' port='6313'/>
                <input type='tablet' bus='usb'/>
        </devices>
        <features>
                <acpi/>
                <apic/>
        </features>
        <devices>
                <channel type='unix'>
                        <source mode='bind'/><target type='virtio' name='org.qemu.guest_agent.0'/>
                </channel>
        </devices>
        <metadata>
                <one:vm xmlns:one="http://opennebula.org/xmlns/libvirt/1.0">
                        <one:system_datastore><![CDATA[/var/lib/one//datastores/103/413]]></one:system_datastore>
                        <one:name><![CDATA[test]]></one:name>
                        <one:uname><![CDATA[oneadminx509]]></one:uname>
                        <one:uid>42</one:uid>
                        <one:gname><![CDATA[oneadmin]]></one:gname>
                        <one:gid>0</one:gid>
                        <one:opennebula_version>5.10.1</one:opennebula_version>
                        <one:stime>1579075717</one:stime>
                        <one:deployment_time>1579077619</one:deployment_time>
                </one:vm>
        </metadata>
</domain>

And that’s a deployment when it works

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
        <name>one-413</name>
        <title>test</title>
        <vcpu><![CDATA[40]]></vcpu>
        <cputune>
                <shares>40960</shares>
                <vcpupin vcpu='0' cpuset='10'/>
                <vcpupin vcpu='1' cpuset='30'/>
                <vcpupin vcpu='2' cpuset='11'/>
                <vcpupin vcpu='3' cpuset='31'/>
                <vcpupin vcpu='4' cpuset='12'/>
                <vcpupin vcpu='5' cpuset='32'/>
                <vcpupin vcpu='6' cpuset='13'/>
                <vcpupin vcpu='7' cpuset='33'/>
                <vcpupin vcpu='8' cpuset='14'/>
                <vcpupin vcpu='9' cpuset='34'/>
                <vcpupin vcpu='10' cpuset='15'/>
                <vcpupin vcpu='11' cpuset='35'/>
                <vcpupin vcpu='12' cpuset='16'/>
                <vcpupin vcpu='13' cpuset='36'/>
                <vcpupin vcpu='14' cpuset='17'/>
                <vcpupin vcpu='15' cpuset='37'/>
                <vcpupin vcpu='16' cpuset='18'/>
                <vcpupin vcpu='17' cpuset='38'/>
                <vcpupin vcpu='18' cpuset='19'/>
                <vcpupin vcpu='19' cpuset='39'/>
                <vcpupin vcpu='20' cpuset='0'/>
                <vcpupin vcpu='21' cpuset='20'/>
                <vcpupin vcpu='22' cpuset='1'/>
                <vcpupin vcpu='23' cpuset='21'/>
                <vcpupin vcpu='24' cpuset='2'/>
                <vcpupin vcpu='25' cpuset='22'/>
                <vcpupin vcpu='26' cpuset='3'/>
                <vcpupin vcpu='27' cpuset='23'/>
                <vcpupin vcpu='28' cpuset='4'/>
                <vcpupin vcpu='29' cpuset='24'/>
                <vcpupin vcpu='30' cpuset='5'/>
                <vcpupin vcpu='31' cpuset='25'/>
                <vcpupin vcpu='32' cpuset='6'/>
                <vcpupin vcpu='33' cpuset='26'/>
                <vcpupin vcpu='34' cpuset='7'/>
                <vcpupin vcpu='35' cpuset='27'/>
                <vcpupin vcpu='36' cpuset='8'/>
                <vcpupin vcpu='37' cpuset='28'/>
                <vcpupin vcpu='38' cpuset='9'/>
                <vcpupin vcpu='39' cpuset='29'/>
                <emulatorpin cpuset='10,30,11,31,12,32,13,33,14,34,15,35,16,36,17,37,18,38,19,39,0,20,1,21,2,22,3,23,4,24,5,25,6,26,7,27,8,28,9,29'/>
        </cputune>
        <memory>65011712</memory>
        <os>
                <type arch='x86_64'>hvm</type>
        </os>
        <cpu mode='host-passthrough'>
                <topology sockets='2' cores='10' threads='2'/>
                <numa>
                        <cell id='0' memory='32505856' cpus='0-19'/>
                        <cell id='1' memory='32505856' cpus='20-39'/>
                </numa>
        </cpu>
        <numatune>
                <memnode cellid='0' mode='strict' nodeset='1'/>
                <memnode cellid='1' mode='strict' nodeset='0'/>
                <memory mode='strict' nodeset='1,0'/>
        </numatune>
        <devices>
                <emulator><![CDATA[/usr/bin/qemu-system-x86_64]]></emulator>
                <disk type='file' device='cdrom'>
                        <source file='/var/lib/one//datastores/103/413/disk.0'/>
                        <target dev='hda' bus='ide'/>
                        <readonly/>
                        <driver name='qemu' type='raw'/>
                </disk>
                <interface type='bridge'>
                        <source bridge='vmbr5'/>
                        <mac address='02:00:0a:0a:04:01'/>
                        <target dev='one-413-0'/>
                        <model type='virtio'/>
                </interface>
                <graphics type='vnc' listen='0.0.0.0' port='6313'/>
                <input type='tablet' bus='usb'/>
        </devices>
        <features>
                <acpi/>
                <apic/>
        </features>
        <devices>
                <channel type='unix'>
                        <source mode='bind'/><target type='virtio' name='org.qemu.guest_agent.0'/>
                </channel>
        </devices>
        <metadata>
                <one:vm xmlns:one="http://opennebula.org/xmlns/libvirt/1.0">
                        <one:system_datastore><![CDATA[/var/lib/one//datastores/103/413]]></one:system_datastore>
                        <one:name><![CDATA[test]]></one:name>
                        <one:uname><![CDATA[oneadminx509]]></one:uname>
                        <one:uid>42</one:uid>
                        <one:gname><![CDATA[oneadmin]]></one:gname>
                        <one:gid>0</one:gid>
                        <one:opennebula_version>5.10.1</one:opennebula_version>
                        <one:stime>1579075717</one:stime>
                        <one:deployment_time>1579077308</one:deployment_time>
                </one:vm>
        </metadata>
</domain>
#5

YOu mentioned that sunstone maybe submitting a wrong value (e.g. because of an overflow in a variable) can you confirm if this issue is present using just the CLI. onetemplate update to set the memory and onetemplate instantiate?

Thanks

#6

If you are meaning to use onevm resize 417 --memory 64GB, then the issue is exactly the same. I did instantiate with 32GB ram and it worked. Then powered off and resized to 64GB from cli, and then refresh the view in sunstone. I see the same empty screen with cores missing.

I also tried to instantiate from cli and result is also the same.

#7

Thanks for the information, we’ll try to reproduce it locally to see if there is a bug in the NUMA placement logic

#8

We are not able to reproduce this locally. Could you send us onevm show -x one-413 with 63 and 64 GB? THANKS!

#9

63GB also doesn’t work, so here it is with 62GB working

  <ID>419</ID>
  <UID>42</UID>
  <GID>0</GID>
  <UNAME>oneadminx509</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>test</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>0</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <LAST_POLL>1579699475</LAST_POLL>
  <STATE>3</STATE>
  <LCM_STATE>3</LCM_STATE>
  <PREV_STATE>3</PREV_STATE>
  <PREV_LCM_STATE>3</PREV_LCM_STATE>
  <RESCHED>0</RESCHED>
  <STIME>1579699394</STIME>
  <ETIME>0</ETIME>
  <DEPLOY_ID>one-419</DEPLOY_ID>
  <MONITORING>
    <CPU><![CDATA[24.04]]></CPU>
    <DISKRDBYTES><![CDATA[65536]]></DISKRDBYTES>
    <DISKRDIOPS><![CDATA[32]]></DISKRDIOPS>
    <DISKWRBYTES><![CDATA[0]]></DISKWRBYTES>
    <DISKWRIOPS><![CDATA[0]]></DISKWRIOPS>
    <MEMORY><![CDATA[157368]]></MEMORY>
    <NETRX><![CDATA[14071597]]></NETRX>
    <NETTX><![CDATA[1169055]]></NETTX>
    <STATE><![CDATA[a]]></STATE>
  </MONITORING>
  <TEMPLATE>
    <AUTOMATIC_DS_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_DS_REQUIREMENTS>
    <AUTOMATIC_NIC_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_NIC_REQUIREMENTS>
    <AUTOMATIC_REQUIREMENTS><![CDATA[(CLUSTER_ID = 0) & !(PUBLIC_CLOUD = YES) & (PIN_POLICY = PINNED)]]></AUTOMATIC_REQUIREMENTS>
    <CONTEXT>
      <DISK_ID><![CDATA[0]]></DISK_ID>
      <ETH0_CONTEXT_FORCE_IPV4><![CDATA[]]></ETH0_CONTEXT_FORCE_IPV4>
      <ETH0_DNS><![CDATA[8.8.8.8 8.8.4.4]]></ETH0_DNS>
      <ETH0_EXTERNAL><![CDATA[]]></ETH0_EXTERNAL>
      <ETH0_GATEWAY><![CDATA[10.10.0.2]]></ETH0_GATEWAY>
      <ETH0_GATEWAY6><![CDATA[]]></ETH0_GATEWAY6>
      <ETH0_IP><![CDATA[10.10.4.77]]></ETH0_IP>
      <ETH0_IP6><![CDATA[]]></ETH0_IP6>
      <ETH0_IP6_PREFIX_LENGTH><![CDATA[]]></ETH0_IP6_PREFIX_LENGTH>
      <ETH0_IP6_ULA><![CDATA[]]></ETH0_IP6_ULA>
      <ETH0_MAC><![CDATA[02:00:0a:0a:04:4d]]></ETH0_MAC>
      <ETH0_MASK><![CDATA[255.255.248.0]]></ETH0_MASK>
      <ETH0_MTU><![CDATA[]]></ETH0_MTU>
      <ETH0_NETWORK><![CDATA[]]></ETH0_NETWORK>
      <ETH0_SEARCH_DOMAIN><![CDATA[]]></ETH0_SEARCH_DOMAIN>
      <ETH0_VLAN_ID><![CDATA[]]></ETH0_VLAN_ID>
      <ETH0_VROUTER_IP><![CDATA[]]></ETH0_VROUTER_IP>
      <ETH0_VROUTER_IP6><![CDATA[]]></ETH0_VROUTER_IP6>
      <ETH0_VROUTER_MANAGEMENT><![CDATA[]]></ETH0_VROUTER_MANAGEMENT>
      <NETWORK><![CDATA[YES]]></NETWORK>
      <TARGET><![CDATA[hda]]></TARGET>
    </CONTEXT>
    <CPU><![CDATA[40]]></CPU>
    <CPU_MODEL>
      <MODEL><![CDATA[host-passthrough]]></MODEL>
    </CPU_MODEL>
    <FEATURES>
      <ACPI><![CDATA[yes]]></ACPI>
      <APIC><![CDATA[yes]]></APIC>
      <GUEST_AGENT><![CDATA[yes]]></GUEST_AGENT>
    </FEATURES>
    <GRAPHICS>
      <LISTEN><![CDATA[0.0.0.0]]></LISTEN>
      <PORT><![CDATA[6319]]></PORT>
      <TYPE><![CDATA[VNC]]></TYPE>
    </GRAPHICS>
    <INPUT>
      <BUS><![CDATA[usb]]></BUS>
      <TYPE><![CDATA[tablet]]></TYPE>
    </INPUT>
    <MEMORY><![CDATA[63488]]></MEMORY>
    <NIC>
      <AR_ID><![CDATA[0]]></AR_ID>
      <BRIDGE><![CDATA[vmbr5]]></BRIDGE>
      <BRIDGE_TYPE><![CDATA[linux]]></BRIDGE_TYPE>
      <CLUSTER_ID><![CDATA[0]]></CLUSTER_ID>
      <IP><![CDATA[10.10.4.77]]></IP>
      <MAC><![CDATA[02:00:0a:0a:04:4d]]></MAC>
      <MODEL><![CDATA[virtio]]></MODEL>
      <NAME><![CDATA[NIC0]]></NAME>
      <NETWORK><![CDATA[Our VMS]]></NETWORK>
      <NETWORK_ID><![CDATA[121]]></NETWORK_ID>
      <NIC_ID><![CDATA[0]]></NIC_ID>
      <SECURITY_GROUPS><![CDATA[0]]></SECURITY_GROUPS>
      <TARGET><![CDATA[one-419-0]]></TARGET>
      <VN_MAD><![CDATA[bridge]]></VN_MAD>
    </NIC>
    <NIC_DEFAULT>
      <MODEL><![CDATA[virtio]]></MODEL>
    </NIC_DEFAULT>
    <NUMA_NODE>
      <CPUS><![CDATA[10,30,11,31,12,32,13,33,14,34,15,35,16,36,17,37,18,38,19,39]]></CPUS>
      <MEMORY><![CDATA[32505856]]></MEMORY>
      <MEMORY_NODE_ID><![CDATA[1]]></MEMORY_NODE_ID>
      <NODE_ID><![CDATA[1]]></NODE_ID>
      <TOTAL_CPUS><![CDATA[20]]></TOTAL_CPUS>
    </NUMA_NODE>
    <NUMA_NODE>
      <CPUS><![CDATA[0,20,1,21,2,22,3,23,4,24,5,25,6,26,7,27,8,28,9,29]]></CPUS>
      <MEMORY><![CDATA[32505856]]></MEMORY>
      <MEMORY_NODE_ID><![CDATA[0]]></MEMORY_NODE_ID>
      <NODE_ID><![CDATA[0]]></NODE_ID>
      <TOTAL_CPUS><![CDATA[20]]></TOTAL_CPUS>
    </NUMA_NODE>
    <OS>
      <ARCH><![CDATA[x86_64]]></ARCH>
      <BOOT><![CDATA[]]></BOOT>
    </OS>
    <SECURITY_GROUP_RULE>
      <PROTOCOL><![CDATA[ALL]]></PROTOCOL>
      <RULE_TYPE><![CDATA[OUTBOUND]]></RULE_TYPE>
      <SECURITY_GROUP_ID><![CDATA[0]]></SECURITY_GROUP_ID>
      <SECURITY_GROUP_NAME><![CDATA[default]]></SECURITY_GROUP_NAME>
    </SECURITY_GROUP_RULE>
    <SECURITY_GROUP_RULE>
      <PROTOCOL><![CDATA[ALL]]></PROTOCOL>
      <RULE_TYPE><![CDATA[INBOUND]]></RULE_TYPE>
      <SECURITY_GROUP_ID><![CDATA[0]]></SECURITY_GROUP_ID>
      <SECURITY_GROUP_NAME><![CDATA[default]]></SECURITY_GROUP_NAME>
    </SECURITY_GROUP_RULE>
    <TEMPLATE_ID><![CDATA[42]]></TEMPLATE_ID>
    <TM_MAD_SYSTEM><![CDATA[shared]]></TM_MAD_SYSTEM>
    <TOPOLOGY>
      <CORES><![CDATA[10]]></CORES>
      <PIN_POLICY><![CDATA[THREAD]]></PIN_POLICY>
      <SOCKETS><![CDATA[2]]></SOCKETS>
      <THREADS><![CDATA[2]]></THREADS>
    </TOPOLOGY>
    <VCPU><![CDATA[40]]></VCPU>
    <VMID><![CDATA[419]]></VMID>
  </TEMPLATE>
  <USER_TEMPLATE>
    <HYPERVISOR><![CDATA[kvm]]></HYPERVISOR>
    <INPUTS_ORDER><![CDATA[]]></INPUTS_ORDER>
    <LOGO><![CDATA[images/logos/windows8.png]]></LOGO>
    <MEMORY_UNIT_COST><![CDATA[MB]]></MEMORY_UNIT_COST>
  </USER_TEMPLATE>
  <HISTORY_RECORDS>
    <HISTORY>
      <OID>419</OID>
      <SEQ>0</SEQ>
      <HOSTNAME>node3</HOSTNAME>
      <HID>16</HID>
      <CID>0</CID>
      <STIME>1579699423</STIME>
      <ETIME>0</ETIME>
      <VM_MAD><![CDATA[kvm]]></VM_MAD>
      <TM_MAD><![CDATA[shared]]></TM_MAD>
      <DS_ID>103</DS_ID>
      <PSTIME>1579699423</PSTIME>
      <PETIME>1579699423</PETIME>
      <RSTIME>1579699423</RSTIME>
      <RETIME>0</RETIME>
      <ESTIME>0</ESTIME>
      <EETIME>0</EETIME>
      <ACTION>0</ACTION>
      <UID>-1</UID>
      <GID>-1</GID>
      <REQUEST_ID>-1</REQUEST_ID>
    </HISTORY>
  </HISTORY_RECORDS>
</VM>

And 64GB not working

<VM>
  <ID>419</ID>
  <UID>42</UID>
  <GID>0</GID>
  <UNAME>oneadminx509</UNAME>
  <GNAME>oneadmin</GNAME>
  <NAME>test</NAME>
  <PERMISSIONS>
    <OWNER_U>1</OWNER_U>
    <OWNER_M>1</OWNER_M>
    <OWNER_A>0</OWNER_A>
    <GROUP_U>0</GROUP_U>
    <GROUP_M>0</GROUP_M>
    <GROUP_A>0</GROUP_A>
    <OTHER_U>0</OTHER_U>
    <OTHER_M>0</OTHER_M>
    <OTHER_A>0</OTHER_A>
  </PERMISSIONS>
  <LAST_POLL>0</LAST_POLL>
  <STATE>8</STATE>
  <LCM_STATE>0</LCM_STATE>
  <PREV_STATE>8</PREV_STATE>
  <PREV_LCM_STATE>0</PREV_LCM_STATE>
  <RESCHED>0</RESCHED>
  <STIME>1579699394</STIME>
  <ETIME>0</ETIME>
  <DEPLOY_ID>one-419</DEPLOY_ID>
  <MONITORING>
    <CPU><![CDATA[0.0]]></CPU>
    <DISKRDBYTES><![CDATA[65536]]></DISKRDBYTES>
    <DISKRDIOPS><![CDATA[32]]></DISKRDIOPS>
    <DISKWRBYTES><![CDATA[0]]></DISKWRBYTES>
    <DISKWRIOPS><![CDATA[0]]></DISKWRIOPS>
    <DISK_SIZE>
      <ID><![CDATA[0]]></ID>
      <SIZE><![CDATA[1]]></SIZE>
    </DISK_SIZE>
    <MEMORY><![CDATA[0]]></MEMORY>
    <NETRX><![CDATA[131682541]]></NETRX>
    <NETTX><![CDATA[10933685]]></NETTX>
  </MONITORING>
  <TEMPLATE>
    <AUTOMATIC_DS_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_DS_REQUIREMENTS>
    <AUTOMATIC_NIC_REQUIREMENTS><![CDATA[("CLUSTERS/ID" @> 0)]]></AUTOMATIC_NIC_REQUIREMENTS>
    <AUTOMATIC_REQUIREMENTS><![CDATA[(CLUSTER_ID = 0) & !(PUBLIC_CLOUD = YES) & (PIN_POLICY = PINNED)]]></AUTOMATIC_REQUIREMENTS>
    <CONTEXT>
      <DISK_ID><![CDATA[0]]></DISK_ID>
      <ETH0_CONTEXT_FORCE_IPV4><![CDATA[]]></ETH0_CONTEXT_FORCE_IPV4>
      <ETH0_DNS><![CDATA[8.8.8.8 8.8.4.4]]></ETH0_DNS>
      <ETH0_EXTERNAL><![CDATA[]]></ETH0_EXTERNAL>
      <ETH0_GATEWAY><![CDATA[10.10.0.2]]></ETH0_GATEWAY>
      <ETH0_GATEWAY6><![CDATA[]]></ETH0_GATEWAY6>
      <ETH0_IP><![CDATA[10.10.4.77]]></ETH0_IP>
      <ETH0_IP6><![CDATA[]]></ETH0_IP6>
      <ETH0_IP6_PREFIX_LENGTH><![CDATA[]]></ETH0_IP6_PREFIX_LENGTH>
      <ETH0_IP6_ULA><![CDATA[]]></ETH0_IP6_ULA>
      <ETH0_MAC><![CDATA[02:00:0a:0a:04:4d]]></ETH0_MAC>
      <ETH0_MASK><![CDATA[255.255.248.0]]></ETH0_MASK>
      <ETH0_MTU><![CDATA[]]></ETH0_MTU>
      <ETH0_NETWORK><![CDATA[]]></ETH0_NETWORK>
      <ETH0_SEARCH_DOMAIN><![CDATA[]]></ETH0_SEARCH_DOMAIN>
      <ETH0_VLAN_ID><![CDATA[]]></ETH0_VLAN_ID>
      <ETH0_VROUTER_IP><![CDATA[]]></ETH0_VROUTER_IP>
      <ETH0_VROUTER_IP6><![CDATA[]]></ETH0_VROUTER_IP6>
      <ETH0_VROUTER_MANAGEMENT><![CDATA[]]></ETH0_VROUTER_MANAGEMENT>
      <NETWORK><![CDATA[YES]]></NETWORK>
      <TARGET><![CDATA[hda]]></TARGET>
    </CONTEXT>
    <CPU><![CDATA[40]]></CPU>
    <CPU_MODEL>
      <MODEL><![CDATA[host-passthrough]]></MODEL>
    </CPU_MODEL>
    <FEATURES>
      <ACPI><![CDATA[yes]]></ACPI>
      <APIC><![CDATA[yes]]></APIC>
      <GUEST_AGENT><![CDATA[yes]]></GUEST_AGENT>
    </FEATURES>
    <GRAPHICS>
      <LISTEN><![CDATA[0.0.0.0]]></LISTEN>
      <PORT><![CDATA[6319]]></PORT>
      <TYPE><![CDATA[VNC]]></TYPE>
    </GRAPHICS>
    <INPUT>
      <BUS><![CDATA[usb]]></BUS>
      <TYPE><![CDATA[tablet]]></TYPE>
    </INPUT>
    <MEMORY><![CDATA[65536]]></MEMORY>
    <NIC>
      <AR_ID><![CDATA[0]]></AR_ID>
      <BRIDGE><![CDATA[vmbr5]]></BRIDGE>
      <BRIDGE_TYPE><![CDATA[linux]]></BRIDGE_TYPE>
      <CLUSTER_ID><![CDATA[0]]></CLUSTER_ID>
      <IP><![CDATA[10.10.4.77]]></IP>
      <MAC><![CDATA[02:00:0a:0a:04:4d]]></MAC>
      <MODEL><![CDATA[virtio]]></MODEL>
      <NAME><![CDATA[NIC0]]></NAME>
      <NETWORK><![CDATA[Our VMS]]></NETWORK>
      <NETWORK_ID><![CDATA[121]]></NETWORK_ID>
      <NIC_ID><![CDATA[0]]></NIC_ID>
      <SECURITY_GROUPS><![CDATA[0]]></SECURITY_GROUPS>
      <TARGET><![CDATA[one-419-0]]></TARGET>
      <VN_MAD><![CDATA[bridge]]></VN_MAD>
    </NIC>
    <NIC_DEFAULT>
      <MODEL><![CDATA[virtio]]></MODEL>
    </NIC_DEFAULT>
    <NUMA_NODE>
      <MEMORY><![CDATA[33554432]]></MEMORY>
      <TOTAL_CPUS><![CDATA[20]]></TOTAL_CPUS>
    </NUMA_NODE>
    <NUMA_NODE>
      <MEMORY><![CDATA[33554432]]></MEMORY>
      <TOTAL_CPUS><![CDATA[20]]></TOTAL_CPUS>
    </NUMA_NODE>
    <OS>
      <ARCH><![CDATA[x86_64]]></ARCH>
      <BOOT><![CDATA[]]></BOOT>
    </OS>
    <SECURITY_GROUP_RULE>
      <PROTOCOL><![CDATA[ALL]]></PROTOCOL>
      <RULE_TYPE><![CDATA[OUTBOUND]]></RULE_TYPE>
      <SECURITY_GROUP_ID><![CDATA[0]]></SECURITY_GROUP_ID>
      <SECURITY_GROUP_NAME><![CDATA[default]]></SECURITY_GROUP_NAME>
    </SECURITY_GROUP_RULE>
    <SECURITY_GROUP_RULE>
      <PROTOCOL><![CDATA[ALL]]></PROTOCOL>
      <RULE_TYPE><![CDATA[INBOUND]]></RULE_TYPE>
      <SECURITY_GROUP_ID><![CDATA[0]]></SECURITY_GROUP_ID>
      <SECURITY_GROUP_NAME><![CDATA[default]]></SECURITY_GROUP_NAME>
    </SECURITY_GROUP_RULE>
    <TEMPLATE_ID><![CDATA[42]]></TEMPLATE_ID>
    <TM_MAD_SYSTEM><![CDATA[shared]]></TM_MAD_SYSTEM>
    <TOPOLOGY>
      <CORES><![CDATA[10]]></CORES>
      <PIN_POLICY><![CDATA[THREAD]]></PIN_POLICY>
      <SOCKETS><![CDATA[2]]></SOCKETS>
      <THREADS><![CDATA[2]]></THREADS>
    </TOPOLOGY>
    <VCPU><![CDATA[40]]></VCPU>
    <VMID><![CDATA[419]]></VMID>
  </TEMPLATE>
  <USER_TEMPLATE>
    <ERROR><![CDATA[Wed Jan 22 15:35:01 2020 : Error deploying virtual machine: [deploy-tweaks] Could not create domain from /var/lib/one/vms/419/deployment.1]]></ERROR>
    <HYPERVISOR><![CDATA[kvm]]></HYPERVISOR>
    <INPUTS_ORDER><![CDATA[]]></INPUTS_ORDER>
    <LOGO><![CDATA[images/logos/windows8.png]]></LOGO>
    <MEMORY_UNIT_COST><![CDATA[MB]]></MEMORY_UNIT_COST>
  </USER_TEMPLATE>
  <HISTORY_RECORDS>
    <HISTORY>
      <OID>419</OID>
      <SEQ>0</SEQ>
      <HOSTNAME>node3</HOSTNAME>
      <HID>16</HID>
      <CID>0</CID>
      <STIME>1579699423</STIME>
      <ETIME>1579699830</ETIME>
      <VM_MAD><![CDATA[kvm]]></VM_MAD>
      <TM_MAD><![CDATA[shared]]></TM_MAD>
      <DS_ID>103</DS_ID>
      <PSTIME>1579699423</PSTIME>
      <PETIME>1579699423</PETIME>
      <RSTIME>1579699423</RSTIME>
      <RETIME>1579699830</RETIME>
      <ESTIME>0</ESTIME>
      <EETIME>0</EETIME>
      <ACTION>20</ACTION>
      <UID>42</UID>
      <GID>0</GID>
      <REQUEST_ID>6800</REQUEST_ID>
    </HISTORY>
    <HISTORY>
      <OID>419</OID>
      <SEQ>1</SEQ>
      <HOSTNAME>node3</HOSTNAME>
      <HID>16</HID>
      <CID>0</CID>
      <STIME>1579700099</STIME>
      <ETIME>1579700101</ETIME>
      <VM_MAD><![CDATA[kvm]]></VM_MAD>
      <TM_MAD><![CDATA[shared]]></TM_MAD>
      <DS_ID>103</DS_ID>
      <PSTIME>0</PSTIME>
      <PETIME>0</PETIME>
      <RSTIME>1579700099</RSTIME>
      <RETIME>1579700101</RETIME>
      <ESTIME>0</ESTIME>
      <EETIME>0</EETIME>
      <ACTION>11</ACTION>
      <UID>42</UID>
      <GID>0</GID>
      <REQUEST_ID>9152</REQUEST_ID>
    </HISTORY>
  </HISTORY_RECORDS>
</VM>
#10

no luck still :frowning: we are thinking of a different configuration in the host. Could you send onehost show -x 16 (you can remove any sensible data, like hostname etc…) we are interested in the NUMA topology of the host where the VM is being deployed

#11 
<HOST>
  <ID>16</ID>
  <NAME>node3</NAME>
  <STATE>2</STATE>
  <PREV_STATE>2</PREV_STATE>
  <IM_MAD><![CDATA[kvm]]></IM_MAD>
  <VM_MAD><![CDATA[kvm]]></VM_MAD>
  <LAST_MON_TIME>1579776764</LAST_MON_TIME>
  <CLUSTER_ID>0</CLUSTER_ID>
  <CLUSTER>default</CLUSTER>
  <HOST_SHARE>
    <DISK_USAGE>0</DISK_USAGE>
    <MEM_USAGE>33554432</MEM_USAGE>
    <CPU_USAGE>4000</CPU_USAGE>
    <TOTAL_MEM>70070688</TOTAL_MEM>
    <TOTAL_CPU>4000</TOTAL_CPU>
    <MAX_DISK>3221</MAX_DISK>
    <MAX_MEM>70070688</MAX_MEM>
    <MAX_CPU>4000</MAX_CPU>
    <FREE_DISK>1450</FREE_DISK>
    <FREE_MEM>34422628</FREE_MEM>
    <FREE_CPU>2200</FREE_CPU>
    <USED_DISK>1589</USED_DISK>
    <USED_MEM>35648056</USED_MEM>
    <USED_CPU>1800</USED_CPU>
    <RUNNING_VMS>1</RUNNING_VMS>
    <VMS_THREAD>1</VMS_THREAD>
    <DATASTORES/>
    <PCI_DEVICES/>
    <NUMA_NODES>
      <NODE>
        <CORE>
          <CPUS><![CDATA[0:312,20:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[0]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[1:312,21:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[1]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[2:312,22:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[2]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[3:312,23:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[3]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[4:312,24:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[4]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[5:312,25:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[8]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[6:312,26:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[9]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[7:312,27:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[10]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[8:312,28:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[11]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[9:312,29:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[12]]></ID>
        </CORE>
        <HUGEPAGE>
          <FREE><![CDATA[308]]></FREE>
          <PAGES><![CDATA[8500]]></PAGES>
          <SIZE><![CDATA[2048]]></SIZE>
          <USAGE><![CDATA[8192]]></USAGE>
        </HUGEPAGE>
        <HUGEPAGE>
          <FREE><![CDATA[0]]></FREE>
          <PAGES><![CDATA[0]]></PAGES>
          <SIZE><![CDATA[1048576]]></SIZE>
          <USAGE><![CDATA[0]]></USAGE>
        </HUGEPAGE>
        <MEMORY>
          <DISTANCE><![CDATA[0 1]]></DISTANCE>
          <FREE><![CDATA[19195460]]></FREE>
          <TOTAL><![CDATA[37044320]]></TOTAL>
          <USAGE><![CDATA[16777216]]></USAGE>
          <USED><![CDATA[17848860]]></USED>
        </MEMORY>
        <NODE_ID><![CDATA[0]]></NODE_ID>
      </NODE>
      <NODE>
        <CORE>
          <CPUS><![CDATA[10:312,30:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[0]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[11:312,31:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[1]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[12:312,32:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[2]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[13:312,33:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[3]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[14:312,34:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[4]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[15:312,35:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[8]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[16:312,36:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[9]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[17:312,37:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[10]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[18:312,38:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[11]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[19:312,39:312]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[0]]></FREE>
          <ID><![CDATA[12]]></ID>
        </CORE>
        <HUGEPAGE>
          <FREE><![CDATA[308]]></FREE>
          <PAGES><![CDATA[8500]]></PAGES>
          <SIZE><![CDATA[2048]]></SIZE>
          <USAGE><![CDATA[8192]]></USAGE>
        </HUGEPAGE>
        <HUGEPAGE>
          <FREE><![CDATA[0]]></FREE>
          <PAGES><![CDATA[0]]></PAGES>
          <SIZE><![CDATA[1048576]]></SIZE>
          <USAGE><![CDATA[0]]></USAGE>
        </HUGEPAGE>
        <MEMORY>
          <DISTANCE><![CDATA[1 0]]></DISTANCE>
          <FREE><![CDATA[14990172]]></FREE>
          <TOTAL><![CDATA[33026364]]></TOTAL>
          <USAGE><![CDATA[16777216]]></USAGE>
          <USED><![CDATA[18036192]]></USED>
        </MEMORY>
        <NODE_ID><![CDATA[1]]></NODE_ID>
      </NODE>
    </NUMA_NODES>
  </HOST_SHARE>
  <VMS>
    <ID>312</ID>
  </VMS>
  <TEMPLATE>
    <ARCH><![CDATA[x86_64]]></ARCH>
    <CLUSTER_ID><![CDATA[0]]></CLUSTER_ID>
    <CPUSPEED><![CDATA[2600]]></CPUSPEED>
    <HOSTNAME><![CDATA[node3]]></HOSTNAME>
    <HYPERVISOR><![CDATA[kvm]]></HYPERVISOR>
    <IM_MAD><![CDATA[kvm]]></IM_MAD>
    <ISOLCPUS><![CDATA[]]></ISOLCPUS>
    <KVM_CPU_MODEL><![CDATA[IvyBridge-IBRS]]></KVM_CPU_MODEL>
    <KVM_CPU_MODELS><![CDATA[486 pentium pentium2 pentium3 pentiumpro coreduo n270 core2duo qemu32 kvm32 cpu64-rhel5 cpu64-rhel6 kvm64 qemu64 Conroe Penryn Nehalem Nehalem-IBRS Westmere Westmere-IBRS SandyBridge SandyBridge-IBRS IvyBridge IvyBridge-IBRS Haswell-noTSX Haswell-noTSX-IBRS Haswell Haswell-IBRS Broadwell-noTSX Broadwell-noTSX-IBRS Broadwell Broadwell-IBRS Skylake-Client Skylake-Client-IBRS athlon phenom Opteron_G1 Opteron_G2 Opteron_G3 Opteron_G4 Opteron_G5]]></KVM_CPU_MODELS>
    <KVM_MACHINES><![CDATA[pc-i440fx-2.8 pc pc-0.12 pc-i440fx-2.4 pc-1.3 pc-q35-2.7 pc-q35-2.6 xenpv pc-i440fx-1.7 pc-i440fx-1.6 pc-i440fx-2.7 pc-0.11 pc-i440fx-2.3 pc-0.10 pc-1.2 pc-i440fx-2.2 isapc pc-q35-2.5 xenfv pc-0.15 pc-0.14 pc-i440fx-1.5 pc-i440fx-2.6 pc-i440fx-1.4 pc-i440fx-2.5 pc-1.1 pc-i440fx-2.1 pc-q35-2.8 q35 pc-1.0 pc-i440fx-2.0 pc-q35-2.4 pc-0.13]]></KVM_MACHINES>
    <MODELNAME><![CDATA[Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz]]></MODELNAME>
    <NAME><![CDATA[node3]]></NAME>
    <NETRX><![CDATA[5147227085290]]></NETRX>
    <NETTX><![CDATA[2131097346415]]></NETTX>
    <PIN_POLICY><![CDATA[PINNED]]></PIN_POLICY>
    <RESERVED_CPU><![CDATA[]]></RESERVED_CPU>
    <RESERVED_MEM><![CDATA[]]></RESERVED_MEM>
    <VERSION><![CDATA[5.10.1]]></VERSION>
    <VM_MAD><![CDATA[kvm]]></VM_MAD>
  </TEMPLATE>
</HOST>

When I do tests, i set the hugepages to 0, just to be sure that i’m not hitting some other configuration issue. I’ll try hugepages when we get the main issue fixed.

#12

I believe I’ve found a hint to where the issue is coming from. Tried a VM on a server and it worked so here it is a working server

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29
node 0 size: 32154 MB
node 0 free: 14687 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39
node 1 size: 36273 MB
node 1 free: 18724 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

and not working server

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 20 21 22 23 24 25 26 27 28 29
node 0 size: 36176 MB
node 0 free: 18589 MB
node 1 cpus: 10 11 12 13 14 15 16 17 18 19 30 31 32 33 34 35 36 37 38 39
node 1 size: 32252 MB
node 1 free: 14867 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

By the looks of it, when using numa and you assign 64GB, it tries to allocate 32GB from each numa, but it isn’t enough on one of them. When you don’t use numa, it uses the available memory as a whole, which is always enough.