Onprem Cannot schedule VM, there is no suitable host -

Community Support GUI - Sunstone
1

Hello,

Thanks for your time reviewing and responding. I’m new to OpenNebula and I’m currently preforming and evaluation using minione with v6.10.0 for about 2 weeks now. Its been going very well so far. However, I’m having trouble testing one use-case where I select the specific host placement to launch the VM.

I’ve noticed two issues. Issue #1 I’ve worked around, but thought to mention it as possible bug. (as a new user it took me many hours to find the issue/workaround.)

6.10.0 Env to Create a NEW VM and Placement on a specific host.

  • Sunstone web UI.
  • 2 physical host (OnPrem) with Ubuntu22.04
  • New setup, no other VMs on hosts
  • If placement isn’t used then VMs are properly places across the 2 hosts. (so host memory, cpu, datastores all seem fine. (even testing with 3 VMs in one launch spreads them across the two host as expected.)
  • Using Placement for “cluster” works fine too.
  • The issues are only seen with selecting a specific HOST placement

Issue#1 - using AppStore Ubuntu 20.04 template & selecting 1 HOST

Web interface creates "host requirement expression"  automatically as " & (HOST_ID = 20)"

This causes the following failure on launch:

Wed Oct 23 22:15:53 2024: Error in SCHED_REQUIREMENTS: ‘(CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( & (HOST_ID = 20) )’, error: syntax error, unexpected ‘&’, expecting ‘!’ or STRING or ‘(’ at line 1948281953, columns 73:74

The WORKAROUND is to change webUI “host requirement expression” to only “HOST_ID = 20”

I then get further, but stuck at Issue#2

Issue#2 using AppStore Ubuntu 20.04 template & selecting 1 HOST with workaround to issue#1

Always fails with:
Wed Oct 23 22:19:24 2024: Cannot dispatch VM: No host meets capacity and SCHED_REQUIREMENTS: (CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( HOST_ID = 20 )

I’ve spent better part of a day trying to find what I have setup wrong or doing wrong but can’t find it.

The 2 host are ON state. Zero VMs on both. In cluster 117. OnPrem KVM. No Pinning.

By default my host didn’t have attributes PUBLIC_CLOUD or PIN_POLICY. I assume not present would satisfy the “NOT”. I did explicitly add it to host 20, but that didn’t help. I have set scheduler logs to level 5 highest debug level. Still not explicit statement as to what is not matching the criteria to schedule on host.
I tried adding also sorts of other policy statements etc, none helped.

I’ll tried to attached the scheduler log, but as new user to forum I’m blocked per upload error., but here is a clip. I can paste more if needed.
I’ve listed onehost show 19 & 20 below too. (removing only ssh key info and full host name)

Wed Oct 23 21:50:12 2024 [Z0][VM][D]: Pending/rescheduling VM and capacity requirements:
  ACTION       VM  CPU      Memory PCI   System DS  Image DS
------------------------------------------------------------
  DEPLOY       58    8    33554432   0       40960  DS 126: 0 

Wed Oct 23 21:50:12 2024 [Z0][HOST][D]: Discovered Hosts (enabled):
ID          : 19
CLUSTER_ID  : 117
PUBLIC      : 0
MEM_USAGE   : 0
CPU_USAGE   : 0
MAX_MEM     : 527984072
MAX_CPU     : 9600
FREE_DISK   : 80973
RUNNING_VMS : 0

 DSID         FREE_MB 
------------------------------
    1           80973 
  125           80973 
  126           80973 
  127           80973 
  133           80973 

Wed Oct 23 21:50:22 2024 [Z0][HOST][D]: Discovered Hosts (enabled):
ID          : 19
CLUSTER_ID  : 117
PUBLIC      : 0
MEM_USAGE   : 0
CPU_USAGE   : 0
MAX_MEM     : 527984072
MAX_CPU     : 9600
FREE_DISK   : 80973
RUNNING_VMS : 0

 DSID         FREE_MB 
------------------------------
    1           80973 
  125           80973 
  126           80973 
  127           80973 
  133           80973 

Wed Oct 23 21:50:22 2024 [Z0][SCHED][DD]: Setting VM groups placement constraints. Total time: 0.00s
Wed Oct 23 21:50:22 2024 [Z0][SCHED][DD]: Host 19 discarded for VM 58. It does not fulfill SCHED_REQUIREMENTS: (CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( (HYPERVISOR=undefined) & (HOST_ID = 20) )
Wed Oct 23 21:50:22 2024 [Z0][SCHED][DD]: Host 20 discarded for VM 58. It does not fulfill SCHED_REQUIREMENTS: (CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( (HYPERVISOR=undefined) & (HOST_ID = 20) )
Wed Oct 23 21:50:22 2024 [Z0][SCHED][D]: Match-making results for VM 58:
        Cannot schedule VM, there is no suitable host.

Wed Oct 23 21:50:22 2024 [Z0][SCHED][DD]: Match Making statistics:
        Number of VMs:             1
        Total time:                0s
        Total Cluster Match time:  0s
        Total Host Match time:     0.00s
        Total Host Ranking time:   0s
        Total DS Match time:       0s
        Total DS Ranking time:     0s
        Total Network Match time:  0s
        Total Network Ranking time:0s

Wed Oct 23 21:50:22 2024 [Z0][SCHED][DDD]: Scheduling Results:

Wed Oct 23 21:50:22 2024 [Z0][SCHED][DD]: Match scheduled resources, sort by priorities. Total time: 0.01s
Wed Oct 23 21:50:22 2024 [Z0][SCHED][DD]: Call external Scheduler. Total time: 0.00s
Wed Oct 23 21:50:22 2024 [Z0][SCHED][D]: Dispatching VMs to hosts:
        VMID    Priority        Host    System DS
        --------------------------------------------------------------
root@opennebular-1:/etc/one# onevm list
  ID USER     GROUP    NAME                                                                                           STAT  CPU     MEM HOST                                                                  TIME
  60 oneadmin oneadmin test-placement-AppVM                                                                           pend    1    768M                                                                   0d 00h06
root@opennebular-1:/etc/one# onehost list
  ID NAME                                                                                                                                                CLUSTER    TVM      ALLOCATED_CPU      ALLOCATED_MEM STAT
  20 s002-m5-40g-kvm27.mitg-bxb300.cisco.com                                                                                                             onprem-clu   0      0 / 9600 (0%)   0K / 503.5G (0%) on  
  19 s002-m5-40g-kvm38.mitg-bxb300.cisco.com                                                                                                             onprem-clu   0      0 / 9600 (0%)   0K / 503.5G (0%) on  
root@opennebular-1:/etc/one# 
root@opennebular-1:/etc/one# onehost show 20
HOST 20 INFORMATION                                                             
ID                    : 20                  
NAME                  : s002-m5-40g-kvm27

CLUSTER : onprem-cluster-laas-1a
STATE : MONITORED
IM_MAD : kvm
VM_MAD : kvm
LAST MONITORING TIME : 10/23 22:30:04

HOST SHARES
RUNNING VMS : 0
MEMORY
TOTAL : 503.5G
TOTAL +/- RESERVED : 503.5G
USED (REAL) : 3.3G
USED (ALLOCATED) : 0K
CPU
TOTAL : 9600
TOTAL +/- RESERVED : 9600
USED (REAL) : 0
USED (ALLOCATED) : 0

LOCAL SYSTEM DATASTORE #133 CAPACITY
TOTAL: : 97.9G
USED: : 11.9G
FREE: : 81G

MONITORING INFORMATION
ARCH=“x86_64”
CGROUPS_VERSION=“2”
CONTEXT=[
SSH_PUBLIC_KEY=“”
CPUSPEED=“0”
HOSTNAME=“s002-m5-40g-kvm27”
HYPERVISOR=“kvm”
IM_MAD=“kvm”
KVM_CPU_FEATURES=“3dnowprefetch,abm,acpi,adx,aes,apic,arat,arch-capabilities,avx,avx2,avx512-bf16,avx512bw,avx512cd,avx512dq,avx512f,avx512vl,avx512vnni,bmi1,bmi2,clflush,clflushopt,clwb,cmov,cx16,cx8,dca,de,ds,ds_cpl,dtes64,erms,est,f16c,fma,fpu,fsgsbase,fxsr,hle,ht,ibrs-all,intel-pt,invpcid,invtsc,lahf_lm,lm,mca,mce,md-clear,mds-no,mmx,monitor,movbe,mpx,msr,mtrr,nx,pae,pat,pbe,pcid,pclmuldq,pdcm,pdpe1gb,pge,pku,pni,popcnt,pschange-mc-no,pse,pse36,rdctl-no,rdrand,rdseed,rdtscp,rtm,sep,skip-l1dfl-vmentry,smap,smep,smx,spec-ctrl,ss,ssbd,sse,sse2,sse4.1,sse4.2,ssse3,stibp,syscall,taa-no,tm,tm2,tsc,tsc-deadline,tsc_adjust,tsx-ctrl,vme,vmx,x2apic,xgetbv1,xsave,xsavec,xsaveopt,xsaves,xtpr”
KVM_CPU_MODEL=“Cascadelake-Server-noTSX”
KVM_CPU_MODELS=“486 pentium pentium2 pentium3 pentiumpro coreduo n270 core2duo qemu32 kvm32 cpu64-rhel5 cpu64-rhel6 kvm64 Conroe Penryn Nehalem Nehalem-IBRS Westmere Westmere-IBRS SandyBridge SandyBridge-IBRS IvyBridge IvyBridge-IBRS Haswell-noTSX Haswell-noTSX-IBRS Broadwell-noTSX Broadwell-noTSX-IBRS Skylake-Client-noTSX-IBRS Skylake-Server-noTSX-IBRS Cascadelake-Server-noTSX SapphireRapids SapphireRapids-noTSX Opteron_G1”
KVM_MACHINES=“pc-i440fx-jammy ubuntu pc-i440fx-impish-hpb pc-q35-5.2 pc-i440fx-2.12 pc-i440fx-2.0 pc-i440fx-xenial pc-i440fx-6.2 pc pc-q35-4.2 pc-i440fx-2.5 pc-i440fx-4.2 pc-i440fx-focal pc-i440fx-hirsute pc-q35-xenial pc-i440fx-jammy-hpb pc-i440fx-5.2 pc-i440fx-1.5 pc-q35-2.7 pc-q35-eoan-hpb pc-i440fx-zesty pc-i440fx-disco-hpb pc-q35-groovy pc-i440fx-groovy pc-q35-artful pc-i440fx-2.2 pc-i440fx-trusty pc-i440fx-eoan-hpb pc-q35-focal-hpb pc-q35-jammy-maxcpus pc-q35-bionic-hpb pc-i440fx-artful pc-i440fx-2.7 pc-q35-6.1 pc-i440fx-jammy-maxcpus pc-i440fx-yakkety pc-q35-2.4 pc-q35-cosmic-hpb pc-q35-2.10 x-remote pc-i440fx-1.7 pc-q35-5.1 pc-q35-2.9 pc-i440fx-2.11 pc-i440fx-jammy-hpb-maxcpus pc-q35-3.1 pc-i440fx-6.1 pc-q35-4.1 pc-q35-jammy ubuntu-q35 pc-i440fx-2.4 pc-i440fx-4.1 pc-q35-eoan pc-q35-jammy-hpb pc-i440fx-5.1 pc-i440fx-2.9 pc-i440fx-bionic-hpb isapc pc-i440fx-1.4 pc-q35-cosmic pc-q35-2.6 pc-i440fx-3.1 pc-q35-bionic pc-q35-disco-hpb pc-i440fx-cosmic pc-q35-2.12 pc-i440fx-bionic pc-q35-groovy-hpb pc-q35-disco pc-i440fx-cosmic-hpb pc-i440fx-2.1 pc-i440fx-wily pc-q35-impish pc-q35-6.0 pc-i440fx-impish pc-i440fx-2.6 pc-q35-impish-hpb pc-q35-hirsute pc-q35-4.0.1 pc-q35-hirsute-hpb pc-i440fx-1.6 pc-q35-5.0 pc-q35-2.8 pc-i440fx-2.10 pc-q35-3.0 pc-i440fx-6.0 pc-q35-zesty pc-q35-4.0 pc-q35-focal microvm pc-i440fx-2.3 pc-q35-jammy-hpb-maxcpus pc-i440fx-focal-hpb pc-i440fx-disco pc-i440fx-4.0 pc-i440fx-groovy-hpb pc-i440fx-hirsute-hpb pc-i440fx-5.0 pc-i440fx-2.8 pc-q35-6.2 q35 pc-i440fx-eoan pc-q35-2.5 pc-i440fx-3.0 pc-q35-yakkety pc-q35-2.11”
MODELNAME=“Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz”
PIN_POLICY=“NONE”
PROVISION=[
ANSIBLE_PLAYBOOK=“onprem”,
COUNT=“1”,
HOSTNAME=“s002-m5-40g-kvm27
ID=“20”,
INDEX=“1”,
PROVIDER_NAME=“onprem” ]
PROVISION_CONFIGURATION_BASE64=“LS0tCm9uZXByb3Zpc2lvbl9oeXBlcnZpc29yOiBrdm0Kb3Blbm5lYnVsYV9zc2hfZGVwbG95X3ByaXZhdGVfa2V5OiB0cnVlCm9uZXByb3Zpc2lvbl9wcml2YXRlX3BoeWRldjogZW5wOTRzMGYxCg==”
PROVISION_CONNECTION=[
PRIVATE_KEY=”/var/lib/one/.ssh-oneprovision/id_rsa",
PUBLIC_KEY=“/var/lib/one/.ssh-oneprovision/id_rsa.pub”,
REMOTE_PORT=“22”,
REMOTE_USER=“root” ]
PUBLIC_CLOUD=“NO”
VERSION=“6.10.0”
VM_MAD=“kvm”

NUMA NODES

ID CORES USED FREE
0 – – – – – – – – – – – – – – – – – – – – – – – – 0 48
1 – – – – – – – – – – – – – – – – – – – – – – – – 0 48

NUMA MEMORY

NODE_ID TOTAL USED_REAL USED_ALLOCATED FREE
0 251.6G 1.7G 0K 249.9G
1 251.9G 3.7G 0K 248.2G

NUMA HUGEPAGES

NODE_ID SIZE TOTAL FREE USED
0 2M 0 0 0
0 1024M 0 0 0
1 2M 0 0 0
1 1024M 0 0 0

WILD VIRTUAL MACHINES

NAME IMPORT_ID CPU MEMORY

VIRTUAL MACHINES

ID USER GROUP NAME STAT CPU MEM HOST TIME
root@opennebular-1:/etc/one#
opennebular-1:/etc/one# onehost show 19
HOST 19 INFORMATION
ID : 19
NAME : s002-m5-40g-kvm38
CLUSTER : onprem-cluster-laas-1a
STATE : MONITORED
IM_MAD : kvm
VM_MAD : kvm
LAST MONITORING TIME : 10/23 22:32:26

HOST SHARES
RUNNING VMS : 0
MEMORY
TOTAL : 503.5G
TOTAL +/- RESERVED : 503.5G
USED (REAL) : 3.3G
USED (ALLOCATED) : 0K
CPU
TOTAL : 9600
TOTAL +/- RESERVED : 9600
USED (REAL) : 0
USED (ALLOCATED) : 0

LOCAL SYSTEM DATASTORE #1 CAPACITY
TOTAL: : 97.9G
USED: : 13.8G
FREE: : 79.1G

LOCAL SYSTEM DATASTORE #125 CAPACITY
TOTAL: : 97.9G
USED: : 13.8G
FREE: : 79.1G

LOCAL SYSTEM DATASTORE #126 CAPACITY
TOTAL: : 97.9G
USED: : 13.8G
FREE: : 79.1G

LOCAL SYSTEM DATASTORE #127 CAPACITY
TOTAL: : 97.9G
USED: : 13.8G
FREE: : 79.1G

LOCAL SYSTEM DATASTORE #133 CAPACITY
TOTAL: : 97.9G
USED: : 13.8G
FREE: : 79.1G

MONITORING INFORMATION
ARCH=“x86_64”
CGROUPS_VERSION=“2”
CONTEXT=[
SSH_PUBLIC_KEY=“removed by me”
CPUSPEED=“0”
HOSTNAME=“s002-m5-40g-kvm38”
HYPERVISOR=“kvm”
IM_MAD=“kvm”
KVM_CPU_FEATURES=“3dnowprefetch,abm,acpi,adx,aes,apic,arat,arch-capabilities,avx,avx2,avx512-bf16,avx512bw,avx512cd,avx512dq,avx512f,avx512vl,avx512vnni,bmi1,bmi2,clflush,clflushopt,clwb,cmov,cx16,cx8,dca,de,ds,ds_cpl,dtes64,erms,est,f16c,fma,fpu,fsgsbase,fxsr,hle,ht,ibrs-all,intel-pt,invpcid,invtsc,lahf_lm,lm,mca,mce,md-clear,mds-no,mmx,monitor,movbe,mpx,msr,mtrr,nx,pae,pat,pbe,pcid,pclmuldq,pdcm,pdpe1gb,pge,pku,pni,popcnt,pschange-mc-no,pse,pse36,rdctl-no,rdrand,rdseed,rdtscp,rtm,sep,skip-l1dfl-vmentry,smap,smep,smx,spec-ctrl,ss,ssbd,sse,sse2,sse4.1,sse4.2,ssse3,stibp,syscall,taa-no,tm,tm2,tsc,tsc-deadline,tsc_adjust,tsx-ctrl,vme,vmx,x2apic,xgetbv1,xsave,xsavec,xsaveopt,xsaves,xtpr”
KVM_CPU_MODEL=“Cascadelake-Server-noTSX”
KVM_CPU_MODELS=“486 pentium pentium2 pentium3 pentiumpro coreduo n270 core2duo qemu32 kvm32 cpu64-rhel5 cpu64-rhel6 kvm64 Conroe Penryn Nehalem Nehalem-IBRS Westmere Westmere-IBRS SandyBridge SandyBridge-IBRS IvyBridge IvyBridge-IBRS Haswell-noTSX Haswell-noTSX-IBRS Broadwell-noTSX Broadwell-noTSX-IBRS Skylake-Client-noTSX-IBRS Skylake-Server-noTSX-IBRS Cascadelake-Server-noTSX SapphireRapids SapphireRapids-noTSX Opteron_G1”
KVM_MACHINES=“pc-i440fx-jammy ubuntu pc-i440fx-impish-hpb pc-q35-5.2 pc-i440fx-2.12 pc-i440fx-2.0 pc-i440fx-xenial pc-i440fx-6.2 pc pc-q35-4.2 pc-i440fx-2.5 pc-i440fx-4.2 pc-i440fx-focal pc-i440fx-hirsute pc-q35-xenial pc-i440fx-jammy-hpb pc-i440fx-5.2 pc-i440fx-1.5 pc-q35-2.7 pc-q35-eoan-hpb pc-i440fx-zesty pc-i440fx-disco-hpb pc-q35-groovy pc-i440fx-groovy pc-q35-artful pc-i440fx-2.2 pc-i440fx-trusty pc-i440fx-eoan-hpb pc-q35-focal-hpb pc-q35-jammy-maxcpus pc-q35-bionic-hpb pc-i440fx-artful pc-i440fx-2.7 pc-q35-6.1 pc-i440fx-jammy-maxcpus pc-i440fx-yakkety pc-q35-2.4 pc-q35-cosmic-hpb pc-q35-2.10 x-remote pc-i440fx-1.7 pc-q35-5.1 pc-q35-2.9 pc-i440fx-2.11 pc-i440fx-jammy-hpb-maxcpus pc-q35-3.1 pc-i440fx-6.1 pc-q35-4.1 pc-q35-jammy ubuntu-q35 pc-i440fx-2.4 pc-i440fx-4.1 pc-q35-eoan pc-q35-jammy-hpb pc-i440fx-5.1 pc-i440fx-2.9 pc-i440fx-bionic-hpb isapc pc-i440fx-1.4 pc-q35-cosmic pc-q35-2.6 pc-i440fx-3.1 pc-q35-bionic pc-q35-disco-hpb pc-i440fx-cosmic pc-q35-2.12 pc-i440fx-bionic pc-q35-groovy-hpb pc-q35-disco pc-i440fx-cosmic-hpb pc-i440fx-2.1 pc-i440fx-wily pc-q35-impish pc-q35-6.0 pc-i440fx-impish pc-i440fx-2.6 pc-q35-impish-hpb pc-q35-hirsute pc-q35-4.0.1 pc-q35-hirsute-hpb pc-i440fx-1.6 pc-q35-5.0 pc-q35-2.8 pc-i440fx-2.10 pc-q35-3.0 pc-i440fx-6.0 pc-q35-zesty pc-q35-4.0 pc-q35-focal microvm pc-i440fx-2.3 pc-q35-jammy-hpb-maxcpus pc-i440fx-focal-hpb pc-i440fx-disco pc-i440fx-4.0 pc-i440fx-groovy-hpb pc-i440fx-hirsute-hpb pc-i440fx-5.0 pc-i440fx-2.8 pc-q35-6.2 q35 pc-i440fx-eoan pc-q35-2.5 pc-i440fx-3.0 pc-q35-yakkety pc-q35-2.11”
MODELNAME=“Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz”
PROVISION=[
ANSIBLE_PLAYBOOK=“onprem”,
COUNT=“1”,
HOSTNAME=“s002-m5-40g-kvm38.mitg-bxb300.cisco.com”,
ID=“20”,
INDEX=“0”,
PROVIDER_NAME=“onprem” ]
PROVISION_CONFIGURATION_BASE64=“LS0tCm9uZXByb3Zpc2lvbl9oeXBlcnZpc29yOiBrdm0Kb3Blbm5lYnVsYV9zc2hfZGVwbG95X3ByaXZhdGVfa2V5OiB0cnVlCm9uZXByb3Zpc2lvbl9wcml2YXRlX3BoeWRldjogZW5wOTRzMGYxCg==”
PROVISION_CONNECTION=[
PRIVATE_KEY=“/var/lib/one/.ssh-oneprovision/id_rsa”,
PUBLIC_KEY=“/var/lib/one/.ssh-oneprovision/id_rsa.pub”,
REMOTE_PORT=“22”,
REMOTE_USER=“root” ]
RESERVED_CPU=“”
RESERVED_MEM=“”
VERSION=“6.10.0”
VM_MAD=“kvm”

NUMA NODES

ID CORES USED FREE
0 – – – – – – – – – – – – – – – – – – – – – – – – 0 48
1 – – – – – – – – – – – – – – – – – – – – – – – – 0 48

NUMA MEMORY

NODE_ID TOTAL USED_REAL USED_ALLOCATED FREE
0 251.6G 4.2G 0K 247.4G
1 251.9G 1.6G 0K 250.3G

NUMA HUGEPAGES

NODE_ID SIZE TOTAL FREE USED
0 2M 0 0 0
0 1024M 0 0 0
1 2M 0 0 0
1 1024M 0 0 0

WILD VIRTUAL MACHINES

NAME IMPORT_ID CPU MEMORY

VIRTUAL MACHINES

ID USER GROUP NAME STAT CPU MEM HOST TIME
root@opennebular-1:/etc/one#

2

Hello,
Please take a look at the following bold parts

Wed Oct 23 21:50:22 2024 [Z0][SCHED][DD]: Host 20 discarded for VM 58. It does not fulfill SCHED_REQUIREMENTS: (CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( (HYPERVISOR=undefined) & (HOST_ID = 20) )

and

root@opennebular-1:/etc/one# onehost show 20
…
HYPERVISOR=“kvm”
…
opennebular-1:/etc/one# onehost show 19
HYPERVISOR=“kvm”

So, you should select a hypervisor for the VM to be KVM for hosts matching to succeed or figure-out how to remove the hypervisor from the SCHED_REQUIREMENTS

Hope this helps,
Anton Todorov

3

Hi Anton,

Thanks for the prompt response! I thought I had only included the logs from the last run, but seems I pasted some older ones.

There is a 3rd WebUI issue I was leaving off; in some instances selecting a placment host the “host requirement expression” is automatically set to
( (HYPERVISOR=undefined) & (HOST_ID = 20) )

The first few times I missed it. Later in those tests I changed undefined to kvm manually at the web UI and the same issue came.

Here are complete/clean logs running again just now.


Thu Oct 24 13:30:51 2024 [Z0][VM][D]: Pending/rescheduling VM and capacity requirements:                                                                                                                          
  ACTION       VM  CPU      Memory PCI   System DS  Image DS                                                                                                                                                      
------------------------------------------------------------                                                                                                                                                      
  DEPLOY       62    8    33554432   0       40960  DS 126: 0                                                                                                                                                     

Thu Oct 24 13:30:51 2024 [Z0][HOST][D]: Discovered Hosts (enabled):                                                                                                                                               
ID          : 19                                                                                                                                                                                                  
CLUSTER_ID  : 117                                                                                                                                                                                                 
PUBLIC      : 0                                                                                                                                                                                                   
MEM_USAGE   : 0                                                                                                                                                                                                   
CPU_USAGE   : 0                                                                                                                                                                                                   
MAX_MEM     : 527984072                                                                                                                                                                                           
MAX_CPU     : 9600                                                                                                                                                                                                
FREE_DISK   : 80972                                                                                                                                                                                               
RUNNING_VMS : 0                                                                                                                                                                                                   

 DSID         FREE_MB                                                                                                                                                                                             
------------------------------                                                                                                                                                                                    
    1           80972                                                                                                                                                                                             
  125           80972                                                                                                                                                                                             
  126           80972                                                                                                                                                                                             
  127           80972                                                                                                                                                                                             
  133           80972                                                                                                                                                                                             

    PCI ADDRESS    CLASS   VENDOR   DEVICE     VMID                                                                                                                                                               
-------------------------------------------------------                                                                                                                                                           

---------------------------------------------------------------------------                                                                                                                                       
Node: 0 Memory: 0/251G                                                                                                                                                                                            
---------------------------------------------------------------------------                                                                                                                                       
( 0 48)( 1 49)( 2 50)( 3 51)( 4 52)( 5 53)( 6 54)( 7 55)( 8 56)( 9 57)(10 58)(11 59)(12 60)(13 61)(14 62)(15 63)(16 64)(17 65)(18 66)(19 67)(20 68)(21 69)(22 70)(23 71)                                          
( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)                                          
---------------------------------------------------------------------------                                                                                                                                       
Node: 1 Memory: 0/251G                                                                                                                                                                                            
---------------------------------------------------------------------------                                                                                                                                       
(24 72)(25 73)(26 74)(27 75)(28 76)(29 77)(30 78)(31 79)(32 80)(33 81)(34 82)(35 83)(36 84)(37 85)(38 86)(39 87)(40 88)(41 89)(42 90)(43 91)(44 92)(45 93)(46 94)(47 95)                                          
( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)                                          
---------------------------------------------------------------------------                                                                                                                                       

ID          : 20                                                                                                                                                                                                  
CLUSTER_ID  : 117                                                                                                                                                                                                 
PUBLIC      : 0                                                                                                                                                                                                   
MEM_USAGE   : 0                                                                                                                                                                                                   
CPU_USAGE   : 0                                                                                                                                                                                                   
MAX_MEM     : 527984044                                                                                                                                                                                           
MAX_CPU     : 9600                                                                                                                                                                                                
FREE_DISK   : 82898                                                                                                                                                                                               
RUNNING_VMS : 0                                                                                                                                                                                                   

 DSID         FREE_MB                                                                                                                                                                                             
------------------------------                                                                                                                                                                                    
  133           82898                                                                                                                                                                                             

    PCI ADDRESS    CLASS   VENDOR   DEVICE     VMID                                                                                                                                                               
-------------------------------------------------------                                                                                                                                                           

---------------------------------------------------------------------------                                                                                                                                       
Node: 0 Memory: 0/251G                                                                                                                                                                                            
---------------------------------------------------------------------------                                                                                                                                       
( 0 48)( 1 49)( 2 50)( 3 51)( 4 52)( 5 53)( 6 54)( 7 55)( 8 56)( 9 57)(10 58)(11 59)(12 60)(13 61)(14 62)(15 63)(16 64)(17 65)(18 66)(19 67)(20 68)(21 69)(22 70)(23 71)                                          
( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)                                          
---------------------------------------------------------------------------                                                                                                                                       
Node: 1 Memory: 0/251G                                                                                                                                                                                            
---------------------------------------------------------------------------                                                                                                                                       
(24 72)(25 73)(26 74)(27 75)(28 76)(29 77)(30 78)(31 79)(32 80)(33 81)(34 82)(35 83)(36 84)(37 85)(38 86)(39 87)(40 88)(41 89)(42 90)(43 91)(44 92)(45 93)(46 94)(47 95)                                          
( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)                                          
---------------------------------------------------------------------------                                                                                                                                       


Thu Oct 24 13:30:51 2024 [Z0][VM][D]: VMs in VMGroups:                                                                                                                                                            

Thu Oct 24 13:30:51 2024 [Z0][VNET][D]: Discovered VNETS:                                                                                                                                                         
      ID   Leases                                                                                                                                                                                                 
--------------------                                                                                                                                                                                              
      15        9                                                                                                                                                                                                 
      18        8                                                                                                                                                                                                 

Thu Oct 24 13:30:51 2024 [Z0][SCHED][DD]: Getting VM and Host information. Total time: 0.04s                                                                                                                      
Thu Oct 24 13:30:51 2024 [Z0][VMGRP][DDD]: VM Group Scheduling information
********************************************************************************
SCHEDULING RESULTS FOR VM GROUP 0, Smoke1
********************************************************************************
ROLE ID NAME     POLICY       VMS
--------------------------------------------------------------------------------
0       smokeTesting None         

RULES
--------------------------------------------------------------------------------

                                                                                                                                                                                                        
--------------------------------------------------------------------------------
Intra-role affinity requirements
VMID     REQUIREMENTS
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Inter-role affinity requirements
VMID     REQUIREMENTS
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Intra-role Anti-affinity rules 
ROLE     VM       ANTI_AFFINITY REQUIRMENTS
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Inter-role Anti-affinity rules 
ROLE     VM       ANTI_AFFINITY REQUIRMENTS
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Host affinity rules 
ROLE     VM       AFFINITY REQUIRMENTS
--------------------------------------------------------------------------------

Thu Oct 24 13:30:51 2024 [Z0][SCHED][DD]: Setting VM groups placement constraints. Total time: 0.00s
Thu Oct 24 13:30:51 2024 [Z0][SCHED][DD]: Host 19 discarded for VM 62. It does not fulfill SCHED_REQUIREMENTS: (CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( (HYPERVISOR = kvm) & (HOST_ID = 20) )
Thu Oct 24 13:30:51 2024 [Z0][SCHED][DD]: Host 20 discarded for VM 62. It does not fulfill SCHED_REQUIREMENTS: (CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( (HYPERVISOR = kvm) & (HOST_ID = 20) )                                                                                                                                                                                  
Thu Oct 24 13:30:51 2024 [Z0][SCHED][D]: Match-making results for VM 62:
        Cannot schedule VM, there is no suitable host.

Thu Oct 24 13:30:51 2024 [Z0][SCHED][DD]: Match Making statistics:
        Number of VMs:             1
        Total time:                0s
        Total Cluster Match time:  0s
        Total Host Match time:     0.00s
        Total Host Ranking time:   0s
        Total DS Match time:       0s
        Total DS Ranking time:     0s
        Total Network Match time:  0s
        Total Network Ranking time:0s

Thu Oct 24 13:30:51 2024 [Z0][SCHED][DDD]: Scheduling Results:

Thu Oct 24 13:30:51 2024 [Z0][SCHED][DD]: Match scheduled resources, sort by priorities. Total time: 0.01s
Thu Oct 24 13:30:51 2024 [Z0][SCHED][DD]: Call external Scheduler. Total time: 0.00s
Thu Oct 24 13:30:51 2024 [Z0][SCHED][D]: Dispatching VMs to hosts:
        VMID    Priority        Host    System DS
        --------------------------------------------------------------

Thu Oct 24 13:30:51 2024 [Z0][SCHED][DD]: Dispatching VMs to hosts. Total time: 0.00s



oneadmin@opennebular-1:/var/log/one$ onehost list 
  ID NAME                                                                                                                                                CLUSTER    TVM      ALLOCATED_CPU      ALLOCATED_MEM STAT
  20 s002-m5-40g-kvm27                                                                                                             onprem-clu   0      0 / 9600 (0%)   0K / 503.5G (0%) on  
  19 s002-m5-40g-kvm38                                                                                                             onprem-clu   0      0 / 9600 (0%)   0K / 503.5G (0%) on  
oneadmin@opennebular-1:/var/log/one$ onehost show 20
HOST 20 INFORMATION                                                             
ID                    : 20                  
NAME                  : s002-m5-40g-kvm27
CLUSTER               : onprem-cluster-laas-1a
STATE                 : MONITORED           
IM_MAD                : kvm                 
VM_MAD                : kvm                 
LAST MONITORING TIME  : 10/24 13:23:38      

HOST SHARES                                                                     
RUNNING VMS           : 0                   
MEMORY                                                                          
  TOTAL               : 503.5G              
  TOTAL +/- RESERVED  : 503.5G              
  USED (REAL)         : 3.3G                
  USED (ALLOCATED)    : 0K                  
CPU                                                                             
  TOTAL               : 9600                
  TOTAL +/- RESERVED  : 9600                
  USED (REAL)         : 0                   
  USED (ALLOCATED)    : 0                   

LOCAL SYSTEM DATASTORE #133 CAPACITY                                            
TOTAL:                : 97.9G               
USED:                 : 11.9G               
FREE:                 : 81G                 

MONITORING INFORMATION                                                          
ARCH="x86_64"
CGROUPS_VERSION="2"
CONTEXT=[
  SSH_PUBLIC_KEY="ssh-rsa " ]
CPUSPEED="0"
HOSTNAME="s002-m5-40g-kvm27"
HYPERVISOR="kvm"
IM_MAD="kvm"
KVM_CPU_FEATURES="3dnowprefetch,abm,acpi,adx,aes,apic,arat,arch-capabilities,avx,avx2,avx512-bf16,avx512bw,avx512cd,avx512dq,avx512f,avx512vl,avx512vnni,bmi1,bmi2,clflush,clflushopt,clwb,cmov,cx16,cx8,dca,de,ds,ds_cpl,dtes64,erms,est,f16c,fma,fpu,fsgsbase,fxsr,hle,ht,ibrs-all,intel-pt,invpcid,invtsc,lahf_lm,lm,mca,mce,md-clear,mds-no,mmx,monitor,movbe,mpx,msr,mtrr,nx,pae,pat,pbe,pcid,pclmuldq,pdcm,pdpe1gb,pge,pku,pni,popcnt,pschange-mc-no,pse,pse36,rdctl-no,rdrand,rdseed,rdtscp,rtm,sep,skip-l1dfl-vmentry,smap,smep,smx,spec-ctrl,ss,ssbd,sse,sse2,sse4.1,sse4.2,ssse3,stibp,syscall,taa-no,tm,tm2,tsc,tsc-deadline,tsc_adjust,tsx-ctrl,vme,vmx,x2apic,xgetbv1,xsave,xsavec,xsaveopt,xsaves,xtpr"
KVM_CPU_MODEL="Cascadelake-Server-noTSX"
KVM_CPU_MODELS="486 pentium pentium2 pentium3 pentiumpro coreduo n270 core2duo qemu32 kvm32 cpu64-rhel5 cpu64-rhel6 kvm64 Conroe Penryn Nehalem Nehalem-IBRS Westmere Westmere-IBRS SandyBridge SandyBridge-IBRS IvyBridge IvyBridge-IBRS Haswell-noTSX Haswell-noTSX-IBRS Broadwell-noTSX Broadwell-noTSX-IBRS Skylake-Client-noTSX-IBRS Skylake-Server-noTSX-IBRS Cascadelake-Server-noTSX SapphireRapids SapphireRapids-noTSX Opteron_G1"
KVM_MACHINES="pc-i440fx-jammy ubuntu pc-i440fx-impish-hpb pc-q35-5.2 pc-i440fx-2.12 pc-i440fx-2.0 pc-i440fx-xenial pc-i440fx-6.2 pc pc-q35-4.2 pc-i440fx-2.5 pc-i440fx-4.2 pc-i440fx-focal pc-i440fx-hirsute pc-q35-xenial pc-i440fx-jammy-hpb pc-i440fx-5.2 pc-i440fx-1.5 pc-q35-2.7 pc-q35-eoan-hpb pc-i440fx-zesty pc-i440fx-disco-hpb pc-q35-groovy pc-i440fx-groovy pc-q35-artful pc-i440fx-2.2 pc-i440fx-trusty pc-i440fx-eoan-hpb pc-q35-focal-hpb pc-q35-jammy-maxcpus pc-q35-bionic-hpb pc-i440fx-artful pc-i440fx-2.7 pc-q35-6.1 pc-i440fx-jammy-maxcpus pc-i440fx-yakkety pc-q35-2.4 pc-q35-cosmic-hpb pc-q35-2.10 x-remote pc-i440fx-1.7 pc-q35-5.1 pc-q35-2.9 pc-i440fx-2.11 pc-i440fx-jammy-hpb-maxcpus pc-q35-3.1 pc-i440fx-6.1 pc-q35-4.1 pc-q35-jammy ubuntu-q35 pc-i440fx-2.4 pc-i440fx-4.1 pc-q35-eoan pc-q35-jammy-hpb pc-i440fx-5.1 pc-i440fx-2.9 pc-i440fx-bionic-hpb isapc pc-i440fx-1.4 pc-q35-cosmic pc-q35-2.6 pc-i440fx-3.1 pc-q35-bionic pc-q35-disco-hpb pc-i440fx-cosmic pc-q35-2.12 pc-i440fx-bionic pc-q35-groovy-hpb pc-q35-disco pc-i440fx-cosmic-hpb pc-i440fx-2.1 pc-i440fx-wily pc-q35-impish pc-q35-6.0 pc-i440fx-impish pc-i440fx-2.6 pc-q35-impish-hpb pc-q35-hirsute pc-q35-4.0.1 pc-q35-hirsute-hpb pc-i440fx-1.6 pc-q35-5.0 pc-q35-2.8 pc-i440fx-2.10 pc-q35-3.0 pc-i440fx-6.0 pc-q35-zesty pc-q35-4.0 pc-q35-focal microvm pc-i440fx-2.3 pc-q35-jammy-hpb-maxcpus pc-i440fx-focal-hpb pc-i440fx-disco pc-i440fx-4.0 pc-i440fx-groovy-hpb pc-i440fx-hirsute-hpb pc-i440fx-5.0 pc-i440fx-2.8 pc-q35-6.2 q35 pc-i440fx-eoan pc-q35-2.5 pc-i440fx-3.0 pc-q35-yakkety pc-q35-2.11"
MODELNAME="Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz"
PIN_POLICY="NONE"
PROVISION=[
  ANSIBLE_PLAYBOOK="onprem",
  COUNT="1",
  HOSTNAME="s002-m5-40g-kvm27",
  ID="20",
  INDEX="1",
  PROVIDER_NAME="onprem" ]
PROVISION_CONFIGURATION_BASE64="LS0tCm9uZXByb3Zpc2lvbl9oeXBlcnZpc29yOiBrdm0Kb3Blbm5lYnVsYV9zc2hfZGVwbG95X3ByaXZhdGVfa2V5OiB0cnVlCm9uZXByb3Zpc2lvbl9wcml2YXRlX3BoeWRldjogZW5wOTRzMGYxCg=="
PROVISION_CONNECTION=[
  PRIVATE_KEY="/var/lib/one/.ssh-oneprovision/id_rsa",
  PUBLIC_KEY="/var/lib/one/.ssh-oneprovision/id_rsa.pub",
  REMOTE_PORT="22",
  REMOTE_USER="root" ]
PUBLIC_CLOUD="NO"
VERSION="6.10.0"
VM_MAD="kvm"

NUMA NODES

  ID CORES                                                                    USED FREE
   0 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --  0    48
   1 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --  0    48

NUMA MEMORY

 NODE_ID TOTAL    USED_REAL            USED_ALLOCATED       FREE    
       0 251.6G   1.7G                 0K                   249.9G
       1 251.9G   3.7G                 0K                   248.3G

NUMA HUGEPAGES

 NODE_ID SIZE     TOTAL    FREE     USED    
       0 2M       0        0        0
       0 1024M    0        0        0
       1 2M       0        0        0
       1 1024M    0        0        0

WILD VIRTUAL MACHINES

NAME                                                      IMPORT_ID  CPU     MEMORY

VIRTUAL MACHINES

  ID USER     GROUP    NAME                                                                                           STAT  CPU     MEM HOST                                                                  TIME

oneadmin@opennebular-1:/var/log/one$ onehost show 19
HOST 19 INFORMATION                                                             
ID                    : 19                  
NAME                  : s002-m5-40g-kvm38
CLUSTER               : onprem-cluster-laas-1a
STATE                 : MONITORED           
IM_MAD                : kvm                 
VM_MAD                : kvm                 
LAST MONITORING TIME  : 10/24 13:24:01      

HOST SHARES                                                                     
RUNNING VMS           : 0                   
MEMORY                                                                          
  TOTAL               : 503.5G              
  TOTAL +/- RESERVED  : 503.5G              
  USED (REAL)         : 3.3G                
  USED (ALLOCATED)    : 0K                  
CPU                                                                             
  TOTAL               : 9600                
  TOTAL +/- RESERVED  : 9600                
  USED (REAL)         : 0                   
  USED (ALLOCATED)    : 0                   

LOCAL SYSTEM DATASTORE #1 CAPACITY                                              
TOTAL:                : 97.9G               
USED:                 : 13.8G               
FREE:                 : 79.1G               

LOCAL SYSTEM DATASTORE #125 CAPACITY                                            
TOTAL:                : 97.9G               
USED:                 : 13.8G               
FREE:                 : 79.1G               

LOCAL SYSTEM DATASTORE #126 CAPACITY                                            
TOTAL:                : 97.9G               
USED:                 : 13.8G               
FREE:                 : 79.1G               

LOCAL SYSTEM DATASTORE #127 CAPACITY                                            
TOTAL:                : 97.9G               
USED:                 : 13.8G               
FREE:                 : 79.1G               

LOCAL SYSTEM DATASTORE #133 CAPACITY                                            
TOTAL:                : 97.9G               
USED:                 : 13.8G               
FREE:                 : 79.1G               

MONITORING INFORMATION                                                          
ARCH="x86_64"
CGROUPS_VERSION="2"
CONTEXT=[
  SSH_PUBLIC_KEY="ssh-rsa" ]
CPUSPEED="0"
HOSTNAME="s002-m5-40g-kvm38"
HYPERVISOR="kvm"
IM_MAD="kvm"
KVM_CPU_FEATURES="3dnowprefetch,abm,acpi,adx,aes,apic,arat,arch-capabilities,avx,avx2,avx512-bf16,avx512bw,avx512cd,avx512dq,avx512f,avx512vl,avx512vnni,bmi1,bmi2,clflush,clflushopt,clwb,cmov,cx16,cx8,dca,de,ds,ds_cpl,dtes64,erms,est,f16c,fma,fpu,fsgsbase,fxsr,hle,ht,ibrs-all,intel-pt,invpcid,invtsc,lahf_lm,lm,mca,mce,md-clear,mds-no,mmx,monitor,movbe,mpx,msr,mtrr,nx,pae,pat,pbe,pcid,pclmuldq,pdcm,pdpe1gb,pge,pku,pni,popcnt,pschange-mc-no,pse,pse36,rdctl-no,rdrand,rdseed,rdtscp,rtm,sep,skip-l1dfl-vmentry,smap,smep,smx,spec-ctrl,ss,ssbd,sse,sse2,sse4.1,sse4.2,ssse3,stibp,syscall,taa-no,tm,tm2,tsc,tsc-deadline,tsc_adjust,tsx-ctrl,vme,vmx,x2apic,xgetbv1,xsave,xsavec,xsaveopt,xsaves,xtpr"
KVM_CPU_MODEL="Cascadelake-Server-noTSX"
KVM_CPU_MODELS="486 pentium pentium2 pentium3 pentiumpro coreduo n270 core2duo qemu32 kvm32 cpu64-rhel5 cpu64-rhel6 kvm64 Conroe Penryn Nehalem Nehalem-IBRS Westmere Westmere-IBRS SandyBridge SandyBridge-IBRS IvyBridge IvyBridge-IBRS Haswell-noTSX Haswell-noTSX-IBRS Broadwell-noTSX Broadwell-noTSX-IBRS Skylake-Client-noTSX-IBRS Skylake-Server-noTSX-IBRS Cascadelake-Server-noTSX SapphireRapids SapphireRapids-noTSX Opteron_G1"
KVM_MACHINES="pc-i440fx-jammy ubuntu pc-i440fx-impish-hpb pc-q35-5.2 pc-i440fx-2.12 pc-i440fx-2.0 pc-i440fx-xenial pc-i440fx-6.2 pc pc-q35-4.2 pc-i440fx-2.5 pc-i440fx-4.2 pc-i440fx-focal pc-i440fx-hirsute pc-q35-xenial pc-i440fx-jammy-hpb pc-i440fx-5.2 pc-i440fx-1.5 pc-q35-2.7 pc-q35-eoan-hpb pc-i440fx-zesty pc-i440fx-disco-hpb pc-q35-groovy pc-i440fx-groovy pc-q35-artful pc-i440fx-2.2 pc-i440fx-trusty pc-i440fx-eoan-hpb pc-q35-focal-hpb pc-q35-jammy-maxcpus pc-q35-bionic-hpb pc-i440fx-artful pc-i440fx-2.7 pc-q35-6.1 pc-i440fx-jammy-maxcpus pc-i440fx-yakkety pc-q35-2.4 pc-q35-cosmic-hpb pc-q35-2.10 x-remote pc-i440fx-1.7 pc-q35-5.1 pc-q35-2.9 pc-i440fx-2.11 pc-i440fx-jammy-hpb-maxcpus pc-q35-3.1 pc-i440fx-6.1 pc-q35-4.1 pc-q35-jammy ubuntu-q35 pc-i440fx-2.4 pc-i440fx-4.1 pc-q35-eoan pc-q35-jammy-hpb pc-i440fx-5.1 pc-i440fx-2.9 pc-i440fx-bionic-hpb isapc pc-i440fx-1.4 pc-q35-cosmic pc-q35-2.6 pc-i440fx-3.1 pc-q35-bionic pc-q35-disco-hpb pc-i440fx-cosmic pc-q35-2.12 pc-i440fx-bionic pc-q35-groovy-hpb pc-q35-disco pc-i440fx-cosmic-hpb pc-i440fx-2.1 pc-i440fx-wily pc-q35-impish pc-q35-6.0 pc-i440fx-impish pc-i440fx-2.6 pc-q35-impish-hpb pc-q35-hirsute pc-q35-4.0.1 pc-q35-hirsute-hpb pc-i440fx-1.6 pc-q35-5.0 pc-q35-2.8 pc-i440fx-2.10 pc-q35-3.0 pc-i440fx-6.0 pc-q35-zesty pc-q35-4.0 pc-q35-focal microvm pc-i440fx-2.3 pc-q35-jammy-hpb-maxcpus pc-i440fx-focal-hpb pc-i440fx-disco pc-i440fx-4.0 pc-i440fx-groovy-hpb pc-i440fx-hirsute-hpb pc-i440fx-5.0 pc-i440fx-2.8 pc-q35-6.2 q35 pc-i440fx-eoan pc-q35-2.5 pc-i440fx-3.0 pc-q35-yakkety pc-q35-2.11"
MODELNAME="Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz"
PROVISION=[
  ANSIBLE_PLAYBOOK="onprem",
  COUNT="1",
  HOSTNAME="s002-m5-40g-kvm38",
  ID="20",
  INDEX="0",
  PROVIDER_NAME="onprem" ]
PROVISION_CONFIGURATION_BASE64="LS0tCm9uZXByb3Zpc2lvbl9oeXBlcnZpc29yOiBrdm0Kb3Blbm5lYnVsYV9zc2hfZGVwbG95X3ByaXZhdGVfa2V5OiB0cnVlCm9uZXByb3Zpc2lvbl9wcml2YXRlX3BoeWRldjogZW5wOTRzMGYxCg=="
PROVISION_CONNECTION=[
  PRIVATE_KEY="/var/lib/one/.ssh-oneprovision/id_rsa",
  PUBLIC_KEY="/var/lib/one/.ssh-oneprovision/id_rsa.pub",
  REMOTE_PORT="22",
  REMOTE_USER="root" ]
RESERVED_CPU=""
RESERVED_MEM=""
VERSION="6.10.0"
VM_MAD="kvm"

NUMA NODES

  ID CORES                                                                    USED FREE
   0 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --  0    48
   1 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --  0    48

NUMA MEMORY

 NODE_ID TOTAL    USED_REAL            USED_ALLOCATED       FREE    
       0 251.6G   4.2G                 0K                   247.4G
       1 251.9G   1.6G                 0K                   250.4G

NUMA HUGEPAGES

 NODE_ID SIZE     TOTAL    FREE     USED    
       0 2M       0        0        0
       0 1024M    0        0        0
       1 2M       0        0        0
       1 1024M    0        0        0

WILD VIRTUAL MACHINES

NAME                                                      IMPORT_ID  CPU     MEMORY

VIRTUAL MACHINES

  ID USER     GROUP    NAME                                                                                           STAT  CPU     MEM HOST                                                                  TIME
oneadmin@opennebular-1:/var/log/one$
4

Hello Ken,

This is the summary I’ve extracted from the logs:

Thu Oct 24 13:30:51 2024 [Z0][HOST][D]: Discovered Hosts (enabled):
ID          : 19  -> HOST_ID
CLUSTER_ID  : 117
PUBLIC      : 0
...
ID          : 20  -> HOST_ID
CLUSTER_ID  : 117
PUBLIC      : 0


HOST 20 INFORMATION
ID                    : 20
NAME                  : s002-m5-40g-kvm27
CLUSTER               : onprem-cluster-laas-1a
HYPERVISOR="kvm"
PIN_POLICY="NONE"
PUBLIC_CLOUD="NO"

HOST 19 INFORMATION
ID                    : 19
NAME                  : s002-m5-40g-kvm38
CLUSTER               : onprem-cluster-laas-1a
HYPERVISOR="kvm"

According to the requirements, host 19 should be discarded due to HOST_ID = 20
But indeed there is no excuse why host 20 is not selected.
Could you clarify that the host CLUSTER_ID = 117, by looking at the output of onehost show -x 20? I am assuming that onprem-cluster-laas-1a is cluster 117…

But in that case, I have no idea what sched requirements are not fillfuled for host 20 :confused: (HOST_ID - check, HYPERVISOR - check, not PIN_POLICY - check, not PUBLIC CLOUD - check, CLUSTER_ID - check…)

Best Regards,
Anton Todorov

5

Hi Anton,

Thanks once again for the prompt response. Yes host 20 is part off cluster 117.
Here is the xml.

<CLUSTER_ID>117</CLUSTER_ID>
onprem-cluster-laas-1a

Any suggestions what I can do to debug further? Any other service I should increase the log level to get info that’s being sent to the scheduler process? (or something like that.)

full output


oneadmin@opennebular-1:/var/log/one$ onehost show -x 20
<HOST>
  <ID>20</ID>
  <NAME>s002-m5-40g-kvm27</NAME>
  <STATE>2</STATE>
  <PREV_STATE>2</PREV_STATE>
  <IM_MAD><![CDATA[kvm]]></IM_MAD>
  <VM_MAD><![CDATA[kvm]]></VM_MAD>
  <CLUSTER_ID>117</CLUSTER_ID>
  <CLUSTER>onprem-cluster-laas-1a</CLUSTER>
  <HOST_SHARE>
    <MEM_USAGE>0</MEM_USAGE>
    <CPU_USAGE>0</CPU_USAGE>
    <TOTAL_MEM>527984044</TOTAL_MEM>
    <TOTAL_CPU>9600</TOTAL_CPU>
    <MAX_MEM>527984044</MAX_MEM>
    <MAX_CPU>9600</MAX_CPU>
    <RUNNING_VMS>0</RUNNING_VMS>
    <VMS_THREAD>1</VMS_THREAD>
    <DATASTORES>
      <DISK_USAGE><![CDATA[0]]></DISK_USAGE>
      <DS>
        <FREE_MB><![CDATA[82897]]></FREE_MB>
        <ID><![CDATA[133]]></ID>
        <TOTAL_MB><![CDATA[100221]]></TOTAL_MB>
        <USED_MB><![CDATA[12189]]></USED_MB>
      </DS>
      <FREE_DISK><![CDATA[82897]]></FREE_DISK>
      <MAX_DISK><![CDATA[100221]]></MAX_DISK>
      <USED_DISK><![CDATA[12189]]></USED_DISK>
    </DATASTORES>
    <PCI_DEVICES/>
    <NUMA_NODES>
      <NODE>
        <CORE>
          <CPUS><![CDATA[21:-1,69:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[27]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[11:-1,59:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[13]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[9:-1,57:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[11]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[20:-1,68:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[26]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[10:-1,58:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[12]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[0:-1,48:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[0]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[7:-1,55:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[9]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[18:-1,66:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[22]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[8:-1,56:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[10]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[5:-1,53:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[6]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[16:-1,64:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[20]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[6:-1,54:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[8]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[3:-1,51:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[3]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[14:-1,62:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[18]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[4:-1,52:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[5]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[22:-1,70:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[28]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[1:-1,49:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[1]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[12:-1,60:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[16]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[2:-1,50:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[2]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[19:-1,67:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[25]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[17:-1,65:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[21]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[15:-1,63:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[19]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[23:-1,71:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[29]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[13:-1,61:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[17]]></ID>
        </CORE>
        <HUGEPAGE>
          <PAGES><![CDATA[0]]></PAGES>
          <SIZE><![CDATA[2048]]></SIZE>
          <USAGE><![CDATA[0]]></USAGE>
        </HUGEPAGE>
        <HUGEPAGE>
          <PAGES><![CDATA[0]]></PAGES>
          <SIZE><![CDATA[1048576]]></SIZE>
          <USAGE><![CDATA[0]]></USAGE>
        </HUGEPAGE>
        <MEMORY>
          <DISTANCE><![CDATA[0 1]]></DISTANCE>
          <TOTAL><![CDATA[263809940]]></TOTAL>
          <USAGE><![CDATA[0]]></USAGE>
        </MEMORY>
        <NODE_ID><![CDATA[0]]></NODE_ID>
      </NODE>
      <NODE>
        <CORE>
          <CPUS><![CDATA[40:-1,88:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[20]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[30:-1,78:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[6]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[38:-1,86:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[18]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[28:-1,76:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[4]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[46:-1,94:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[28]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[36:-1,84:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[16]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[26:-1,74:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[2]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[44:-1,92:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[26]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[34:-1,82:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[12]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[24:-1,72:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[0]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[42:-1,90:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[24]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[32:-1,80:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[10]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[41:-1,89:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[21]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[31:-1,79:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[9]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[39:-1,87:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[19]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[29:-1,77:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[5]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[47:-1,95:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[29]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[37:-1,85:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[17]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[27:-1,75:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[3]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[45:-1,93:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[27]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[35:-1,83:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[13]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[25:-1,73:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[1]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[43:-1,91:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[25]]></ID>
        </CORE>
        <CORE>
          <CPUS><![CDATA[33:-1,81:-1]]></CPUS>
          <DEDICATED><![CDATA[NO]]></DEDICATED>
          <FREE><![CDATA[2]]></FREE>
          <ID><![CDATA[11]]></ID>
        </CORE>
        <HUGEPAGE>
          <PAGES><![CDATA[0]]></PAGES>
          <SIZE><![CDATA[2048]]></SIZE>
          <USAGE><![CDATA[0]]></USAGE>
        </HUGEPAGE>
        <HUGEPAGE>
          <PAGES><![CDATA[0]]></PAGES>
          <SIZE><![CDATA[1048576]]></SIZE>
          <USAGE><![CDATA[0]]></USAGE>
        </HUGEPAGE>
        <MEMORY>
          <DISTANCE><![CDATA[1 0]]></DISTANCE>
          <TOTAL><![CDATA[264174104]]></TOTAL>
          <USAGE><![CDATA[0]]></USAGE>
        </MEMORY>
        <NODE_ID><![CDATA[1]]></NODE_ID>
      </NODE>
    </NUMA_NODES>
  </HOST_SHARE>
  <VMS/>
  <TEMPLATE>
    <ARCH><![CDATA[x86_64]]></ARCH>
    <CGROUPS_VERSION><![CDATA[2]]></CGROUPS_VERSION>
    <CONTEXT>
      <SSH_PUBLIC_KEY><![CDATA[ssh-rsa A]]></SSH_PUBLIC_KEY>
    </CONTEXT>
    <CPUSPEED><![CDATA[0]]></CPUSPEED>
    <HOSTNAME><![CDATA[s002-m5-40g-kvm27]]></HOSTNAME>
    <HYPERVISOR><![CDATA[kvm]]></HYPERVISOR>
    <IM_MAD><![CDATA[kvm]]></IM_MAD>
    <KVM_CPU_FEATURES><![CDATA[3dnowprefetch,abm,acpi,adx,aes,apic,arat,arch-capabilities,avx,avx2,avx512-bf16,avx512bw,avx512cd,avx512dq,avx512f,avx512vl,avx512vnni,bmi1,bmi2,clflush,clflushopt,clwb,cmov,cx16,cx8,dca,de,ds,ds_cpl,dtes64,erms,est,f16c,fma,fpu,fsgsbase,fxsr,hle,ht,ibrs-all,intel-pt,invpcid,invtsc,lahf_lm,lm,mca,mce,md-clear,mds-no,mmx,monitor,movbe,mpx,msr,mtrr,nx,pae,pat,pbe,pcid,pclmuldq,pdcm,pdpe1gb,pge,pku,pni,popcnt,pschange-mc-no,pse,pse36,rdctl-no,rdrand,rdseed,rdtscp,rtm,sep,skip-l1dfl-vmentry,smap,smep,smx,spec-ctrl,ss,ssbd,sse,sse2,sse4.1,sse4.2,ssse3,stibp,syscall,taa-no,tm,tm2,tsc,tsc-deadline,tsc_adjust,tsx-ctrl,vme,vmx,x2apic,xgetbv1,xsave,xsavec,xsaveopt,xsaves,xtpr]]></KVM_CPU_FEATURES>
    <KVM_CPU_MODEL><![CDATA[Cascadelake-Server-noTSX]]></KVM_CPU_MODEL>
    <KVM_CPU_MODELS><![CDATA[486 pentium pentium2 pentium3 pentiumpro coreduo n270 core2duo qemu32 kvm32 cpu64-rhel5 cpu64-rhel6 kvm64 Conroe Penryn Nehalem Nehalem-IBRS Westmere Westmere-IBRS SandyBridge SandyBridge-IBRS IvyBridge IvyBridge-IBRS Haswell-noTSX Haswell-noTSX-IBRS Broadwell-noTSX Broadwell-noTSX-IBRS Skylake-Client-noTSX-IBRS Skylake-Server-noTSX-IBRS Cascadelake-Server-noTSX SapphireRapids SapphireRapids-noTSX Opteron_G1]]></KVM_CPU_MODELS>
    <KVM_MACHINES><![CDATA[pc-i440fx-jammy ubuntu pc-i440fx-impish-hpb pc-q35-5.2 pc-i440fx-2.12 pc-i440fx-2.0 pc-i440fx-xenial pc-i440fx-6.2 pc pc-q35-4.2 pc-i440fx-2.5 pc-i440fx-4.2 pc-i440fx-focal pc-i440fx-hirsute pc-q35-xenial pc-i440fx-jammy-hpb pc-i440fx-5.2 pc-i440fx-1.5 pc-q35-2.7 pc-q35-eoan-hpb pc-i440fx-zesty pc-i440fx-disco-hpb pc-q35-groovy pc-i440fx-groovy pc-q35-artful pc-i440fx-2.2 pc-i440fx-trusty pc-i440fx-eoan-hpb pc-q35-focal-hpb pc-q35-jammy-maxcpus pc-q35-bionic-hpb pc-i440fx-artful pc-i440fx-2.7 pc-q35-6.1 pc-i440fx-jammy-maxcpus pc-i440fx-yakkety pc-q35-2.4 pc-q35-cosmic-hpb pc-q35-2.10 x-remote pc-i440fx-1.7 pc-q35-5.1 pc-q35-2.9 pc-i440fx-2.11 pc-i440fx-jammy-hpb-maxcpus pc-q35-3.1 pc-i440fx-6.1 pc-q35-4.1 pc-q35-jammy ubuntu-q35 pc-i440fx-2.4 pc-i440fx-4.1 pc-q35-eoan pc-q35-jammy-hpb pc-i440fx-5.1 pc-i440fx-2.9 pc-i440fx-bionic-hpb isapc pc-i440fx-1.4 pc-q35-cosmic pc-q35-2.6 pc-i440fx-3.1 pc-q35-bionic pc-q35-disco-hpb pc-i440fx-cosmic pc-q35-2.12 pc-i440fx-bionic pc-q35-groovy-hpb pc-q35-disco pc-i440fx-cosmic-hpb pc-i440fx-2.1 pc-i440fx-wily pc-q35-impish pc-q35-6.0 pc-i440fx-impish pc-i440fx-2.6 pc-q35-impish-hpb pc-q35-hirsute pc-q35-4.0.1 pc-q35-hirsute-hpb pc-i440fx-1.6 pc-q35-5.0 pc-q35-2.8 pc-i440fx-2.10 pc-q35-3.0 pc-i440fx-6.0 pc-q35-zesty pc-q35-4.0 pc-q35-focal microvm pc-i440fx-2.3 pc-q35-jammy-hpb-maxcpus pc-i440fx-focal-hpb pc-i440fx-disco pc-i440fx-4.0 pc-i440fx-groovy-hpb pc-i440fx-hirsute-hpb pc-i440fx-5.0 pc-i440fx-2.8 pc-q35-6.2 q35 pc-i440fx-eoan pc-q35-2.5 pc-i440fx-3.0 pc-q35-yakkety pc-q35-2.11]]></KVM_MACHINES>
    <MODELNAME><![CDATA[Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz]]></MODELNAME>
    <PIN_POLICY><![CDATA[NONE]]></PIN_POLICY>
    <PROVISION>
      <ANSIBLE_PLAYBOOK><![CDATA[onprem]]></ANSIBLE_PLAYBOOK>
      <COUNT><![CDATA[1]]></COUNT>
      <HOSTNAME><![CDATA[s002-m5-40g-kvm27.mitg-bxb300.cisco.com]]></HOSTNAME>
      <ID><![CDATA[20]]></ID>
      <INDEX><![CDATA[1]]></INDEX>
      <PROVIDER_NAME><![CDATA[onprem]]></PROVIDER_NAME>
    </PROVISION>
    <PROVISION_CONFIGURATION_BASE64><![CDATA[LS0tCm9uZXByb3Zpc2lvbl9oeXBlcnZpc29yOiBrdm0Kb3Blbm5lYnVsYV9zc2hfZGVwbG95X3ByaXZhdGVfa2V5OiB0cnVlCm9uZXByb3Zpc2lvbl9wcml2YXRlX3BoeWRldjogZW5wOTRzMGYxCg==]]></PROVISION_CONFIGURATION_BASE64>
    <PROVISION_CONNECTION>
      <PRIVATE_KEY><![CDATA[/var/lib/one/.ssh-oneprovision/id_rsa]]></PRIVATE_KEY>
      <PUBLIC_KEY><![CDATA[/var/lib/one/.ssh-oneprovision/id_rsa.pub]]></PUBLIC_KEY>
      <REMOTE_PORT><![CDATA[22]]></REMOTE_PORT>
      <REMOTE_USER><![CDATA[root]]></REMOTE_USER>
    </PROVISION_CONNECTION>
    <PUBLIC_CLOUD><![CDATA[NO]]></PUBLIC_CLOUD>
    <VERSION><![CDATA[6.10.0]]></VERSION>
    <VM_MAD><![CDATA[kvm]]></VM_MAD>
  </TEMPLATE>
  <MONITORING>
    <TIMESTAMP>1729852552</TIMESTAMP>
    <ID>20</ID>
    <CAPACITY>
      <FREE_CPU><![CDATA[9600]]></FREE_CPU>
      <FREE_MEMORY><![CDATA[524513872]]></FREE_MEMORY>
      <USED_CPU><![CDATA[0]]></USED_CPU>
      <USED_MEMORY><![CDATA[3470172]]></USED_MEMORY>
    </CAPACITY>
    <SYSTEM>
      <NETRX><![CDATA[6635898670]]></NETRX>
      <NETTX><![CDATA[9117652932]]></NETTX>
    </SYSTEM>
    <NUMA_NODE>
      <HUGEPAGE>
        <FREE><![CDATA[0]]></FREE>
        <SIZE><![CDATA[2048]]></SIZE>
      </HUGEPAGE>
      <HUGEPAGE>
        <FREE><![CDATA[0]]></FREE>
        <SIZE><![CDATA[1048576]]></SIZE>
      </HUGEPAGE>
      <MEMORY>
        <FREE><![CDATA[262023888]]></FREE>
        <USED><![CDATA[1786052]]></USED>
      </MEMORY>
      <NODE_ID><![CDATA[0]]></NODE_ID>
    </NUMA_NODE>
    <NUMA_NODE>
      <HUGEPAGE>
        <FREE><![CDATA[0]]></FREE>
        <SIZE><![CDATA[2048]]></SIZE>
      </HUGEPAGE>
      <HUGEPAGE>
        <FREE><![CDATA[0]]></FREE>
        <SIZE><![CDATA[1048576]]></SIZE>
      </HUGEPAGE>
      <MEMORY>
        <FREE><![CDATA[260324000]]></FREE>
        <USED><![CDATA[3850104]]></USED>
      </MEMORY>
      <NODE_ID><![CDATA[1]]></NODE_ID>
    </NUMA_NODE>
  </MONITORING>
</HOST>
oneadmin@opennebular-1:/var/log/one$
6

After some significant debugging and reviewing the Scheduler.cc code I found the issue/WebUI bug.

This works: (HYPERVISOR=kvm) & (ID = 20)
WebUI give this by default: (HYPERVISOR=undefined) & (HOST_ID = 20)
It was somewhat obvious to change “undefined” to “kvm” but the HOST ID… well that took a few days to figure out.

=== Debugging Details

Workaround # was to put HOST_ID as an attribute on each host, but that wasn’t the best fix.

The HOST have an “ID” value, but needed to have “HOST_ID” value (I thought). I had to add the attribute to each host manually. I assumed was a bug? Maybe previous release had HOST_ID in the XML where it is only ID now?

Reviewing the scheduler code I was failing at this point:

logs:

Wed Oct 30 18:51:47 2024 [Z0][SCHED][DD]: Setting VM groups placement constraints. Total time: 0.00s
Wed Oct 30 18:51:47 2024 [Z0][SCHED][DD]: Host 19 discarded for VM 109. It does not fulfill SCHED_REQUIREMENTS: (CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( (HYPERVISOR=kvm) & (HOST_ID = 20) )
Wed Oct 30 18:51:47 2024 [Z0][SCHED][DD]: Host 20 discarded for VM 109. It does not fulfill SCHED_REQUIREMENTS: (CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( (HYPERVISOR=kvm) & (HOST_ID = 20) )
Wed Oct 30 18:51:47 2024 [Z0][SCHED][D]: Match-making results for VM 109:
        Cannot schedule VM, there is no suitable host.

Tracing the code I’m at this point and failing the simple bool statement:

static bool match_host

        if ( host->eval_bool(vm->get_requirements(), matched, &estr) != 0 )
        {
            ostringstream oss;

            n_error++;

            oss << "Error in SCHED_REQUIREMENTS: '" << vm->get_requirements()
                << "', error: " << estr;

            vm->log(oss.str());

            error = oss.str();

            free(estr);

            return false;
        }

        if (matched == false)
        {
            error = "It does not fulfill SCHED_REQUIREMENTS: " +
                    vm->get_requirements();
            return false;
        }
    }

get_requirements() is this: (CLUSTER_ID = 117) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( (HYPERVISOR=kvm) & (HOST_ID = 20) )

I then started manually adding attributes to each host until finding out that the one that mattered was HOST_ID. PUBLIC_CLOUD and PIN_POLICY there or not did not matter.

CLUSTER_ID is in the host xml already, but HOST_ID is NOT. Its there as “ID”.
This explains why all other use-case work as they trigger off the Cluster ID or Default cluster. This one use-case needs the HOST_ID to match.

oneadmin@opennebular-1:/var/log/one$ onehost show -x s002-m5-40g-kvm27.mitg-bxb300.cisco.com | grep HOST_ID
oneadmin@opennebular-1:/var/log/one$ onehost show -x s002-m5-40g-kvm27.mitg-bxb300.cisco.com | grep ID
  <ID>20</ID>
  <CLUSTER_ID>117</CLUSTER_ID>

Then as writing this realized… what If I just change the sandstone auto-filled in value to ID instead of HOST_ID… That worked!!! So I guess the bug is the WebUI is giving a bad example?

This works: (HYPERVISOR=kvm) & (ID = 20)
WebUI give this by default: (HYPERVISOR=undefined) & (HOST_ID = 20)
It was somewhat obvious to change “undefined” to “kvm” but the HOST ID… well that took a fee days to figure out.

1 Like