Cannot dispatch VM to any Host. Possible reasons: Not enough capacity in Host or System DS, dispatch limit reached, or limit of free leases reached.

Describe:
Cannot dispatch VM to any Host. Possible reasons: Not enough capacity in Host or System DS, dispatch limit reached, or limit of free leases reached.

Front-end can ssh to kvm-node and itself without encryption.

Versions of the related components and OS:
OS:Ubuntu 20.04.6 LTS
Front-end 6.6
KVM Node: libvirtd (libvirt) 6.0.0

VMs_template:

CPU = "4"
DISK = [
  IMAGE = "Windows2016-CDROM",
  IMAGE_UNAME = "oneadmin" ]
DISK = [
  DEV_PREFIX = "vd",
  IMAGE = "windows2016-OS",
  IMAGE_UNAME = "oneadmin" ]
GRAPHICS = [
  LISTEN = "0.0.0.0",
  TYPE = "VNC" ]
INPUT = [
  BUS = "usb",
  TYPE = "tablet" ]
MEMORY = "8192"
MEMORY_UNIT_COST = "MB"
NIC = [
  NETWORK = "net-bjxd",
  NETWORK_UNAME = "oneadmin",
  SECURITY_GROUPS = "0" ]
NIC_DEFAULT = [
  MODEL = "virtio" ]
OS = [
  BOOT = "disk0" ]
VCPU = "4"

log level
/etc/one/sched.conf
LOG = [
system = “file”,
debug_level = 3
]

sched.log

Wed Oct 18 19:08:33 2023 [Z0][VM][D]: Found 1 pending/rescheduling VMs.
Wed Oct 18 19:08:33 2023 [Z0][HOST][D]: Discovered 1 enabled hosts.
Wed Oct 18 19:08:33 2023 [Z0][VM][D]: VMs in VMGroups:

Wed Oct 18 19:08:33 2023 [Z0][VNET][D]: Discovered 1 vnets.
Wed Oct 18 19:08:33 2023 [Z0][SCHED][D]: Dispatching VMs to hosts:
VMID Priority Host System DS
--------------------------------------------------------------

When the VM is instantiated, it’s instance template will get several attributes defining the placement policy that the scheduler will have to process in order to select an adequate host and system datastore for the VM.

These attributes are expressions that get evaluated and the message

Cannot dispatch VM to any Host. Possible reasons: Not enough capacity in Host or System DS, dispatch limit reached, or limit of free leases reached.

means that no resource was compatible with the whole expression. In order to figure out why

  • increase the debugging level to 5 in sched.conf
  • restart the scheduler
  • try to reproduce the problem again

You should know get more informative logs about the problem

The debugging level of sched.conf is changed to 5, and the opennebula scheduler service is restarted

Instantiate the template, error message:

Scheduler Error log:

Wed Oct 25 11:38:58 2023 [Z0][SCHED][I]: Init Scheduler Log system
Wed Oct 25 11:38:58 2023 [Z0][SCHED][I]: Starting Scheduler Daemon
----------------------------------------
     Scheduler Configuration File
----------------------------------------
COLD_MIGRATE_MODE=0
DEFAULT_DS_SCHED=POLICY=1
DEFAULT_NIC_SCHED=POLICY=1
DEFAULT_SCHED=POLICY=1
DIFFERENT_VNETS=YES
LIVE_RESCHEDS=0
LOG=DEBUG_LEVEL=5,SYSTEM=file
MAX_BACKUP=5
MAX_BACKUPS=5
MAX_BACKUPS_HOST=2
MAX_BACKUP_HOST=2
MAX_DISPATCH=30
MAX_HOST=1
MAX_VM=5000
MEMORY_SYSTEM_DS_SCALE=0
MESSAGE_SIZE=1073741824
ONE_XMLRPC=http://localhost:2633/RPC2
SCHED_INTERVAL=15
TIMEOUT=60
----------------------------------------
Wed Oct 25 11:38:58 2023 [Z0][SCHED][I]: XML-RPC client using 1073741824 bytes for response buffer.

Wed Oct 25 11:38:58 2023 [Z0][SCHED][I]: oned successfully contacted.
Wed Oct 25 11:38:58 2023 [Z0][SCHED][I]: Configuring scheduler for Zone ID: 0
Wed Oct 25 11:38:58 2023 [Z0][SCHED][I]: Starting scheduler loop...
Wed Oct 25 11:39:13 2023 [Z0][VM][D]: Pending/rescheduling VM and capacity requirements:
  ACTION       VM  CPU      Memory PCI   System DS  Image DS
------------------------------------------------------------
  DEPLOY        4  400     8388608   0      108291  DS 1: 0

Wed Oct 25 11:39:13 2023 [Z0][HOST][D]: Discovered Hosts (enabled):
ID          : 0
CLUSTER_ID  : 0
PUBLIC      : 0
MEM_USAGE   : 0
CPU_USAGE   : 0
MAX_MEM     : 65569300
MAX_CPU     : 6400
FREE_DISK   : 83423
RUNNING_VMS : 0

 DSID         FREE_MB
------------------------------

    PCI ADDRESS    CLASS   VENDOR   DEVICE     VMID
-------------------------------------------------------

---------------------------------------------------------------------------
Node: 0	Memory: 0/31G
---------------------------------------------------------------------------
( 0 32)( 4 36)( 8 40)(12 44)(14 46)(10 42)( 6 38)( 2 34)(16 48)(20 52)(24 56)(28 60)(30 62)(26 58)(22 54)(18 50)
( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)
---------------------------------------------------------------------------
Node: 1	Memory: 0/31G
---------------------------------------------------------------------------
( 1 33)( 5 37)( 9 41)(13 45)(15 47)(11 43)( 7 39)( 3 35)(17 49)(21 53)(25 57)(29 61)(31 63)(27 59)(23 55)(19 51)
( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)
---------------------------------------------------------------------------


Wed Oct 25 11:39:13 2023 [Z0][VM][D]: VMs in VMGroups:

Wed Oct 25 11:39:13 2023 [Z0][VNET][D]: Discovered VNETS:
      ID   Leases
--------------------
       0       19

You should get more information about the decision process further on the logfile. When a VM is instantiated, there will be a bunch of automatic scheduling requirements generated for such VM.

That expression is then evaluated according to the HOST attributes.

Wed Oct 25 11:39:13 2023 [Z0][HOST][D]: Discovered Hosts (enabled):
ID          : 0
CLUSTER_ID  : 0
PUBLIC      : 0
MEM_USAGE   : 0
CPU_USAGE   : 0
MAX_MEM     : 65569300
MAX_CPU     : 6400
FREE_DISK   : 83423
RUNNING_VMS : 0

 DSID         FREE_MB
------------------------------

    PCI ADDRESS    CLASS   VENDOR   DEVICE     VMID
-------------------------------------------------------

---------------------------------------------------------------------------
Node: 0	Memory: 0/31G
---------------------------------------------------------------------------
( 0 32)( 4 36)( 8 40)(12 44)(14 46)(10 42)( 6 38)( 2 34)(16 48)(20 52)(24 56)(28 60)(30 62)(26 58)(22 54)(18 50)
( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)
---------------------------------------------------------------------------
Node: 1	Memory: 0/31G
---------------------------------------------------------------------------
( 1 33)( 5 37)( 9 41)(13 45)(15 47)(11 43)( 7 39)( 3 35)(17 49)(21 53)(25 57)(29 61)(31 63)(27 59)(23 55)(19 51)
( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)( -  -)
---------------------------------------------------------------------------


With level 5 you’ll get the result of the expression evaluation for each host in the host pool until a suitable host is found. Something like this, for example

Mon Jul 26 14:13:25 2021 [Z0][SCHED][D]: Setting VM groups placement constraints. Total time: 0.00s
Mon Jul 26 14:13:25 2021 [Z0][SCHED][D]: Host 30 discarded for VM 224. Not enough memory: 9345024/8149924
Mon Jul 26 14:13:25 2021 [Z0][SCHED][D]: Host 31 discarded for VM 224. Not enough memory: 9345024/8149900
Mon Jul 26 14:13:26 2021 [Z0][SCHED][D]: Match-making results for VM 224:
Cannot schedule VM, there is no suitable host.

or

Mon Jul 26 14:16:42 2021 [Z0][SCHED][D]: Host 30 discarded for VM 225. It does not fulfill SCHED_REQUIREMENTS: (CLUSTER_ID = 0 | CLUSTER_ID = 122) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( HYPERVISOR!="vcenter" & QOS = GOLD )
Mon Jul 26 14:16:42 2021 [Z0][SCHED][D]: Host 31 discarded for VM 225. It does not fulfill SCHED_REQUIREMENTS: (CLUSTER_ID = 0 | CLUSTER_ID = 122) & !(PUBLIC_CLOUD = YES) & !(PIN_POLICY = PINNED) & ( HYPERVISOR!="vcenter" & QOS = GOLD )
Mon Jul 26 14:16:42 2021 [Z0][SCHED][D]: Match-making results for VM 225:
Cannot schedule VM, there is no suitable host.

sched.conf

MESSAGE_SIZE = 1073741824
TIMEOUT      = 60
ONE_XMLRPC = "http://localhost:2633/RPC2"
SCHED_INTERVAL = 15
MAX_VM       = 5000
MAX_DISPATCH = 30
MAX_HOST     = 1
MAX_BACKUP      = 5
MAX_BACKUP_HOST = 2
LIVE_RESCHEDS  = 0
COLD_MIGRATE_MODE = 0
MEMORY_SYSTEM_DS_SCALE = 0
DIFFERENT_VNETS = YES
DEFAULT_SCHED = [
   POLICY = 4
]
DEFAULT_DS_SCHED = [
   POLICY = 3
]
DEFAULT_NIC_SCHED = [
   POLICY = 3
]
LOG = [
  SYSTEM      = "file",
  DEBUG_LEVEL = 5
]
onehost list
  ID NAME                                                                                                 CLUSTER    TVM      ALLOCATED_CPU      ALLOCATED_MEM STAT
   0 bjxd-ap-10-16-16-12                                                                                  default      0      0 / 6400 (0%)    0K / 62.5G (0%) on
root@bjxd-ap-10-16-16-11:/etc/one#   ID NAME                                                                                                 CLUSTER    TVM      AL

I don’t know how to troubleshoot this problem, provided the configuration file for sched.conf and the information for the onehost list.

Is it possible to provide a sched.conf and template that can be instantiated by normal templates for debugging? Thank you very much.

As mentioned before, your sched.log should contain more information about the decision making process of the scheduler based on the VMs on PENDING state + their requirements vs the infrastructure available. Your log contained the initial information (detected VMs, detected hosts and datastores) it should have been populated with more information like the example shown.

Are you not getting any more logging information ?

In any case you can also manually figure out the problems by checking out the VM instance SCHED_REQUIREMENTS, SCHED_DS_REQUIREMENTS and AUTOMATIC_REQUIREMENTS attributes on the VM template. Then match those vs what the host attributes have. The logging approach mentioned before helps with showcasing this expression evaluation process from the scheduler perspective.

The virtual machine has been successfully instantiated, the use of the new version and the old version of the function has changed greatly, did not carefully look for, thank you very much for your help, best wishes.