# OneD not responding after failed VM Creation attempt

**URL:** <https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960>\
**Category:** Bug Reporting\
**Created:** [July 7, 2025, 11:15am UTC](https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960 "2025-07-07T11:15:20Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![e-discoveryinfra-tid](https://avatars.discourse-cdn.com/v4/letter/e/48db29/32.png) [@e-discoveryinfra-tid](https://forum.opennebula.io/u/e-discoveryinfra-tid)\
**Post date:** [July 7, 2025, 11:15am UTC](https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960/1 "2025-07-07T11:15:20Z")

</div>

**Version:** OpenNebula 6.10.3 (64156dd6) Enterprise Edition  
**Setup:** 1 Zone with 3 Frontends

**Problem description:**  
After a unsuccessful VM creation attempt (via Sunstone GUI), all the services and the requests to OneD (via GUI or via CLI with commands like “onevm”) aren’t being processed by the oned daemon.

The oned service log (_/var/log/one/oned.log_) shows every second the following error:

```auto
Mon Jul 7 12:29:08 2025 [Z0][ONE][E]: SQL command was: INSERT INTO vm_pool (
  oid, name, body, uid, gid, state, lcm_state, owner_u, group_u, other_u, short_body, body_json
) VALUES (
  1339,
  'vm-machine-name',
  '<VM>...</VM>', -- (very long XML, see below)
  5, 0, 1, 0, 1, 0, 0,
  '<VM>...</VM>', -- (shorter XML, see below)
  '{...}' -- (JSON, see below)
)

```

The JSON body contains the template used to create the VM with the specific values (hidden the private data):

```json
{
  "VM": {
    "ID": "1339",
    "UID": "...",
    "GID": "...",
    "UNAME": "...",
    "GNAME": "...",
    "NAME": "...",
    "LAST_POLL": "...",
    "STATE": "...",
    "LCM_STATE": "...",
    "PREV_STATE": "...",
    "PREV_LCM_STATE": "...",
    "RESCHED": "...",
    "STIME": "...",
    "ETIME": "...",
    "DEPLOY_ID": "...",
    "TEMPLATE": {
      "AUTOMATIC_DS_REQUIREMENTS": "...",
      "AUTOMATIC_NIC_REQUIREMENTS": "...",
      "AUTOMATIC_REQUIREMENTS": "...",
      "CONTEXT": {
        "DISK_ID": "...",
        "ETH0_DNS": "...",
        "ETH0_EXTERNAL": "...",
        "ETH0_GATEWAY": "...",
        "ETH0_IP": "...",
        "ETH0_IP6": "...",
        "ETH0_IP6_GATEWAY": "...",
        "ETH0_IP6_METHOD": "...",
        "ETH0_IP6_METRIC": "...",
        "ETH0_IP6_PREFIX_LENGTH": "...",
        "ETH0_IP6_ULA": "...",
        "ETH0_MAC": "...",
        "ETH0_MASK": "...",
        "ETH0_METHOD": "...",
        "ETH0_METRIC": "...",
        "ETH0_MTU": "...",
        "ETH0_NETWORK": "...",
        "ETH0_SEARCH_DOMAIN": "...",
        "ETH0_VLAN_ID": "...",
        "ETH0_VROUTER_IP": "...",
        "ETH0_VROUTER_IP6": "...",
        "ETH0_VROUTER_MANAGEMENT": "...",
        "NETWORK": "...",
        "PASSWORD": "...",
        "PCI0_ADDRESS": "...",
        "PCI0_IP": "...",
        "PCI0_MAC": "...",
        "SET_HOSTNAME": "...",
        "SSH_PUBLIC_KEY": "...",
        "START_SCRIPT_BASE64": "...",
        "TARGET": "..."
      },
      "CPU": "...",
      "CPU_MODEL": {
        "MODEL": "..."
      },
      "DISK": [
        {
          "ALLOW_ORPHANS": "...",
          "CEPH_HOST": "...",
          "CEPH_SECRET": "...",
          "CEPH_USER": "...",
          "CLONE": "...",
          "CLONE_TARGET": "...",
          "CLUSTER_ID": "...",
          "DATASTORE": "...",
          "DATASTORE_ID": "...",
          "DEV_PREFIX": "...",
          "DISK_ID": "...",
          "DISK_SNAPSHOT_TOTAL_SIZE": "...",
          "DISK_TYPE": "...",
          "DRIVER": "...",
          "FORMAT": "...",
          "IMAGE": "...",
          "IMAGE_ID": "...",
          "IMAGE_STATE": "...",
          "IMAGE_UNAME": "...",
          "LN_TARGET": "...",
          "ORIGINAL_SIZE": "...",
          "POOL_NAME": "...",
          "READONLY": "...",
          "SAVE": "...",
          "SIZE": "...",
          "SOURCE": "...",
          "TARGET": "...",
          "TM_MAD": "...",
          "TYPE": "..."
        }
      ],
      "GRAPHICS": {
        "LISTEN": "...",
        "TYPE": "..."
      },
      "MEMORY": "...",
      "MEMORY_MAX": "...",
      "MEMORY_RESIZE_MODE": "...",
      "NIC": [
        {
          "AR_ID": "...",
          "BRIDGE": "...",
          "BRIDGE_TYPE": "...",
          "CLUSTER_ID": "...",
          "DNS": "...",
          "GATEWAY": "...",
          "IP": "...",
          "MAC": "...",
          "NAME": "...",
          "NETWORK": "...",
          "NETWORK_ID": "...",
          "NETWORK_UNAME": "...",
          "NIC_ID": "...",
          "SECURITY_GROUPS": "...",
          "TARGET": "...",
          "VN_MAD": "..."
        }
      ],
      "OS": {
        "UUID": "..."
      },
      "PCI": {
        "AR_ID": "...",
        "BRIDGE": "...",
        "BRIDGE_TYPE": "...",
        "CLASS": "...",
        "CLUSTER_ID": "...",
        "DEVICE": "...",
        "IP": "...",
        "MAC": "...",
        "NAME": "...",
        "NETWORK": "...",
        "NETWORK_ID": "...",
        "NIC_ID": "...",
        "PCI_ID": "...",
        "SECURITY_GROUPS": "...",
        "TARGET": "...",
        "TYPE": "...",
        "VENDOR": "...",
        "VM_ADDRESS": "...",
        "VM_BUS": "...",
        "VM_DOMAIN": "...",
        "VM_FUNCTION": "...",
        "VM_SLOT": "...",
        "VN_MAD": "..."
      },
      "SECURITY_GROUP_RULE": [
        {
          "PROTOCOL": "...",
          "RULE_TYPE": "...",
          "SECURITY_GROUP_ID": "...",
          "SECURITY_GROUP_NAME": "..."
        }
        // ... (possibly more rules)
      ],
      "TEMPLATE_ID": "...",
      "VCPU": "...",
      "VCPU_MAX": "...",
      "VMID": "..."
    },
    "USER_TEMPLATE": {
      "HOT_RESIZE": {
        "CPU_HOT_ADD_ENABLED": "...",
        "MEMORY_HOT_ADD_ENABLED": "..."
      },
      "HYPERVISOR": "...",
      "LOGO": "...",
      "MEMORY_UNIT_COST": "...",
      "SCHED_REQUIREMENTS": "..."
    }
  }
}

```

Steps tried to solve:

- Restart the Opennebula services on the 3 zone hosts (`systemctl restart opennebula`)
- Stop services and repair consistency on the DB (`onedb fsck`), getting the following output:

```auto
Removing possibly corrupted records from VM showback please run 'oneshowback calculate` to recalculate the showback
VM 1339 is in Image XXX VM id list, but it should not
VNet XX AR 0 has leased ... to VM 1339, but it is actually free
VNet XX has 20 used leases, but it is actually 19
VNet XXX AR 0 has leased ... to VM 1339, but it is actually free
....

```

so multiple actions were performed to cleanup remaining configurations of the failed VM, but the action is constantly hitting the service, avoiding perform any action (stop a VM, create a VNet…) making the service unusable.

- Additionally, the schedule logs (_/var/log/one/sched.log_) from the different hosts only show the following entry:

```auto
Mon Jul 7 11:37:50 2025 [Z0][SCHED][E]: oned is not leader

```

We haven’t found any workaround to cleanup this action to unlock the requests to the daemon.

---

<div class="post-metadata">

**Author:** ![FrancJP](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/francjp/32/7690_2.png) [@FrancJP](https://forum.opennebula.io/u/FrancJP)\
**Post date:** [July 7, 2025, 3:21pm UTC](https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960/2 "2025-07-07T15:21:21Z")

</div>

Hello @e-discoveryinfra-tid,

If you are using Enterprise Edition, you have a support specific for you, please check your portal for that: [https://support.opennebula.pro/hc/en-us](https://support.opennebula.pro/hc/en-us)

If you are using an Elemental License, please let us know.

Regards,

---

<div class="post-metadata">

**Author:** ![brunorro](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/brunorro/32/7590_2.png) [@brunorro](https://forum.opennebula.io/u/brunorro)\
**Post date:** [July 8, 2025, 11:43am UTC](https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960/3 "2025-07-08T11:43:30Z")

</div>

Hello,

There may be some illegal chars in your database. Can you please check if you for the error `4025` in OpenNebula logs?

A `grep 4025 /var/log/one/oned.log` may have to do with this problem. In that case, you will have the SQL command that triggered this problem.

---

<div class="post-metadata">

**Author:** ![e-discoveryinfra-tid](https://avatars.discourse-cdn.com/v4/letter/e/48db29/32.png) [@e-discoveryinfra-tid](https://forum.opennebula.io/u/e-discoveryinfra-tid)\
**Post date:** [July 8, 2025, 3:50pm UTC](https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960/4 "2025-07-08T15:50:17Z")

</div>

Hi Bruno! Thanks you for answering,  
You’re right, the oned.log show the command that has failed (a VM creation from template with an invalid name);

`[Z0][ONE][E]: SQL command was: INSERT INTO vm_pool (oid, name, body, uid, gid, state, lcm_state, owner_u, group_u, other_u, short_body, body_json) VALUES (1339,'	pqcpoc-locust-backend'... .... error 4025 : CONSTRAINT 'vm_pool.body_json' failed for 'opennebula'.'vm_pool'`

The problem, in this case, is that after having stopped/restarted the oned service and cleaned up on the DB, the error seems to be stuck on the Control Plane and it disallows us to perform any action on Opennebula.

Thanks!

---

<div class="post-metadata">

**Author:** ![brunorro](https://yyz1.discourse-cdn.com/flex031/user_avatar/forum.opennebula.io/brunorro/32/7590_2.png) [@brunorro](https://forum.opennebula.io/u/brunorro)\
**Post date:** [July 9, 2025, 9:22am UTC](https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960/5 "2025-07-09T09:22:28Z")

</div>

This looks related to a bug fixed in 6.10.4 issued when an entity (i.e., a VM) is created from the API forcing its name with special chars (`\t` or similar)

In this case, the best way to proceed is backup the database and fix it before restarting OpenNebula, and after that upgrade to 6.10.4

---

<div class="post-metadata">

**Author:** ![e-discoveryinfra-tid](https://avatars.discourse-cdn.com/v4/letter/e/48db29/32.png) [@e-discoveryinfra-tid](https://forum.opennebula.io/u/e-discoveryinfra-tid)\
**Post date:** [July 10, 2025, 9:53am UTC](https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960/6 "2025-07-10T09:53:50Z")

</div>

Hi again, Bruno,  
we’ve tried te following queries on the “vm\_pool” table;

```auto
update vm_pool v set v.name = REPLACE(v.name, '\\t', '') where v.name LIKE '%\\t%';
update vm_pool v set v.body = REPLACE(v.body, '\\t', '') where v.body LIKE '%\\t%';
update vm_pool v set v.short_body = REPLACE(v.short_body, '\\t', '') where v.short_body LIKE '%\\t%';

```

Query outputs:

 ![image](https://canada1.discourse-cdn.com/flex031/uploads/opennebula/original/2X/3/30056e0974fab4e13b2265bae3d15900bf49d189.jpeg)

but no changes have been performed, as the problematic record isn’t created on the DB.  
The error is still present on the Daemon (oned.log) (both on leader and follower hosts) even after restarting alll the services.

Is there any cache or stack where the API stores the operations or something that can we cleanup on the different frontends?

Thanks you for the support!

---

<div class="post-metadata">

**Author:** ![e-discoveryinfra-tid](https://avatars.discourse-cdn.com/v4/letter/e/48db29/32.png) [@e-discoveryinfra-tid](https://forum.opennebula.io/u/e-discoveryinfra-tid)\
**Post date:** [July 10, 2025, 1:31pm UTC](https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960/7 "2025-07-10T13:31:22Z")

</div>

Finally we found a way to remove the DB error from OneD services.

The workaround and the procedure to remove the failing Insert operation was the following.

- Stop oned on 3 frontends (`systemctl stop opennebula`)
- On 1 of the frontends, create a Trigger within the “opennebula” DB (MYSQL DB backend in our case): `mysql -h [ONED_DB_SERVER] -u [ONED_DB_USER] opennebula`

```sql
DELIMITER $$
CREATE TRIGGER vm_pool_replace_fields
BEFORE INSERT ON vm_pool
FOR EACH ROW
BEGIN
    IF NEW.oid = 1339 THEN
        SET NEW.name = REPLACE(NEW.name, '\t', '');
        SET NEW.body = REPLACE(NEW.body, '\t', '');
        SET NEW.short_body = REPLACE(NEW.short_body, '\t', '');
        SET NEW.body_json = REPLACE(NEW.body_json, '\t', '');
    END IF;
END$$
DELIMITER ;

```

This trigger has a condition where oid == 1339 (our specific case), but it can be removed to perform that replace for all the Insert operations on “vm\_pool” table

- Restart oned on that frontend (`systemctl restart opennebula`)

- To replicate the DB to the rest of the frontends:

- [Optional] In our case, we’ve tuned some XML RPC parameters as initially the replication failed due to XMLRPC calls between leader and followers (scheduler)

- Start oned (`systemctl restart opennebula`) on the remaining frontend nodes and wait for replication to be completed

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex031/uploads/opennebula/original/2X/1/14d66f1130b9cd292905f575da9b68a9f574dd03.png) [@system](https://forum.opennebula.io/u/system)\
**Post date:** [July 12, 2025, 1:32pm UTC](https://forum.opennebula.io/t/oned-not-responding-after-failed-vm-creation-attempt/13960/8 "2025-07-12T13:32:49Z")

</div>

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.