[one.vm.migrate] Error getting datastore

Community Support Storage
1

I can’t migrate (offline/live) anymore. This affects multiple VMs
the error is Error getting datastore ID 103

opennebula-node 5.6.1-1

onedatastore list -x
please see here https://pastebin.com/M3W08sCT

onevm show -x 717
https://pastebin.com/Ag4J4qSR

oned.log

Mon Jun 3 18:57:33 2019 [Z0][ReM][D]: Req:5392 UID:0 one.vm.migrate invoked , 717, 30, true, false, -1
Mon Jun 3 18:57:33 2019 [Z0][ReM][E]: Req:5392 UID:0 one.vm.migrate result FAILURE [one.vm.migrate] Error getting datastore [103].

Template shows:
User template

HYPERVISOR = “kvm”
INPUTS_ORDER = “”
LOGO = “images/logos/ubuntu.png”
MEMORY_UNIT_COST = “MB”
SCHED_MESSAGE = “Mon Jun 3 17:10:06 2019 : Cannot dispatch VM to any Host. Possible reasons: Not enough capacity in Host or System DS, or dispatch limit reached”

VM log:

Sun Feb 24 16:30:47 2019 [Z0][VM][I]: New LCM state is ACTIVE
Sun Feb 24 16:30:47 2019 [Z0][VM][I]: New LCM state is PROLOG
Sun Feb 24 16:30:48 2019 [Z0][VM][I]: New LCM state is BOOT
Sun Feb 24 16:30:48 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/717/deployment.0
Sun Feb 24 16:30:49 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Sun Feb 24 16:30:49 2019 [Z0][VMM][I]: ExitCode: 0
Sun Feb 24 16:30:49 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Sun Feb 24 16:30:51 2019 [Z0][VMM][I]: ExitCode: 0
Sun Feb 24 16:30:51 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
Sun Feb 24 16:30:51 2019 [Z0][VMM][I]: ExitCode: 0
Sun Feb 24 16:30:51 2019 [Z0][VMM][I]: Successfully execute network driver operation: post.
Sun Feb 24 16:30:51 2019 [Z0][VM][I]: New LCM state is RUNNING
Sun Feb 24 16:38:25 2019 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Sun Feb 24 16:38:31 2019 [Z0][VMM][I]: ExitCode: 0
Sun Feb 24 16:38:31 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: shutdown.
Sun Feb 24 16:38:32 2019 [Z0][VMM][I]: ExitCode: 0
Sun Feb 24 16:38:32 2019 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Sun Feb 24 16:38:32 2019 [Z0][VM][I]: New state is POWEROFF
Sun Feb 24 16:38:32 2019 [Z0][VM][I]: New LCM state is LCM_INIT
Sun Feb 24 16:39:13 2019 [Z0][VM][I]: New state is ACTIVE
Sun Feb 24 16:39:13 2019 [Z0][VM][I]: New LCM state is BOOT_POWEROFF
Sun Feb 24 16:39:13 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/717/deployment.1
Sun Feb 24 16:39:13 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Sun Feb 24 16:39:14 2019 [Z0][VMM][I]: ExitCode: 0
Sun Feb 24 16:39:14 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Sun Feb 24 16:39:15 2019 [Z0][VMM][I]: ExitCode: 0
Sun Feb 24 16:39:15 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
Sun Feb 24 16:39:15 2019 [Z0][VMM][I]: ExitCode: 0
Sun Feb 24 16:39:15 2019 [Z0][VMM][I]: Successfully execute network driver operation: post.
Sun Feb 24 16:39:15 2019 [Z0][VM][I]: New LCM state is RUNNING
Sat Mar 2 11:20:55 2019 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Sat Mar 2 11:21:01 2019 [Z0][VMM][I]: ExitCode: 0
Sat Mar 2 11:21:01 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: shutdown.
Sat Mar 2 11:21:02 2019 [Z0][VMM][I]: ExitCode: 0
Sat Mar 2 11:21:02 2019 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Sat Mar 2 11:21:02 2019 [Z0][VM][I]: New state is POWEROFF
Sat Mar 2 11:21:02 2019 [Z0][VM][I]: New LCM state is LCM_INIT
Sat Mar 2 11:26:47 2019 [Z0][VM][I]: New state is ACTIVE
Sat Mar 2 11:26:47 2019 [Z0][VM][I]: New LCM state is BOOT_POWEROFF
Sat Mar 2 11:26:47 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/717/deployment.2
Sat Mar 2 11:26:47 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Sat Mar 2 11:26:47 2019 [Z0][VMM][I]: ExitCode: 0
Sat Mar 2 11:26:47 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Sat Mar 2 11:26:49 2019 [Z0][VMM][I]: ExitCode: 0
Sat Mar 2 11:26:49 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
Sat Mar 2 11:26:49 2019 [Z0][VMM][I]: ExitCode: 0
Sat Mar 2 11:26:49 2019 [Z0][VMM][I]: Successfully execute network driver operation: post.
Sat Mar 2 11:26:49 2019 [Z0][VM][I]: New LCM state is RUNNING
Mon Jun 3 16:24:55 2019 [Z0][VM][I]: New state is ACTIVE
Mon Jun 3 16:24:55 2019 [Z0][VM][I]: New LCM state is DISK_RESIZE
Mon Jun 3 16:24:55 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_resize.
Mon Jun 3 16:24:56 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 16:24:56 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: resize_disk.
Mon Jun 3 16:24:56 2019 [Z0][VMM][I]: VM disk successfully resized
Mon Jun 3 16:24:56 2019 [Z0][VM][I]: New LCM state is RUNNING
Mon Jun 3 16:24:56 2019 [Z0][LCM][I]: VM disk resize operation completed.
Mon Jun 3 16:48:02 2019 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Mon Jun 3 16:48:10 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 16:48:10 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: shutdown.
Mon Jun 3 16:48:10 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 16:48:10 2019 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Mon Jun 3 16:48:10 2019 [Z0][VM][I]: New state is POWEROFF
Mon Jun 3 16:48:10 2019 [Z0][VM][I]: New LCM state is LCM_INIT
Mon Jun 3 16:48:29 2019 [Z0][VM][I]: New state is ACTIVE
Mon Jun 3 16:48:29 2019 [Z0][VM][I]: New LCM state is BOOT_POWEROFF
Mon Jun 3 16:48:29 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/717/deployment.4
Mon Jun 3 16:48:29 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Mon Jun 3 16:48:30 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 16:48:30 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Mon Jun 3 16:48:31 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 16:48:31 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
Mon Jun 3 16:48:31 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 16:48:31 2019 [Z0][VMM][I]: Successfully execute network driver operation: post.
Mon Jun 3 16:48:31 2019 [Z0][VM][I]: New LCM state is RUNNING
Mon Jun 3 17:10:21 2019 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Mon Jun 3 17:10:28 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 17:10:28 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: shutdown.
Mon Jun 3 17:10:28 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 17:10:28 2019 [Z0][VMM][I]: Successfully execute network driver operation: clean.
Mon Jun 3 17:10:28 2019 [Z0][VM][I]: New state is POWEROFF
Mon Jun 3 17:10:28 2019 [Z0][VM][I]: New LCM state is LCM_INIT
Mon Jun 3 17:12:46 2019 [Z0][VM][I]: New state is ACTIVE
Mon Jun 3 17:12:46 2019 [Z0][VM][I]: New LCM state is BOOT_POWEROFF
Mon Jun 3 17:12:46 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/717/deployment.5
Mon Jun 3 17:12:46 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Mon Jun 3 17:12:47 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 17:12:47 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Mon Jun 3 17:12:48 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 17:12:48 2019 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
Mon Jun 3 17:12:48 2019 [Z0][VMM][I]: ExitCode: 0
Mon Jun 3 17:12:48 2019 [Z0][VMM][I]: Successfully execute network driver operation: post.
Mon Jun 3 17:12:48 2019 [Z0][VM][I]: New LCM state is RUNNING

2

It seems that all the history records of the VM makes reference to datastore 103. It is possible that you deleted that datastore?

3

I am aware but the only thing saved in 103 is the context cd file. The OS disk is actually on ceph and works fine, but somehow the migration fails due to the context disk.
Neither my colleague nor me remembering deleting a datastore (which we’d probably would have noticed). And I can’t think of any other data storage that we could have used, we have it like this since the very beginning.