Local-datastore image & system with Persistent image is not linking and corrupts image on delete

Community Support Storage
1

Hi, I’m been using OpenNebula successfully for a few months now with “shared” NF storage between my Front-End and all KVM Node. (Nice Product!)

I have a new user that has a server with no network storage options, but has ample local disk storage from 800G SSDs. I’ve been looking at using OpenNebula local-datastore support as defined here:

It’s taken me a bit to get this almost right? I was able to add the system datastore and image datastore. The image data is properly showing the 736G disk with proper available storage. Output below.

I was able to Import a 64G qcow2 image from the Front-end to the KVM node into datastore ID112. It performed ssh copy and dir 112 creation on the node all on its own.

The image datastore is tied to this system datastore COMPATIBLE_SYS_DS=“109”

I have bridge-list set for this single KVM host as well.

I can launch the VM via VM template with the IMAGE set to persistent in the image data store just fine. The VM comes up with my Ubuntu image.

However, unlike my using persistent images on NFS shared env, it made a copy of the qcow2 image to disk.0, not an symbolic link. This doesn’t seem right to me and I can’t seem to find my mistake?

It gets worse, when I terminate the VM. The original image is wiped out with a tiny .snap file.

Below is all the show output after launching the VM and before VM release.
Here is the corrupted image after vm terminate.

root@l14-kvm-tools-host:/var/lib/one/datastores# ls -lrht 112
total 4.0K
drwxrwxr-x 2 oneadmin oneadmin 4.0K Mar  6 23:47 682011c2c808524b8e4e9ff189f4860a.snap
lrwxrwxrwx 1 oneadmin oneadmin   38 Mar  6 23:47 682011c2c808524b8e4e9ff189f4860a -> 682011c2c808524b8e4e9ff189f4860a.snap/
root@l14-kvm-tools-host:/var/lib/one/datastores# du
4       ./107
4       ./102
4       ./112/682011c2c808524b8e4e9ff189f4860a.snap
8       ./112
8       ./109
8       ./100
36      .
root@l14-kvm-tools-host:/var/lib/one/datastores# ls -lrht 109
total 0
root@l14-kvm-tools-host:/var/lib/one/datastores# ls -lrht 112 */*
lrwxrwxrwx 1 oneadmin oneadmin   38 Mar  6 23:47 112/682011c2c808524b8e4e9ff189f4860a -> 682011c2c808524b8e4e9ff189f4860a.snap/

112/682011c2c808524b8e4e9ff189f4860a.snap:
total 0
lrwxrwxrwx 1 oneadmin oneadmin 1 Mar  6 23:47 682011c2c808524b8e4e9ff189f4860a.snap -> .

112:
total 4.0K
drwxrwxr-x 2 oneadmin oneadmin 4.0K Mar  6 23:47 682011c2c808524b8e4e9ff189f4860a.snap
lrwxrwxrwx 1 oneadmin oneadmin   38 Mar  6 23:47 682011c2c808524b8e4e9ff189f4860a -> 682011c2c808524b8e4e9ff189f4860a.snap/
root@l14-kvm-tools-host:/var/lib/one/datastores#

Am I missing some persistent setting somewhere?

I’m watching various logs with debug level turned up to max, but nothing
is jumping out with error or failure.

Thanks for your time/help and guidance,

–Ken

BEFORE RELEASING THE VM

root@l14-kvm-tools-host:/var/lib/one/datastores# ls -lrth */*
-rw-rw-r-- 1 oneadmin oneadmin  17G Mar  6 23:16 112/682011c2c808524b8e4e9ff189f4860a

109/40:
total 17G
-rw-r--r-- 1 oneadmin oneadmin 364K Mar  6 23:18 disk.1
-rw-rw-r-- 1 oneadmin oneadmin 4.3K Mar  6 23:18 vm.xml
-rw-rw-r-- 1 oneadmin oneadmin 1.2K Mar  6 23:18 ds.xml
-rw-rw-r-- 1 oneadmin oneadmin 1.9K Mar  6 23:18 deployment.0
-rw-rw-r-- 1 oneadmin oneadmin  17G Mar  6 23:40 disk.0
root@l14-kvm-tools-host:/var/lib/one/datastores# 

root@l14-kvm-tools-host:/var/lib/one/datastores# ls -lrht 112
total 17G
-rw-rw-r-- 1 oneadmin oneadmin 17G Mar  6 23:16 682011c2c808524b8e4e9ff189f4860a
root@l14-kvm-tools-host:/var/lib/one/datastores# 



oneadmin@lab-openneb-1:/var/log/one$ onedatastore show 10.84.107.94-image 
DATASTORE 112 INFORMATION                                                       
ID             : 112                 
NAME           : 10.84.107.94-image  
USER           : oneadmin            
GROUP          : oneadmin            
CLUSTERS       : 100                 
TYPE           : IMAGE               
DS_MAD         : fs                  
TM_MAD         : shared              
BASE PATH      : /var/lib/one//datastores/112
DISK_TYPE      : FILE                
STATE          : READY               

DATASTORE CAPACITY                                                              
TOTAL:         : 731.4G              
FREE:          : 237.5G              
USED:          : 456.6G              
LIMIT:         : -                   

PERMISSIONS                                                                     
OWNER          : um-                 
GROUP          : u--                 
OTHER          : ---                 

DATASTORE TEMPLATE                                                              
ALLOW_ORPHANS="FORMAT"
BRIDGE_LIST="10.84.107.94"
CLONE_TARGET="SYSTEM"
CLONE_TARGET_SSH="SYSTEM"
CLUSTERS="100"
COMPATIBLE_SYS_DS="109"
CONVERT="DRIVER"
DATASTORE_CAPACITY_CHECK="yes"
DISK_TYPE="FILE"
DISK_TYPE_SSH="FILE"
DS_MAD="fs"
LN_TARGET="NONE"
LN_TARGET_SSH="SYSTEM"
RESTRICTED_DIRS="/"
SAFE_DIRS="/var/tmp"
TM_MAD="shared"
TM_MAD_SYSTEM="ssh"
TYPE="IMAGE_DS"

IMAGES         
16             
oneadmin@lab-openneb-1:/var/log/one$ 

oneadmin@lab-openneb-1:/var/log/one$ onedatastore show 10.84.107.94-system 
DATASTORE 109 INFORMATION                                                       
ID             : 109                 
NAME           : 10.84.107.94-system 
USER           : oneadmin            
GROUP          : oneadmin            
CLUSTERS       : 100                 
TYPE           : SYSTEM              
DS_MAD         : -                   
TM_MAD         : ssh                 
BASE PATH      : /var/lib/one//datastores/109
DISK_TYPE      : FILE                
STATE          : READY               

DATASTORE CAPACITY                                                              
TOTAL:         : -                   
FREE:          : -                   
USED:          : -                   
LIMIT:         : -                   

PERMISSIONS                                                                     
OWNER          : um-                 
GROUP          : u--                 
OTHER          : ---                 

DATASTORE TEMPLATE                                                              
ALLOW_ORPHANS="YES"
DATASTORE_CAPACITY_CHECK="YES"
DISK_TYPE="FILE"
DS_MIGRATE="YES"
NO_DECOMPRESS="NO"
QCOW2_STANDALONE="NO"
RESTIC_SPARSIFY="NO"
RESTRICTED_DIRS="/"
RSYNC_SPARSIFY="NO"
SAFE_DIRS="/var/tmp"
SHARED="NO"
TM_MAD="ssh"
TYPE="SYSTEM_DS"

IMAGES         
oneadmin@lab-openneb-1:/var/log/one$ 

oneadmin@lab-openneb-1:/var/log/one$ onetemplate show  Clone-ubuntu2204-UPF2-8vC
TEMPLATE 4 INFORMATION                                                          
ID             : 4                   
NAME           : Clone-ubuntu2204-UPF2-8vCPU-40G-convert_9
USER           : oneadmin            
GROUP          : oneadmin            
LOCK           : None                
REGISTER TIME  : 03/06 19:00:38      

PERMISSIONS                                                                     
OWNER          : um-                 
GROUP          : ---                 
OTHER          : ---                 

TEMPLATE CONTENTS                                                               
CONTEXT=[
  NETWORK="NO" ]
CPU="0.0833333333333333"
DISK=[
  IMAGE="Clone-ubuntu2204-UPF2-8vCPU-40G-convert_9",
  IMAGE_UNAME="oneadmin" ]
GRAPHICS=[
  LISTEN="0.0.0.0",
  TYPE="VNC" ]
HYPERVISOR="kvm"
MEMORY="32768"
NIC=[
  METHOD="skip",
  MODEL="vmxnet3",
  NETWORK="service",
  NETWORK_UID="0",
  NETWORK_UNAME="oneadmin" ]
SCHED_REQUIREMENTS="(HYPERVISOR=kvm)"
VCPU="8"
oneadmin@lab-openneb-1:/var/log/one$ 

oneadmin@lab-openneb-1:/var/log/one$ oneimage show Clone-ubuntu2204-UPF2-8vCPU-4
IMAGE 16 INFORMATION                                                            
ID             : 16                  
NAME           : Clone-ubuntu2204-UPF2-8vCPU-40G-convert_9
USER           : oneadmin            
GROUP          : oneadmin            
LOCK           : None                
DATASTORE      : 10.84.107.94-image  
TYPE           : OS                  
REGISTER TIME  : 03/06 23:14:47      
PERSISTENT     : Yes                 
SOURCE         : /var/lib/one//datastores/112/682011c2c808524b8e4e9ff189f4860a
PATH           : /var/tmp/Clone-ubuntu2204-UPF2-8vCPU-40G-convert_9.qcow2
FORMAT         : qcow2               
SIZE           : 64G                 
STATE          : used                
RUNNING_VMS    : 1                   

PERMISSIONS                                                                     
OWNER          : um-                 
GROUP          : ---                 
OTHER          : ---                 

IMAGE TEMPLATE                                                                  
DESCRIPTION="laasdev"
DEV_PREFIX="sd"

VIRTUAL MACHINES

  ID USER     GROUP    NAME                                                     
  40 oneadmin oneadmin kentest2                                                 
oneadmin@lab-openneb-1:/var/log/one$

oneadmin@lab-openneb-1:/var/log/one$ onevm show 40
VIRTUAL MACHINE 40 INFORMATION                                                  
ID                  : 40                  
NAME                : kentest2            
USER                : oneadmin            
GROUP               : oneadmin            
STATE               : ACTIVE              
LCM_STATE           : RUNNING             
LOCK                : None                
RESCHED             : No                  
HOST                : 10.84.107.94        
CLUSTER ID          : 100                 
CLUSTER             : omega               
START TIME          : 03/06 23:18:09      
END TIME            : -                   
DEPLOY ID           : 58444f4c-50b4-40cb-9e1d-108513d76455

VIRTUAL MACHINE MONITORING                                                      
CPU                 : 2.0                 
MEMORY              : 18.2G               
NETTX               : 0K                  
NETRX               : 0K                  
DISKRDBYTES         : 527782590           
DISKRDIOPS          : 26396               
DISKWRBYTES         : 32900096            
DISKWRIOPS          : 1670                
ID                  : 40                  
TIMESTAMP           : 1741303426          
VM_QEMU_PING        : QEMU Guest Agent monitoring disabled

PERMISSIONS                                                                     
OWNER               : um-                 
GROUP               : ---                 
OTHER               : ---                 

VM DISKS                                                                        
 ID DATASTORE  TARGET IMAGE                               SIZE      TYPE SAVE
  0 10.84.107. sda    Clone-ubuntu2204-UPF2-8vCPU-40G-con 16.5G/64G file  YES
  1 -          hda    CONTEXT                             1M/-      -       -

VM NICS                                                                         
 ID NETWORK              BRIDGE       IP              MAC               PCI_ID  
  0 service              onebr.111    -               02:00:84:8e:60:c8

SECURITY                                                                        

NIC_ID NETWORK                   SECURITY_GROUPS                                
     0 service                   0

SECURITY GROUP   TYPE     PROTOCOL NETWORK                       RANGE          
  ID NAME                          VNET START             SIZE                  
   0 default     OUTBOUND ALL
   0 default     INBOUND  ALL

VIRTUAL MACHINE HISTORY                                                         
SEQ UID  REQ   HOST         ACTION       DS           START        TIME     PROL
  0 -    -     10.84.107.94 none        109  03/06 23:18:15   0d 00h05m   0h00m1

USER TEMPLATE                                                                   
HYPERVISOR="kvm"
SCHED_REQUIREMENTS="(HYPERVISOR=kvm)"

VIRTUAL MACHINE TEMPLATE                                                        
AUTOMATIC_DS_REQUIREMENTS="(\"CLUSTERS/ID\" @> 100) & (\"ID\" @> 109)"
AUTOMATIC_NIC_REQUIREMENTS="(\"CLUSTERS/ID\" @> 100)"
AUTOMATIC_REQUIREMENTS="(CLUSTER_ID = 100) & !(PUBLIC_CLOUD = YES) & !(PIN_POLIC
CONTEXT=[
  DISK_ID="1",
  NETWORK="NO",
  TARGET="hda" ]
CPU="0.0833333"
GRAPHICS=[
  LISTEN="0.0.0.0",
  PORT="5940",
  TYPE="VNC" ]
MEMORY="32768"
OS=[
  UUID="58444f4c-50b4-40cb-9e1d-108513d76455" ]
TEMPLATE_ID="4"
TM_MAD_SYSTEM="ssh"
VCPU="8"
VMID="40"
oneadmin@lab-openneb-1:/var/log/one$
2

BTW this is with one-deploy fresh install from v6.10.0.1

I’ve been looking at this for another full day. I believe I have a better understanding of the different TM_MAD_CONF types in oned.conf and local-datastores.

In my above case I had added the local-data store with the following settings with similar CLI command:

oneadmin@lab-openneb-1:~/ken$ cat datastore-local.conf 
NAME      = "10.84.107.94-image"
TYPE      = "IMAGE_DS"
DS_MAD    = "fs"
TM_MAD    = "shared"
BRIDGE_LIST = "10.84.107.94"
COMPATIBLE_SYS_DS = "109"
DATASTORE_CAPACITY_CHECK = "yes"
CLUSTERS = "100"
CONVERT = "DRIVER"
oneadmin@lab-openneb-1:~/ken$  onedatastore create ./datastore-local.conf

Note I have BRIDGE_LIST set to the 1 host that will use this Image and system datastore with no network storage available.

When I created the above datastore it created the 112 directory only on kvm host 10.84.107.94 … No directory 112 on the front-end datastore directory. When I imported the image with below CLI command it copied the qcow2 image to dirctory 112 on the kvm host 10.84.107.94 only. This all lead me to believe it could run everything locally on kvm node 10.84.107.94, without needing a copy of the image on the front-end too?

oneadmin@lab-openneb-1:~/ken$ cat image.tmpl 
NAME=Clone-ubuntu2204-UPF2-8vCPU-40G-convert_9
PATH=/var/tmp/Clone-ubuntu2204-UPF2-8vCPU-40G-convert_9.qcow2
#SOURCE=/var/tmp/Clone-ubuntu2204-UPF2-8vCPU-40G-convert_9.qcow2
TYPE=OS
SIZE=65536
PERSISTENT=YES
oneadmin@lab-openneb-1:~/ken$  oneimage create -d 114 image.tmpl

A couple questions

  1. is above a supported configuration without the image on Front-end? Is image datastore create/import w/single KVM BRIDGE_LIST set expected to perform this way?

  2. I’ve changed to creating the datastore with TM_MAD = “ssh”. vs "shared’ originally. If the BRIDGE_LIST var is set it behaves the same way putting the Image datastore on the KVM host only. Nothing on the Front-end. Same behavior on qcow2 import too. The main difference here is at instantiation the VM template will fail in ssh mode as the image is not found on the Front-end to tar/copy.

  3. It seems that with any image datastore configuration the original image must be accessible on the datastore via the front-end. Is that correct? I have local ssh copy working as long as I don’t create the image datastore with the BRIDGE_LIST setting. Then everything goes to the Front-end on import/setup. TM_MAD = “ssh” works with copy or persistent image copy back to front-end on VM delete.

Thanks again for you time and guidance,

–Ken

3

Hi

Maybe my post is too messy or missing the mark. Can anyone confirm that setting a Bridge_list on IMAGE datastore “create” is support for SSH mode?

oneadmin@lab-openneb-1:~/ken$ cat datastore-local-2.conf 
NAME      = "10.84.107.94-image-2"
TYPE      = "IMAGE_DS"
DS_MAD    = "fs"
TM_MAD    = "ssh"
BRIDGE_LIST = "10.84.107.94" <<<<<<<<<<<<<< HERE 
COMPATIBLE_SYS_DS = "109"
DATASTORE_CAPACITY_CHECK = "yes"
#CLUSTERS = "100"
#CONVERT = "DRIVER"
oneadmin@lab-openneb-1:~/ken$

If supported where is the image datastore expected to live on the One Front-End or the KVM node itself?

Thanks for the clarification,

–Ken