Bug with App Market Disk size

I am using ONE - 5.4.6 Version new install. I created few images in image like windows ubuntu and blank datablock disk the size in image tab shows correct until I export that image to marketapps.

After I export the Image to http marketapp it shows wrong disk size, I have tried this many time and it seems to be reproducible.

All the OS TMPL disk are of 50GB and in marketapps it shows only the size of data it has written to disk for example windows disk is a 50GB disk but shows only 13GB in appmarket which is still fine because it shows 50GB inside the VM, but the main issue comes with blank datablock disk

– Created a disk with below command
qemu-img create -f qcow2 blank-50gb 50G

– uploaded the disk to image using opennebula path
– Then exported to htp marketapp using the marketapp button

Here is the output of onemarketapp:

onemarketapp list
ID NAME VERSION SIZE STAT TYPE REGTIME MARKET ZONE
66 TMPL-Debian-9-Server Server Ver 1.8G rdy img 03/07/18 Synccore-App-Market 0
67 TMPL-Ubuntu-1604-Server Server Ver 3G rdy img 03/07/18 Synccore-App-Market 0
68 TMPL-Xubuntu-1604-Desktop Desktop Ve 7.2G rdy img 03/07/18 Synccore-App-Market 0
69 TMPL-Ubuntu-16.04-Desktop Desktop Ve 7.6G rdy img 03/07/18 Synccore-App-Market 0
70 TMPL-CentOS-6-9-Server Server Ver 1.6G rdy img 03/08/18 Synccore-App-Market 0
71 TMPL-CentOS-7-Server Server Ver 1.6G rdy img 03/08/18 Synccore-App-Market 0
72 TMPL-OPNSense-Firewall Firewall V 1.3G rdy img 03/08/18 Synccore-App-Market 0
73 TMPL-PFSense-Firewall Firewall V 863M rdy img 03/08/18 Synccore-App-Market 0
74 TMPL-Windows-2008R2-Deskt Desktop Ve 7.6G rdy img 03/08/18 Synccore-App-Market 0
75 TMPL-Windows-2012R2-Deskt Desktop Ve 8.9G rdy img 03/08/18 Synccore-App-Market 0
76 TMPL-Windows-2012R2-Serve Server Ver 8.9G rdy img 03/08/18 Synccore-App-Market 0
77 TMPL-Windows-2016R2-Serve Server Ver 12.9G rdy img 03/08/18 Synccore-App-Market 0
78 TMPL-Windows-2008R2-Serve Server Ver 7.9G rdy img 03/10/18 Synccore-App-Market 0
79 TMPL-Windows-2016R2-Deskt Desktop Ve 12.9G rdy img 03/10/18 Synccore-App-Market 0
80 ISO-Gparted-LiveCD ISO Versio 291M rdy img 03/12/18 Synccore-App-Market 0
81 ISO-Synccore-Recovery-Med ISO Versio 266M rdy img 03/12/18 Synccore-App-Market 0
84 ODB-Blank-50GB 0.0 1M rdy img 03/12/18 Synccore-App-Market 0

The last disk id 84 is a blank disk and shows 1 MB and when attached to OS it shows the same size while in qemu command it shows correct size.

qemu-img info 98d1af30b1a0b6a429557a667d90abd0
image: 98d1af30b1a0b6a429557a667d90abd0
file format: qcow2
virtual size: 50G (53687091200 bytes)
disk size: 193K
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false

This is making our users nuts as they do not have rights to create new image but only way is to download a disk from appmarket and use it.

Any help will be appreciated.

Hi !

Sometimes this error occured when opennebula cannot move the temporary file in /var/tmp to detect the size.
Are you using a proxy ?
What do you have into /var/log/one/oned.log ?

Nicolas.

I am using nginx server, and ONE server is on different server and Sunstone is on a separate server. Sunstone and nginx are on same server

While uploading the image to image tab - I get correct size on the image tab :screenshot attached

Wed Mar 14 11:54:02 2018 [Z0][ReM][D]: Req:3472 UID:0 one.image.allocate invoked , “NAME=“blank-2TB”…”, 108
Wed Mar 14 11:54:02 2018 [Z0][ImM][I]: Copying /mnt/storage/blank-2tb to repository for image 89
Wed Mar 14 11:54:02 2018 [Z0][ReM][D]: Req:3472 UID:0 one.image.allocate result SUCCESS, 89
Wed Mar 14 11:54:02 2018 [Z0][ReM][D]: Req:2128 UID:0 one.image.info invoked , 89
Wed Mar 14 11:54:02 2018 [Z0][ReM][D]: Req:2128 UID:0 one.image.info result SUCCESS, “89<U…”
Wed Mar 14 11:54:02 2018 [Z0][ImM][I]: Image (89) copied and ready to use.
Wed Mar 14 11:54:02 2018 [Z0][InM][D]: Monitoring datastore NY-DS-IMAGE-LRS (108)
Wed Mar 14 11:54:02 2018 [Z0][ImM][D]: Datastore NY-DS-IMAGE-LRS (108) successfully monitored.


While exporting the image from image tad to marketapp http - After uploading I see 1MB size on Marketplace : screenshot attached

Wed Mar 14 11:56:59 2018 [Z0][AuM][D]: Message received: AUTHENTICATE SUCCESS 458 -

Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:1888 UID:0 one.marketapp.allocate invoked , “NAME=“blank-2TB”…”, 100
Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:1888 UID:0 one.marketapp.allocate result SUCCESS, 86
Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:6000 UID:0 one.marketapp.info invoked , 86
Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:6000 UID:0 one.marketapp.info result SUCCESS, “…”
Wed Mar 14 11:56:59 2018 [Z0][MKP][I]: Marketplace app successfully imported
Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:9184 UID:0 one.imagepool.info invoked , -2, 0, -200, -1
Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:9184 UID:0 one.imagepool.info result SUCCESS, “<IMAGE_POOL><…”
Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:5072 UID:0 one.marketapppool.info invoked , -2, -1, -1
Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:5072 UID:0 one.marketapppool.info result SUCCESS, “<MARKETPLACEAPP_POOL…”
Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:7920 UID:0 one.user.info invoked , 0
Wed Mar 14 11:56:59 2018 [Z0][ReM][D]: Req:7920 UID:0 one.user.info result SUCCESS, “0<GID…”
Wed Mar 14 11:57:03 2018 [Z0][MKP][D]: Monitoring marketplace Synccore-App-Market (100)
Wed Mar 14 11:57:03 2018 [Z0][MKP][D]: Marketplace Synccore-App-Market (100) successfully monitored.
Wed Mar 14 11:57:06 2018 [Z0][InM][D]: Monitoring datastore files (2)

app-tab

Thanks

Well it seems that the problem is in the export script - /var/lib/one/remotes/datastore/fs/export

In the script it uses du -Lm which gives the incorrect disk size as the image is thin disk qcow2 and according to that image size the DD command is then executed which creates only a 1MB disk in AppMarket.

for example the size of disk in qemu-info

qemu-img info 3a5b06d543f0e0fda62347b448bd71d7
image: 3a5b06d543f0e0fda62347b448bd71d7
file format: qcow2
virtual size: 150G (161061273600 bytes)
disk size: 195K
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false


And the size in du -Lm

du -Lm 3a5b06d543f0e0fda62347b448bd71d7
1 3a5b06d543f0e0fda62347b448bd71d7

Is there a way to use thin disk in opennebula or workaround for this issue?

Thanks

is there any solution to this issue ?

I don’t think this has been fixed yet, I have been using different solutions from past few years, but it seems opennebula is still lacking lot of production features, we are migrating to a different solution using xcp hypervisor and xen orchestra.

At least xen orchestra are adding production features which is actually needed when a production deployment is done.

The major down side for one is we need to setup lot of basic things using other third party apps when on production, like monitoring, alerts, backup, DR, failover scripts and list goes on.

One development should be more focused on ease for production and management with atleast basic features

Thanks for your quick response !! its been 3 years for now with the same bug .
i definitely understand you, just trying to find some DR solution …

You can check veeam for DR and backups, if you are using vmware hypervisor then you can do dr at hypervisor level, if u r using kvm then agent based backup and dr need to be used.

Hi @kiranranjane,

Sorry to hear you are struggling to make OpenNebula work for your specific use case. If you are missing some specific feature for your production environment, please feel free to submit a request and we’ll see what we can do: How to Report Bugs and Make Feature Requests · OpenNebula/one Wiki · GitHub

Obviously, although OpenNebula comes with lots of customization options and a very flexible modular architecture, it is not (and will never be) a “ONE-fit-all” cloud/edge solution :wink: As an open source product, it can be adapted to your specific needs, but there will always be some use cases that require integrations with third-party technologies (e.g. Rancher, WHMCS, Zabbix, etc.). Again, if you are missing some interesting integration that might benefit the OpenNebula Community as a whole, please let us know! :nerd_face:

Cheers,

Alberto.