Ceph S3 private market

Integration Support Marketplace Appliances
1

Please, describe the problem here and provide additional information below (if applicable) …

Versions of the related components and OS (frontend, hypervisors, VMs): OpenNebula 7.0.1 and cephadm based Ceph 19.2.3 on Ubuntu 24.04.3 LTS (GNU/Linux 6.8.0-94-generic x86_64) in a HCI configuration.

Steps to reproduce: Setup a Ceph S3 bucket and connect OpenNebula via a private MarketPlace. There are two HA proxies (container within cephadm) with two virtual IPs in front of the Ceph RGW. These two VIPs are setup with the same DNS to be redundant, and loadbalancing.

Current results:
When I create images they show up under apps in the private marketplace, but when I create apps from vm’s I see that they have been transfered to Ceph by the used data size of the S3 bucket, but the VM don’t turn up as an app in the Private MarkedPlace. I see the same strange behavior for VM templates as with VMs.

Expected results:
Expected results would be to also see the VMs as apps under the private MarketPlace.

Does anybody have any idea why this is the case for me?

The last time I installed OpenNebula this worked, but at that time I did not use HA Proxy. Of this reason I suspect it can have someting to do with this. Maybe someone knows if there are some special ajustments that is needed when the S3 connection from OpenNebula passes through HA-Proxy?

2

Hello @strandte !

To isolate the issue have you tried to remove the HA proxy from the equation and use S3 gateway as an endpoint for the marketplace?
Could you, please, share your s3 marketplace config here (e.g. the output of the onemarket show -j <mp_id> . Remember to remove any sensitive information from the output before sharing it here.

3

God idea to share the config:
oneadmin@svonefront-pub:~$ onemarket show -j 103
{
“MARKETPLACE”: {
“ID”: “103”,
“UID”: “0”,
“GID”: “0”,
“UNAME”: “oneadmin”,
“GNAME”: “oneadmin”,
“NAME”: “Hbr MarketPlace”,
“STATE”: “0”,
“MARKET_MAD”: “s3”,
“ZONE_ID”: “0”,
“TOTAL_MB”: “1048576”,
“FREE_MB”: “1048576”,
“USED_MB”: “0”,
“MARKETPLACEAPPS”: {},
“PERMISSIONS”: {
“OWNER_U”: “1”,
“OWNER_M”: “1”,
“OWNER_A”: “0”,
“GROUP_U”: “0”,
“GROUP_M”: “0”,
“GROUP_A”: “0”,
“OTHER_U”: “0”,
“OTHER_M”: “0”,
“OTHER_A”: “0”
},
“TEMPLATE”: {
“ACCESS_KEY_ID”: “XXXXXXXXXXXXXXXXXXXX”,
“AWS”: “NO”,
“BUCKET”: “hbr.opennebula.markedplace”,
“DESCRIPTION”: “Hbr MarketPlace”,
“ENDPOINT”: “http://rgw.example.com:8080”,
“FORCE_PATH_STYLE”: “YES”,
“MARKET_MAD”: “s3”,
“REGION”: “default”,
“SECRET_ACCESS_KEY”: “XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX”,
“SIGNATURE_VERSION”: “s3”
}
}
}

I am as I told able to save images to the S3 bucket by using this command:

oneadmin@svonefront:~$ onemarketapp create --name ‘test’ --image 131 --market “Hbr MarketPlace”
ID: 154

You can se the file with awscli:

C:\Program Files\Amazon\AWSCLIV2>aws s3 ls s3://hbr.opennebula.markedplace/ --recursive --human-readable --endpoint-url http://rgw.example.com:8080
2026-02-04 14:35:17 1.2 MiB marketapp-154

I seem to be able to save a vm to the marked place with the command:

oneadmin@svonefront:~$ onemarketapp vm import 28 --vmname ‘WIN-SRV-fast’ --market 103
Do you want to import images too? (yes/no, default=yes): yes
ID: 155

To delete saved template use: onetemplate delete 73 --recursive

As you see it is no error messages!

But when I check with awscli it is not there, only the image transfered earlier:

C:\Program Files\Amazon\AWSCLIV2>aws s3 ls s3://hbr.opennebula.markedplace/ --recursive --human-readable --endpoint-url http://rgw.example.com:8080
2026-02-04 14:35:17 1.2 MiB marketapp-154

If I try to transfer an image which is in used or error state I get an error like this:
oneadmin@svonefront:~$ onemarketapp create --name ‘test34’ --image 138 --market “Hbr MarketPlace”
[one.marketapp.allocate] Cannot clone image in current state
This makes me believe that if the VM have some of the images in wrong state for transferring to the market place it would probably give a similar error?

4

I have tried to use the RGW directly, by pointing it to http://:8081.

RGW uses port 8081 and HA-Proxy uses port 8080. Both with OpenNebula and awscli the results is no different when going through HA-proxy or directly, so my question about extra settings for HA-proxy can probably be forgotten about.

5

Also, I do not seem to have found the correct logs for troubleshooting this problem. Where should I look for logs?

6

Marketplace related events are written to the /var/log/one/oned.log file.

We can’t reproduce the issue in our lab.

Do you have any relevant error messages in ceph logs?

Does replacing s3 with v4 in SIGNATURE_VERSION change the behavior?

According to our documentation, possible values for that attribute are s3, v2 and v4:

SIGNATURE_VERSION Depends on the S3 implementation, possible values are s3, v2 or v4

To change the value for that attribute one can execute onemarket update <marketplace_id> command, make changes and save them. After that, please, try to repeat the failing operation.

As far your question:

shouldn’t trigger such error for used images (both persistent and non-persistent ones) but it’s raised for the images in the error state.

What is the output of the oneimage show -j 138 command? What state does that image have?

7

oneadmin@svonefront-pub:~$ oneimage show -j 138
{
“IMAGE”: {
“ID”: “138”,
“UID”: “2”,
“GID”: “0”,
“UNAME”: “tstrand”,
“GNAME”: “oneadmin”,
“NAME”: “WINSRVOPN2-disk-1”,
“PERMISSIONS”: {
“OWNER_U”: “1”,
“OWNER_M”: “1”,
“OWNER_A”: “0”,
“GROUP_U”: “0”,
“GROUP_M”: “0”,
“GROUP_A”: “0”,
“OTHER_U”: “0”,
“OTHER_M”: “0”,
“OTHER_A”: “0”
},
“TYPE”: “1”,
“DISK_TYPE”: “3”,
“PERSISTENT”: “1”,
“REGTIME”: “1769779382”,
“MODTIME”: “1769779382”,
“SOURCE”: “one-ceph/one-138”,
“PATH”: “one-ceph/one-126”,
“FORMAT”: “raw”,
“FS”: “”,
“SIZE”: “754”,
“STATE”: “8”,
“PREV_STATE”: “8”,
“RUNNING_VMS”: “1”,
“CLONING_OPS”: “0”,
“CLONING_ID”: “-1”,
“TARGET_SNAPSHOT”: “-1”,
“DATASTORE_ID”: “112”,
“DATASTORE”: “one-ceph”,
“VMS”: {
“ID”: “28”
},
“CLONES”: {},
“APP_CLONES”: {},
“TEMPLATE”: {
“DEV_PREFIX”: “hd”,
“FROM_APP”: “111”,
“FROM_APP_MD5”: “9e650d0e7c6e017a91ca299c8f7ed766”,
“FROM_APP_NAME”: “Windows VirtIO Drivers - v0.1.285”
},
“SNAPSHOTS”: {
“ALLOW_ORPHANS”: “NO”,
“CURRENT_BASE”: “-1”,
“NEXT_SNAPSHOT”: “0”
},
“BACKUP_INCREMENTS”: {},
“BACKUP_DISK_IDS”: {}
}
}

Even if i changed from s3 to v4 it does not seem to be able to transferer a VM to the s3 bucket.

I have been busy with other things for a while, but only wanted to answer quickly your questions. I will look more deeply into this later.

8

Hello @strandte !

The provided information helped to identify the issue: the image #138 is CDROM (TYPE=1) and it’s persistent (PERSISTENT=1). According to its other attributes (e.g. “FROM_APP_NAME”: “Windows VirtIO Drivers - v0.1.285” and “SIZE”: “754”) it looks like it’s Windows VirtIO ISO disk.
We could reproduce the issue in our lab: indeed the commands
onemarketapp vm import <vm_id> --vmname <name> --market <mp_id> --yes and onemarketapp vm-template import <vm_id> --vmname <name> --market <mp_id> --yes
do not upload files to s3 private marketplace and don’t provide any hint/error.

While we are clarifying it with our developers the workaround is to use non-persistent CDROM/ISO images. We also will clarify if assigning persistent flag to read-only ISO/CDROM disks is a valid operation.

So please try to detach the image #138 from the VM, make that image non-persistent one, attach it back to the VM and create app in the s3 private marketplace.

Another test could be to instantiate any test VM with disk type OS and try to create an app from it in your S3 private marketplace.

9

Spot on! That was the issue. Thanks!

10

Correction! It seems to work, as the VM shows up under Apps in the Markedplace, but it does not show up in the S3 storage when I query it with the aws cli tool, and I’m not able to import it from the market place either.

11

This is what shows up in the oned.log while I try to “Create App” of the VM that now does not have any persistent CD images. The VM currently have only a persistent VHD and the CONTEXT hdd.

oneadmin@svonefront:~$ tail -f /var/log/one/oned.log
Mon Feb 16 10:25:34 2026 [Z0][ReM][D]: Req:1488 UID:2 IP:127.0.0.1 one.zonepool.info invoked
Mon Feb 16 10:25:34 2026 [Z0][ReM][D]: Req:1488 UID:2 one.zonepool.info result SUCCESS, “<ZONE_POOL><ID…”
Mon Feb 16 10:25:34 2026 [Z0][ReM][D]: Req:2704 UID:2 IP:127.0.0.1 one.datastorepool.info invoked , 0
Mon Feb 16 10:25:34 2026 [Z0][ReM][D]: Req:2704 UID:2 one.datastorepool.info result SUCCESS, “<DATASTORE_POOL><DAT…”
Mon Feb 16 10:25:43 2026 [Z0][ReM][D]: Req:688 UID:2 IP:127.0.0.1 one.marketpool.info invoked
Mon Feb 16 10:25:43 2026 [Z0][ReM][D]: Req:688 UID:2 one.marketpool.info result SUCCESS, “<MARKETPLACE_POOL><M…”
Mon Feb 16 10:25:46 2026 [Z0][PLM][I]: Starting Plan Manager timer action…
Mon Feb 16 10:25:46 2026 [Z0][PLM][I]: Found 0 active plans
Mon Feb 16 10:25:53 2026 [Z0][DBM][I]: Purging obsolete LogDB records: 0 records purged. Log state: 0,0 - 0,0
Mon Feb 16 10:25:53 2026 [Z0][DBM][I]: Purging obsolete federated LogDB records: 0 records purged. Federated log size: 0.
Mon Feb 16 10:25:59 2026 [Z0][ReM][D]: Req:6288 UID:0 IP:127.0.0.1 one.vm.info invoked , 28, false
Mon Feb 16 10:25:59 2026 [Z0][ReM][D]: Req:6288 UID:0 one.vm.info result SUCCESS, “28…”
Mon Feb 16 10:25:59 2026 [Z0][ReM][D]: Req:8208 UID:0 IP:127.0.0.1 one.vm.info invoked , 28, false
Mon Feb 16 10:25:59 2026 [Z0][ReM][D]: Req:8208 UID:0 one.vm.info result SUCCESS, “28…”
Mon Feb 16 10:25:59 2026 [Z0][ReM][D]: Req:9952 UID:0 IP:127.0.0.1 one.template.clone invoked , 39, “WINSRVOPN2-saved”, false
Mon Feb 16 10:26:00 2026 [Z0][ReM][D]: Req:9952 UID:0 one.template.clone result SUCCESS, 79
Mon Feb 16 10:26:01 2026 [Z0][PLM][I]: Starting Plan Manager timer action…
Mon Feb 16 10:26:01 2026 [Z0][PLM][I]: Found 0 active plans
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:1408 UID:0 IP:127.0.0.1 one.vm.info invoked , 28, false
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:1408 UID:0 one.vm.info result SUCCESS, “28…”
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:7280 UID:0 IP:127.0.0.1 one.vm.disksaveas invoked , 28, 0, “WINSRVOPN2-saved-dis…”, “”, -1
Mon Feb 16 10:26:02 2026 [Z0][ImM][I]: Creating disk at one-ceph, size 51200MB (format: save_as)
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:7280 UID:0 one.vm.disksaveas result SUCCESS, 183
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:8928 UID:0 IP:127.0.0.1 one.template.update invoked , 79, "DESCRIPTION = “App t…”, 1
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:8928 UID:0 one.template.update result SUCCESS, 79
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:6768 UID:0 IP:127.0.0.1 one.marketpool.info invoked
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:6768 UID:0 one.marketpool.info result SUCCESS, “<MARKETPLACE_POOL><M…”
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:1872 UID:0 IP:127.0.0.1 one.template.info invoked , 79, false, false
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:1872 UID:0 one.template.info result SUCCESS, “79</…”
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:4416 UID:0 IP:127.0.0.1 one.imagepool.info invoked , -1, -1, -1
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:4416 UID:0 one.imagepool.info result SUCCESS, “<IMAGE_POOL><…”
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:8672 UID:0 IP:127.0.0.1 one.image.info invoked , 183, false
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:8672 UID:0 one.image.info result SUCCESS, “183<…”
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:7520 UID:0 IP:127.0.0.1 one.hostpool.info invoked
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:7520 UID:0 one.hostpool.info result SUCCESS, “<HOST_POOL><ID…”
Mon Feb 16 10:26:02 2026 [Z0][ImM][D]: Monitoring datastore one-ceph (112)
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:7728 UID:0 IP:127.0.0.1 one.vm.info invoked , 28, false
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:7728 UID:0 one.vm.info result SUCCESS, “28…”
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:3664 UID:0 IP:127.0.0.1 one.hostpool.info invoked
Mon Feb 16 10:26:02 2026 [Z0][ReM][D]: Req:3664 UID:0 one.hostpool.info result SUCCESS, “<HOST_POOL><ID…”
Mon Feb 16 10:26:04 2026 [Z0][ImM][D]: Datastore one-ceph (112) successfully monitored.
Mon Feb 16 10:26:04 2026 [Z0][ReM][D]: Req:9824 UID:0 IP:127.0.0.1 one.image.info invoked , 183, false
Mon Feb 16 10:26:04 2026 [Z0][ReM][D]: Req:9824 UID:0 one.image.info result SUCCESS, “183<…”
Mon Feb 16 10:26:05 2026 [Z0][InM][D]: Host vx-node04 (3) successfully monitored.
Mon Feb 16 10:26:16 2026 [Z0][PLM][I]: Starting Plan Manager timer action…
Mon Feb 16 10:26:16 2026 [Z0][PLM][I]: Found 0 active plans
Mon Feb 16 10:26:20 2026 [Z0][InM][D]: Host vx-node03 (2) successfully monitored.
Mon Feb 16 10:26:24 2026 [Z0][ReM][D]: Req:2944 UID:0 IP:127.0.0.1 one.image.info invoked , 183, false
Mon Feb 16 10:26:24 2026 [Z0][ReM][D]: Req:2944 UID:0 one.image.info result SUCCESS, “183<…”
Mon Feb 16 10:26:31 2026 [Z0][PLM][I]: Starting Plan Manager timer action…
Mon Feb 16 10:26:31 2026 [Z0][PLM][I]: Found 0 active plans
Mon Feb 16 10:26:46 2026 [Z0][PLM][I]: Starting Plan Manager timer action…
Mon Feb 16 10:26:46 2026 [Z0][PLM][I]: Found 0 active plans
Mon Feb 16 10:26:46 2026 [Z0][TrM][D]: Message received: TRANSFER SUCCESS 28 -

Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:1632 UID:0 IP:127.0.0.1 one.image.info invoked , 183, false
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:1632 UID:0 one.image.info result SUCCESS, “183<…”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:7376 UID:0 IP:127.0.0.1 one.marketapp.allocate invoked , " DISK…", 103
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:7376 UID:0 one.marketapp.allocate result SUCCESS, 160
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:112 UID:2 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, 0, -200, -1, “”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:656 UID:2 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, 200, -200, -1, “”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:9376 UID:2 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, 600, -200, -1, “”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:112 UID:2 one.vmpool.infoextended result SUCCESS, “<VM_POOL>31<…”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:656 UID:2 one.vmpool.infoextended result SUCCESS, “<VM_POOL></VM_POOL>”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:9872 UID:2 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, 400, -200, -1, “”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:9376 UID:2 one.vmpool.infoextended result SUCCESS, “<VM_POOL></VM_POOL>”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:2288 UID:2 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, 800, -200, -1, “”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:9872 UID:2 one.vmpool.infoextended result SUCCESS, “<VM_POOL></VM_POOL>”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:2288 UID:2 one.vmpool.infoextended result SUCCESS, “<VM_POOL></VM_POOL>”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:7136 UID:2 IP:127.0.0.1 one.zonepool.info invoked
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:7136 UID:2 one.zonepool.info result SUCCESS, “<ZONE_POOL><ID…”
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:6208 UID:2 IP:127.0.0.1 one.system.version invoked
Mon Feb 16 10:26:46 2026 [Z0][ReM][D]: Req:6208 UID:2 one.system.version result SUCCESS, “7.0.1”
Mon Feb 16 10:26:47 2026 [Z0][AuM][D]: _authenticate: -
Mon Feb 16 10:26:47 2026 [Z0][ReM][D]: Req:2144 UID:1 IP:127.0.0.1 one.documentpool.info invoked , -2, -1, -1, 100
Mon Feb 16 10:26:47 2026 [Z0][ReM][D]: Req:2144 UID:1 one.documentpool.info result SUCCESS, “<DOCUMENT_POOL></DOC…”
Mon Feb 16 10:26:47 2026 [Z0][ReM][D]: Req:6080 UID:1 IP:127.0.0.1 one.vmpool.monitoring invoked , -2, 0
Mon Feb 16 10:26:47 2026 [Z0][ReM][D]: Req:6080 UID:1 one.vmpool.monitoring result SUCCESS, “<MONITORING_DATA><MO…”
Mon Feb 16 10:26:47 2026 [Z0][ReM][D]: Req:6544 UID:1 IP:127.0.0.1 one.vmpool.infoextended invoked , -2, -1, -1, -1, “”
Mon Feb 16 10:26:47 2026 [Z0][ReM][D]: Req:6544 UID:1 one.vmpool.infoextended result SUCCESS, “<VM_POOL>31<…”

12

I have now setup a market place based om http storage also. The same issue is present in this market place also. Images are stored and alsmost works. I get an error when I try to import the image saying:

[one.image.allocate] parse error: syntax error, unexpected variable, expecting equal or equal empty at line 825451059, columns 26:30

When it comes to VM’s and templates the stored app does not show up in the physical storage on the http server, but shows in the OpenNebula frontend gui for the market place. The app shows as being 0 kB, so I guess it is only the content in the database I see.
The images show up with the actual size, and is saved to the http storage or the S3 storage depending on what was chosen.

I have tried to make an app from a small image that I first imported from the OpenNebula Marked place (TTYlinux KVM) and booted and logged into. I then did a shutdown on the VM and tried to make an app to the http market place. The problem does not seem to be related to size of the image, and since the problems are identical on both S3 and http i guess we can expect it to be related to something else than the storage technology of S3 or http.

Do you have any thoughts as to what it can be?