Opennebula 6.6 marketplace with minio storage backed can be created but image migrations fail

Hi, I have minio object storage on a synology nas and it works fine with terraform when I use the “refaktory/minio” terraform provider alson with the opennebula terraform provider.

terraform {
  required_providers {
    opennebula = {
      source  = "opennebula/opennebula"
      version = "~> 1.2.1"
    }
    minio = {
      source  = "refaktory/minio"
      version = "0.1.0"
      source  = "refaktory/minio"
    }
  }
}
#
provider "opennebula" {
  endpoint = var.one_endpoint
  username = var.one_username
  password = var.one_password
}
#
provider "minio" {
########################################################
# The Minio server endpoint.
# NOTE: do NOT add an http:// or https:// prefix!
# Set the `ssl = true/false` setting instead.
########################################################
  endpoint = "192.168.1.11:9000"
  access_key = var.minio_access_key
  secret_key = var.minio_secret_key
  ssl = false
}

I thought I’d use minio to also host a local s3 marketplace, the market place creates ok but when I try to migrate an image from a datastore it fails with the following error. Ive tried leaving the s3 driver parameter blank and by entering s3.

Has anyone seen this issue or have a solution, I’ve tried on Opennebula 6.4 and 6.6 without any luck.

Mon May 15 13:51:48 2023 : Error importing app into marketplace: /usr/lib/ruby/2.7.0/net/http.rb:960:in `initialize': Failed to open TCP connection to one-marketplace.192.168.1.11:9001 (getaddrinfo: Name or service not known) (Seahorse::Client::NetworkingError) 	from /usr/lib/ruby/2.7.0/net/http.rb:960:in `open' 	from /usr/lib/ruby/2.7.0/net/http.rb:960:in `block in connect' 	from /usr/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout' 	from /usr/lib/ruby/2.7.0/timeout.rb:105:in `timeout' 	from /usr/lib/ruby/2.7.0/net/http.rb:958:in `connect' 	from /usr/lib/ruby/2.7.0/net/http.rb:943:in `do_start' 	from /usr/lib/ruby/2.7.0/net/http.rb:938:in `start' 	from /usr/lib/ruby/2.7.0/delegate.rb:83:in `method_missing' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/connection_pool.rb:307:in `start_session' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/connection_pool.rb:100:in `session_for' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:128:in `session' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:76:in `transmit' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:50:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/content_length.rb:24:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/request_callback.rb:85:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/s3_signer.rb:53:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/s3_host_id.rb:17:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/xml/error_handler.rb:10:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/redirects.rb:20:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:355:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:389:in `retry_request' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:377:in `retry_if_possible' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:366:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:389:in `retry_request' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:377:in `retry_if_possible' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:366:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:389:in `retry_request' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:377:in `retry_if_possible' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:366:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/http_checksum.rb:19:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/accelerate.rb:67:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:136:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/bucket_dns.rb:35:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/dualstack.rb:41:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/expect_100_continue.rb:22:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:26:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/arn.rb:62:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/rest/handler.rb:10:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/endpoint.rb:47:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/arn.rb:88:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/dualstack.rb:27:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/plugins/accelerate.rb:56:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/request_callback.rb:71:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/response_target.rb:24:in `call' 	from /usr/share/one/gems-dist/gems/aws-sdk-core-3.130.0/lib/seahorse/client/request.rb:72:in `send_request' 	from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.113.0/lib/aws-sdk-s3/client.rb:6910:in `head_bucket' 	from /var/lib/one/remotes/market/s3/S3.rb:36:in `bucket=' 	from /var/lib/one/remotes/market/s3/import:121:in `<main>'

Hello,

Looking at the error message, this looks suspicious one-marketplace.192.168.1.11, I don’t think this hostname is correct and as you can see DNS resolver is unable to resolve it :slight_smile:. Could you share the definition of your S3 marketplace in xml or OpenNebula format? :thinking:

Hi, Thanks for getting back to me on this issue.

I tried again to migrate a datatstore image to my local marketplace with a minio backend. Looking at the log extract I’ve attached, the problem this time seems to be with maybe with the following

[Z0][MKP][I]: /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Aws::S3::Errors::BadRequest (Aws::S3::Errors::BadRequest)

Do I need to install an awscli on the host?


Log extract
oned-extract.log (75.7 KB)

The image I’m migrating from the datasore was downloaded from the opennebula marketplace.

Kind regards
Dennis

Hi, I’m getting closer to the problem which may be just user error.
I noticed the minio port in the terraform definition was different, after changing the marketplace definition to use the same port, I’m now seeing a message:

The specified bucket does not exist (Aws::S3::Errors::NoSuchBucket)

Fri May 26 16:51:22 2023 [Z0][MKP][I]: from /var/lib/one/remotes/market/s3/S3.rb:36:in `bucket='
Fri May 26 16:51:22 2023 [Z0][MKP][I]: from /var/lib/one/remotes/market/s3/monitor:100:in `<main>'
Fri May 26 16:51:22 2023 [Z0][MKP][E]: Error monitoring marketplace 100: /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': The specified bucket does not exist (Aws::S3::Errors::NoSuchBucket)      from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.117.2/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'    from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.117.2/lib/aws-sdk-s3/plugins/location_constraint.rb:18:in `call'        from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.117.2/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'  from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'    from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'    from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'      from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'        from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/seahorse/client/plugins/request_callback.rb:71:in `call'    from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'        from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/seahorse/client/plugins/response_target.rb:24:in `call'     from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/seahorse/client/request.rb:72:in `send_request'     from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.117.2/lib/aws-sdk-s3/client.rb:1738:in `create_bucket'  from /var/lib/one/remotes/market/s3/S3.rb:38:in `rescue in bucket='     from /var/lib/one/remotes/market/s3/S3.rb:35:in `bucket='       from /var/lib/one/remotes/market/s3/monitor:100:in `<main>' /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Aws::S3::Errors::NotFound (Aws::S3::Errors::NotFound)         from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.117.2/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'    from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.117.2/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'  from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.117.2/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'         from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'    from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'    from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'      from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'        from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/seahorse/client/plugins/request_callback.rb:71:in `call'    from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'        from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/seahorse/client/plugins/response_target.rb:24:in `call'     from /usr/share/one/gems-dist/gems/aws-sdk-core-3.168.3/lib/seahorse/client/request.rb:72:in `send_request'     from /usr/share/one/gems-dist/gems/aws-sdk-s3-1.117.2/lib/aws-sdk-s3/client.rb:6927:in `head_bucket'    from /var/lib/one/remotes/market/s3/S3.rb:36:in `bucket='       from /var/lib/one/remotes/market/s3/monitor:100:in `<main>'
Fri May 26 16:51:22 2023 [Z0][InM][D]: Monitoring datastore default (1)
Fri May 26 16:51:22 2023 [Z0][InM][D]: Monitoring datastore files (2)
Fri May 26 16:51:22 2023 [Z0][ImM][D]: Datastore default (1) successfully monitored.
Fri May 26 16:51:22 2023 [Z0][ImM][D]: Datastore files (2) successfully monitored.

Hi, It seems the initial problem was user error on my part.
Once I removed the bucket from the endpoint address the image migrated without errors

Marketplace tempate

ACCESS_KEY_ID = "minio"
BUCKET = "one-marketplace"
DESCRIPTION = "Minio s3 Marketplace Backend"
ENDPOINT = "http://192.168.1.11:9000"
MARKET_MAD = "s3"
REGION = "none"
SECRET_ACCESS_KEY = "xxxxxxxxxx"

When the image migration to minio completed I tried to download it to my opennebula datastore but it failed with an error 400.

Investigating…

Fri May 26 17:06:16 2023 [Z0][ImM][I]: cp: Copying local image s3://one-marketplace/marketapp-70 to the image repository
Fri May 26 17:06:16 2023 [Z0][ImM][E]: cp: Command "set -e -o pipefail; /var/lib/one/remotes/datastore/fs/../downloader.sh   's3://one-marketplace/marketapp-70' '/var/lib/one//datastores/1/26bbd3ff522fd35c39c647a980a651be'" failed: curl: (22) The requested URL returned error: 400
Fri May 26 17:06:16 2023 [Z0][ImM][I]: curl: (22) The requested URL returned error: 400
Fri May 26 17:06:16 2023 [Z0][ImM][I]: curl: (22) The requested URL returned error: 400
Fri May 26 17:06:16 2023 [Z0][ImM][I]: curl: (22) The requested URL returned error: 400
Fri May 26 17:06:16 2023 [Z0][ImM][I]: Error copying
Fri May 26 17:06:16 2023 [Z0][ImM][I]: Error copying s3://one-marketplace/marketapp-70 to /var/lib/one//datastores/1/26bbd3ff522fd35c39c647a980a651be
Fri May 26 17:06:16 2023 [Z0][ImM][E]: Error copying image in the datastore: INFO: cp: Copying local image s3://one-marketplace/marketapp-70 to the image repository ERROR: cp: Command "set -e -o pipefail; /var/lib/one/remotes/datastore/fs/../downloader.sh   's3://one-marketplace/marketapp-70' '/var/lib/one//datastores/1/26bbd3ff522fd35c39c647a980a651be'" failed: curl: (22) The requested URL returned error: 400 curl: (22) The requested URL returned error: 400 curl: (22) The requested URL returned error: 400 curl: (22) The requested URL returned error: 400 Error copying Error copying s3://one-marketplace/marketapp-70 to /var/lib/one//datastores/1/26bbd3ff522fd35c39c647a980a651be
Fri May 26 17:06:16 2023 [Z0][InM][D]: Monitoring datastore default (1)
Fri May 26 17:06:16 2023 [Z0][ImM][D]: Datastore default (1) successfully monitored.

OK, Annother user error, once I deleted the failed images from the minio marketplace the download worked correctly.

All is good :smile:

1 Like