Step 1. First I’ve created my cert and key using my own testing CA (though it’s best to use Let’s encrypt if you have your servers on Internet as it’s free).
Step 2. I’ve imported my testing certificate authority file in my browser as a valid certificate issuer, so I can trust any cert signed by this CA for all purposes. I’m using Google Chrome.
Step 3. I’ve set the following in /etc/one/sunstone-server.conf as you’ve already done:
:vnc_proxy_support_wss: only
:vnc_proxy_cert: /etc/one/certs/one-cert.crt
:vnc_proxy_key: /etc/one/certs/one-cert.key
Step 4. I’ve restarted Sunstone.
Step 5. I’ve tried to open VNC connections against my vcenter’s VM. The VNC connection shows it’s encrypted, so far so good.
Step 6. Then I’ve installed nginx and configure it with the same configuration as you’ve proposed.
Step 7. I’ve tried to open Sunstone using https and open a VNC connection and… it works.
I’d check Step 2 as I bet it could be a certificate trust issue.
Are you using a self-signed certificate? Recent browsers are really picky about self-signed that’s why I have used my own CA (I wrote this a long time ago) and then I’ve imported the CA so my certs are trusted. In the past this worked but I’m almost sure it doesn’t work any more: if you visit https://x.x.x.x:29876 maybe your browser allows you to add an exception. In any case I’d try to use a cert signed by a CA your browser trusts.
In any case, you can check what’s happening in your browser using the development tools (you can open it with Ctrl+Shift+I in most browsers) and look for errors in the Console tab that would tell us more about what’s happening.