JSON errors in econe-server opennebula 4.8.0

I am seeing the following error in my econe-server.

From econe-server.error we see the following:

JSON::ParserError - A JSON text must at least contain two octets!:
/usr/lib/ruby/gems/1.8/gems/json-1.4.6/lib/json/common.rb:146:in initialize' /usr/lib/ruby/gems/1.8/gems/json-1.4.6/lib/json/common.rb:146:innew’
/usr/lib/ruby/gems/1.8/gems/json-1.4.6/lib/json/common.rb:146:in parse' /usr/lib/one/ruby/cloud/econe/keypair.rb:57:inget_keypair’
/usr/lib/one/ruby/cloud/econe/keypair.rb:122:in describe_keypairs' /usr/lib/one/ruby/cloud/econe/econe_application.rb:165:indo_http_request’
/usr/lib/one/ruby/cloud/econe/econe-server.rb:135:in POST /' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:incall’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in route' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:ininstance_ev
al’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in route_eval' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:inroute!’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in catch' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:inroute!’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in each' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:inroute!’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in dispatch!' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:incall!’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in instance_ev al' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:ininvoke’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in catch' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:ininvoke’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in call!' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:incall’
/usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in call' /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/showexceptions.rb:24:incall’
/usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in call' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:incall’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in synchroniz e' /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:incall’
/usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in call' /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:incall
/usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in pre_proces s' /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:incatch’
/usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in pre_proces s' /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:inprocess’
/usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in receive_da ta' /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:inru
n_machine’
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in ru n' /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:instart’
/usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in start' /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:inrun’
/usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:946:in `run!’
/usr/lib/one/ruby/cloud/econe/econe-server.rb:148


The only econe-specific code in any of this is the very bottom econe-server.rb and
it is just calling sinatra.

This error is apparently very common in the Ruby on Rails community and I can see a lot
of stuff on the web indicating that this is a fairly common and low level error, and can be caused
by malformed JSON such as a single number.

My question is–what JSON is being passed in this call sequence, econe-server -> sinatra -> lower gems.

Furthermore:
the econe-run-instances call from the command line still works, as does the econe-describe-images call,
but the econe-describe-keypairs call does not work.

-bash-4.1$ econe-describe-keypairs --access-key=/tmp/x509up_u2904 --secret-key=/tmp/x509up_u2904 --url https://fclheadgpvm01.fnal.gov:8444/
econe-describe-keypairs: Could not create X509 certificate from


It appears that my automated system, HTCondor, is failing to launch anything basically because it is
trying to create an ssh keypair for every VM it launches, and failing due to this error.

Any help is appreciated. We do not have resources to upgrade.

One more thing: we run the econe-server behind apache mod_ssl to do the SSL parsing, and
this was recently upgraded as of July 28 to httpd-2.2.15-54.sl6 as shipped by red hat.
It appears that it was at this time that the system, previously working for 2 years, broke.

Steve Timm