Hi Jesus, can you try a workaround while we try to fix this issue?
You need to edit /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/security/java.security
In there, change the priorities to make sun.security.provider.Sun #1 and so on i.e. change:
security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
...
...
security.provider.10=sun.security.smartcardio.SunPCSC
to:
security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
...
...
security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

Using the following providers in java.security:
security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
# the NSS security provider was not enabled for this build; it can be enabled
# if NSS (libnss3) is available on the machine. The nss.cfg file may need
# editing to reflect the location of the NSS installation.
#security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg
I no longer get the CKR_DOMAIN_PARAMS_INVALID:
%% Initialized: [Session-4, SSL_NULL_WITH_NULL_NULL]
%% Negotiating: [Session-4, TLS_RSA_WITH_AES_256_CBC_SHA]
*** ServerHello, TLSv1.2
And when I connect with RestClient I get the actual JSON response I expected:
RestClient.get("https://localhost:8443/candlepin/status")
=> "{\"result\":true,\"version\":\"0.8.26\",\"rulesVersion\":\"4.2\",\"release\":\"1\",\"standalone\":true,\"timeUTC\":\"2013-10-25T18:17:16.391+0000\",\"managerCapabilities\":[\"cores\",\"ram\",\"instance_multiplier\",\"derived_product\",\"cert_v3\"],\"rulesSource\":\"DEFAULT\"}"

Interesting. What is happening here is the following:
1. The client tries to get a crypto provider that supports SSL_NULL_WITH_NULL_NULL.
2. If the NSS provider is not present, no provider responds to this request and the client requests TLS_RSA_WITH_AES_256_CBC_SHA instead.
3. If the NSS provider is present, instead of saying it doesn't support the algorithm it throws an exception which is carried up to the client.
So the NSS provider is giving the wrong response.

(In reply to Andrew John Hughes from comment #16)
> I think https://bugzilla.redhat.com/show_bug.cgi?id=1022950 is related, if
> not the same issue.
>
> This is the difference on Jesus' machine when the PKCS11 NSS provider is
> enabled and when it isn't:
>
>[snip]
>
> So, with it enabled, the SSL connection is trying to use
> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 and failing because NSS doesn't
> actually support it.
>
> I didn't get the ECC algorithms on my local RHEL machine (latest 6.4). Has
> there been a change in NSS?
yes, NSS in 6.5 introduced support for TLSv1.2 and ECC.
But the support is not complete.
In case of TLSv1.2 two features are not supported:
* GCM
* SHA384 as MAC
In case of ECC, only three curves are supported: nistp256, nistp384, nistp521.
so TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 won't work

Yeah, that's what I just worked out, tracing through the code :)
Can you tell us which of the additions in the list above should work, if any? It looks like we're going to need to find where this list is created and patch out the ones our NSS doesn't support.
We came across an issue with the curves before. Oracle's upstream test cases for this provider expect additional curves. See:
http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=469

Created attachment 821065[details]
Specify only the three supported curves
The attached patch should resolve the issue.
Looking at the code, I don't think NSS is being used for its TLS 1.2 implementation at all:
"* TLS 1.2 uses a different hash algorithm than 1.0/1.1 for the PRF calculations. As of 2010, there is no PKCS11-level support for TLS 1.2 PRF calculations, and no known OS's have an internal variant we could use. Therefore for TLS 1.2, we are updating JSSE to request different provider algorithms (e.g. "SunTls12Prf"), and currently only SunJCE has these TLS 1.2 algorithms. If we reused the names such as "SunTlsPrf", the PKCS11 providers would need be updated to fail correctly when presented with the wrong version number (via Provider.Service.supportsParameters()), and we would also need to add the appropriate supportsParamters() checks into KeyGenerators (not currently there).In the future, if PKCS11 support is added, we will restructure this."
which is where the mismatch occurs. Instead of probing providers for which curves they support, the OpenJDK TLS code hardcodes a set list and uses that in its implementation of the Hello ECC curves extension.
The fix just cuts that list down to the three NSS curves. I don't know why this wasn't designed so as to use an API to ask providers for the curves they support. We obviously can't add it to 7 now.

I've managed to reproduce it on there too, but I can't see any obvious differences. It's mystifying.
Are both ends of the transmission Java clients? I did notice that lsof shows postgres, sendmail, cups and ssh to have references to a deleted version of NSS.
If anyone has a simpler reproducer, it would be very welcome :)
As a workaround, there is a property we could change the default of and thus turn off ECC. Thoughts?

(In reply to Andrew John Hughes from comment #40)
> Deepak, there was still a bug to begin with. It just means our patch did
> fix it.
Ah, right, sorry.
Jesus, if this works for you then, can you please close it as CURRENTRELEASE?

Could you guys please send the bugfix, or a link to this bugreport to the openssl guys? Your bugfix is quite worthless to the rest of the world if you don't report this bug upstream and keeping the fix to yourselves. I had to disable nss via. the java.security file in order to use mcabber with my jabber server, because I had this exact exception on a debian box with openfire running on top of it.

(In reply to rhbugzilla.5.nebuchadnezar from comment #46)
> Could you guys please send the bugfix, or a link to this bugreport to the
> openssl guys? Your bugfix is quite worthless to the rest of the world if you
> don't report this bug upstream and keeping the fix to yourselves. I had to
> disable nss via. the java.security file in order to use mcabber with my
> jabber server, because I had this exact exception on a debian box with
> openfire running on top of it.
The fix is not applicable to upstream. Upstream does does not remove support for certain Elliptic Curves. That is something specific to RHEL packages.

(In reply to rhbugzilla.5.nebuchadnezar from comment #46)
> Could you guys please send the bugfix, or a link to this bugreport to the
> openssl guys? Your bugfix is quite worthless to the rest of the world if you
> don't report this bug upstream and keeping the fix to yourselves. I had to
> disable nss via. the java.security file in order to use mcabber with my
> jabber server, because I had this exact exception on a debian box with
> openfire running on top of it.
This bug is not about OpenSSL. You need to comment on their bug report.

It is clearly about openssl, or nss. At least the same reason why this bug report was opened. The workaround proposed here, worked for openfire too. It is something about the handling of tls 1.2. Openfire can't do anything about it, if the error is somewhere below the jvm, isn't it. Unfortunately I'm not well versed with nss/openssl, so I don't really know to whom I should report this, but I figured, due to the fact that the workaround provided here worked too, I can only conclude that the source of the problem is the same; or at least something very similar. Sorry, if I sounded angry before, but it looked to me like, .. nah don't bother with reporting this upstream, we just patch it ourselves, but whatever; I'm a bit helpless here to whom this should be reported. At least, the workaround helped.

(In reply to rhbugzilla.5.nebuchadnezar from comment #50)
> It is clearly about openssl, or nss. At least the same reason why this bug
> report was opened. The workaround proposed here, worked for openfire too. It
> is something about the handling of tls 1.2. Openfire can't do anything about
> it, if the error is somewhere below the jvm, isn't it. Unfortunately I'm not
> well versed with nss/openssl, so I don't really know to whom I should report
> this, but I figured, due to the fact that the workaround provided here
> worked too, I can only conclude that the source of the problem is the same;
> or at least something very similar. Sorry, if I sounded angry before, but it
> looked to me like, .. nah don't bother with reporting this upstream, we just
> patch it ourselves, but whatever; I'm a bit helpless here to whom this
> should be reported. At least, the workaround helped.
This is a bug related to java-1.7.0-openjdk, so no it's not "clearly about openssl or nss".

Re-opening this as we appear to have regressed on this issue with the java-1.8.0-openjdk package. We now ship the SunEC provider which also uses NSS (see bug 1208307) and the same three curves, but the patch from comment #21 was never forward-ported to the OpenJDK 8 packages. It was already in the OpenJDK 7 packages when SunEC support was added, so it was missed as not being one of the changes we made then.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2017-0571.html