Below are the logs. I can’t get any more detailed information by changing the logging level.

kibana_1 | {"type":"log","@timestamp":"2020-04-27T14:55:04Z","tags":["status","plugin:searchguard@6.8.6-19.0","info"],"pid":1,"state":"yellow","message":"Status changed from yellow to yellow - 'searchguard.cookie.secure' is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to 'true'","prevState":"yellow","prevMsg":"Default cookie password detected, please set a password in kibana.yml by setting 'searchguard.cookie.password' (min. 32 characters)."}
kibana_1 | {"type":"log","@timestamp":"2020-04-27T14:55:04Z","tags":["error","searchguard"],"pid":1,"message":"An error occurred while enabling session management: Error: Failed when trying to obtain the endpoints from your IdP"}
kibana_1 | {"type":"log","@timestamp":"2020-04-27T14:55:04Z","tags":["status","plugin:searchguard@6.8.6-19.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - An error occurred during initialisation, please check the logs.","prevState":"yellow","prevMsg":"'searchguard.cookie.secure' is set to false, cookies are transmitted over unsecure HTTP connection. Consider using HTTPS and set this key to 'true'"}

I want to make sure my configuration is correct before digging into potentially deeper issues. Am I missing something?

I see you use /path/to/cacert.pem file as the root certificate when you call the Keycloak connect URI via curl. But in the Kibana and Elasticsearch configuration you use a different file - /path/to/ca.pem. Do these files contain the same root certificate? If not, try to use /path/to/ca.pem for the curl test.

There’s an HAProxy in front of Keycloak and every time I try to connect to the IdP endpoint, the HAProxy logs an SSL handshake failure. Which makes me believe my OpenID SSL configuration for Search Guard is off.

I was able to get past the Kibana/IdP error by adding options in the Kibana configuration file for client certificate and key. So the services are coming up okay.

Running into a similar issue now inside the Elasticsearch service. Whenever I try to access the Kibana dashboard after authenticating through Keycloak, the Elasticsearch container errors out when trying to reach the IdP OpenID configuration endpoint. Here’s the logs:

Thanks for the link about adding a root certificate to the list of trusted certificates!

I think my problem throughout all of this is that the IdP is expected a two-way SSL authentication (server and client). I had to add the options to do that in the Kibana plugin. Now I’m having issues with the Elasticsearch service. It accepts the certificate that the server sends it, but the server then requests for the client to authenticate and the client claims it has no matching certificates.

Is there not a native way in the Search Guard configuration to allow for two-way SSL authentication? It seems like there are options for it (which I included in my sg_config.yml), so I’m confused as to why it isn’t catching on.

The client (SG) doesn’t find a certificate that was signed by any of the signers mentioned in CertificateRequest message of server (Keycloak) https://stackoverflow.com/a/39096286/2393924. And yes, you have it because of the mutual TLS authentication. Can you send the entire Elasticsearch log?

How did you create the certificates? Describe the whole process. Are you sure you have the same root CA for all certificates (both SG and Keycloak)? You can use sgtlsdiag to verify it.

To make the mutual TLS handshake working you need:

Create a certificate for CA

Create certificate A and sign it by CA certificate

Create certificate B and sign it by CA certificate

Then A and B can be authenticated mutually

And you still can curl with the certificates you specified in sg_config, right?

I didn’t personally create the client certificates so was unsure of the exact creation process. However, using OpenSSL verifies that they are signed by the root CA.

I was able to resolve all of the handshake issues in the Elasticsearch service as well. I had to add the root CA to the Java truststore and then create a keystore for the client certificate/key and specify it in the Java options.

Thanks for your help navigating TLS configurations. The links you forwarded were helpful.