I've designed and/or built Oracle Fusion Middleware 11g environments for a dozen or so customers now and one thing that is always a bit tricky is setting up SSL. My notes are, however, improving from experience so I thought I'd publish them through my blog and probably as a user group presentation at some point (along with more advanced configurations).

The Motivation for SSL

Before we get too far we need to understand the reason for configuring SSL, aside from just wanting to present an HTTPS front-end to a web application (which you may or may not choose to do in WebLogic itself). Firstly, I hope you agree that:

the administration channel is a good thing,

production mode is a good thing,

managing the system using WLST is a good thing, and,

using the java Node Manager to run both managed servers and administration servers is a good thing.

If you don't agree with any of the above have a surf and then come back later!

The problem with all this "good stuff" is that, if you use the demonstration certificates created during installation, it either won't work, or WebLogic will constantly complain so you'll find your logs fill up very quickly (one recent customer had GB-worth of log files because of this!). Also you'll have an Administration Console that, apart throwing up an SSL warning in the browser, has a certificate that says "FOR TESTING ONLY" which doesn't look very professional on a production middleware installation!

Please note: it is sensible for WebLogic to warn us that using demo certificates is a bad idea in production mode - we need to be creating new certificates otherwise we are vulnerable to all sorts of man-in-the-middle and denial of service attacks.

The challenges in setting up SSL for the new WebLogic administrator are:

the documentation is scattered and suffers from the usual WebLogic "10 different ways to do the same thing" characteristic,

the default installer creates (sort of) self-signed certificates based on the hostname which is probably not what you want, especially for the admin server in a highly available rig,

blog posts tend to be inaccurate and almost invariably end up telling you to turn off host name validation,

the security configuration has to be carried out consistently in several places to work properly.

Security is set up very differently in WebLogic as to how it
used to be with iAS (or at least the installer doesn't help you out as much). There are two key pieces of information every WebLogic
Server component (i.e. administration server, managed server, WLST client, node
manager, etc) needs:

something to uniquely and securely identify
itself,

knowledge of whom to trust.

These are achieved by means of certificates and certificate
authorities: the server's identity is held in an "Identity Keystore"
and the public keys of those to trust is held in a "Trust Keystore".
These standard java key stores, as their name suggests, hold public and/or private keys, and can be created by various tools.
Note you could have a combined keystorecontaining both identity and trust certificates but, for security reasons, the
identity is stored separately and only used by the host it applies to, whereas
the trust keystore can be shared widely as it is not confidential. The various WebLogic components can
each use the same certificates, as shown in the diagram below:

By default the WebLogic installer creates two keystores, DemoIdentity.jks and DemoTrust.jks. Oracle recommends that these should be used for development purposes only. In particular, if your domain is in production mode and you are using DemoIndentity & Trust, WebLogic will continuously log warnings which, apart from making log files grow large, can hide more important warnings and errors.

The DemoIdentity file is generated during the initial product installation (not domain creation) and contains, depending on the platform, a certificate for either the <hostname> or the <fully qualified hostname>. This is locally signed by CertGenCAB (i.e. using CertGen). There is an entry for CertGenCAB, along with the other Java VM current certificate authorities (CA), in DemoTrust.jks.

Certificates and Trust

When you want to create your own certificates you can either self-sign them in the usual way (e.g. with the JDK's keytool utility), and add each host as a valid CA to the Trust keystore, or create them with your own in-house certification authority.

Note: none of my (typically mid-sized) customers currently buys external certificates for securing WebLogic; there probably isn't a need for them for internal node-to-node communication in a tightly controlled production environment. Your security requirements may vary though, especially if you operate with financial data and, for example, are PCI compliant.

Once you have created identities and a trust keystore for all your hosts you need to configure the WebLogic components to use them. You also need to decide how to manage them: you could store them in the WebLogic installation, alongside the domain configuration, or at the server level, depending on the sophistication of your environments and the degree of isolation you require. So far, I have typically installed them on a domain-by-domain basis, even if physical/virtual servers run multiple domains. This is simply because domains often have different business owners and one owner won't want to risk impact from changes made to another domain (e.g. if a new trading partner is introduced and new certificates need to be incorporated).

WebLogic Server SSL Configuration

Like many aspects of WebLogic you can configure SSL in several different ways. I always do this in the domain via the Admin Console or WLST. There are 4 options, set on a per-WL server basis:

Demo Identity And Demo Trust: this is the default and uses the generated DemoIdentity.jks and DemoTrust.jks,

Custom Identity And Custom Trust: this uses your own identity keystore and own trust keystore,

Custom Identity And Command Line Trust: I haven't used this type before - the manual says "An identity keystore you create and command-line arguments that specify
the location of the trust keystore. Use this option in a production
environment when the administration port is enabled and Managed Servers
are started on the command line instead of by the Node Manager." Given that I always use Node Manager I don't think this option is of any use to me.

I usually choose "Custom Identity And Custom Trust" as I have created my our own self-signed certificates and included all their public keys in my own trust keystore, though sometimes I use an internal CA certificate instead. The approach is very flexible though so there are several other valid ways of organising your keystores.

Node Manager SSL Configuration

The node manager needs to use the server's identity too, along with the environment's trust keystore. This is most easily configured in the nodemanager.properties file, e.g.

KeyStores=CustomIdentityAndCustomTrust

CustomIdentityAlias=*host*

CustomIdentityKeyStoreFileName=<config>/certs/*host*_identity.jks

CustomIdentityKeyStorePassPhrase=*****

CustomIdentityKeyStoreType=JKS

CustomIdentityPrivateKeyPassPhrase=*****

CustomTrustKeyStoreFileName=<config>/certs/*domain*_trust.jks

(The items with asterisks are replaced by environment specific values)

WLST SSL Configuration

This is the trickiest component to configure since the Oracle documentation was incomplete at the time of writing (or I just couldn't find the right page!). When WLST connects to a secure session, i.e. a WebLogic administration server or a Node Manager, it needs to know if it can trust the server, i.e. it too needs a trust keystore. This is configured in one of two ways (though there are probably more):

by including the trusted CA or self-signed server certificate in the Java Standard keystore (jre/lib/security/cacerts for the JVM used to run WLST),

by pointing the WLST client at a pre-defined Trust keystore.

If you already have a domain-wide trust keystore you could use the second approach. To do this for the wlst.sh script, typically used to run the WLST client, you need to set the following environment variables (e.g. in your profile):

Summary

Hopefully this post provides some helpful information as to how to go about configuring secure communications in a production WebLogic Server environment. As I mentioned, there are several ways to achieve similar results and so if you feel I've missed anything important I'd welcome any comments!

the JAVA_VM environment variable that controls the -client or -server flag for the JVM. With development mode this is -client, with production mode this is -server. This can be found in $MW_HOME/wlserver_10.3/common/bin/commEnv.sh. Even for development boxes I change this setting to -server for better GC performance.

The auto deploy feature only works in development mode. The limitations of auto deploy are such that it rules out use for production anyway.

Are you aware of more differences between production mode and development mode that I should know off?

Interesting question Jacco - I think there are several benefits for live systems from being in production mode. For me putting the domain configuration/config.xml under version control/locking is a very important plus point (especially for teams with several middleware administrators), and something we didn't have in iAS. Apart from the demo SSL warnings I mentioned above I think there are also other features including log rotation etc. It would be worth researching all the differences sometime... a blog post for another day perhaps!

Note: there are a few recent posts on the web comparing production with development mode, such as this one by Zeeshan Baig.

Curiously none of the ones I've seen seem to reference the Oracle documentation. There isn't a definitive place in the official docs as far as I can tell, but you might like to read snippets about production mode in: