Thursday, March 15, 2012

Don’t Teach Bad Habits

A large number of
servers and applications are built with SSL configured incorrectly. If
your SSL is misconfigured, it is as good as no SSL. In reality, it may be
worse than no SSL, as it gives the feeling of safety where none exists.
This is most common when developers and administrators leverage a framework
that they don’t fully understand.

Any time a user sees one of
the following errors or warnings, the server admin has failed. A user, no
matter how technical, should NEVER be forced to answer these questions.
Every time a user is trained to hit Yes, or OK to one of these errors, the user
is being trained to fail.

Figure 1 IE SSL Error

Figure 2 Firefox SSL Error

Figure 3 Opera SSL Error

Figure 4 Chrome SSL Error

Different browsers may have
slightly different text for different failures, but most users will think of
them as the same thing, “The place where I have to do an extra click to get to
my site”.

Who is Protecting Whom

This document only covers
standard, run of the mill SSL, where the server has a certificate and the
client does not. Two way or Mutual TLS is not commonly used and not
covered here. One of the things that confuses people about SSL is that it
is designed to allow the client to protect itself and provides on protection or
authentication of the client for the benefit of the server. The
server operator or application owner is supposed to be giving the client
everything it needs to make sure the connection is secure. Presumably the
user is going to submit some data that he or she does not want disclosed.
If you are running an application where the liability is not on the client,
standard one way SSL is not enough for you. You are assuming a risk that
you have no control over and you need additional controls. This is the
case with most online banking in the US. If the customer loses his
password to an attacker, and the attacker drains the account, in many cases,
the bank is liable. There have been a few cases recently where banks have
been able to successfully put the burden on the customer.

It is safe to assume that if
your application is misconfigured such that the client cannot accurately verify
that the connection is secure, your end user will be able to make a successful
case as to why the risk should be shifted to you, despite any Terms of Service
or other contract that may state otherwise.

Help Them Help Themselves

There are many criteria that
the client can use to determine if it should trust a connection to a
server. While some clients may chose to ignore some of the criteria, it
is wise to do everything right, and avoid any issues.

Subject Common Name

The Subject of the
certificate is a description of the entity that offers it to a client. To
those familiar with LDAP, it may look like and x.500 Distinguished Name (DN),
and it is in fact referred to as a DN. For example:

However, most of the DN is
arbitrary and with the inclusion of E=, for email address, it has no
hierarchical convention like an LDAP DN. While some lament this
inclusion, unless there really is an LDAP to look at, who cares if the data is
actually hierarchical…

There are two exceptions to
the “arbitrary” nature of the subject. First, parties may choose to check
the subject for a specific value. Cisco VPN concentrators, for instance,
can check that a certain OU or L is included.

By far the most important,
for SSL/TLS is the CN, or common name. Most clients check to make sure
that the CN matches the name that the client is attempting to connect to.
This may be the most common configuration error. The reason
for this is that as a web developer, you can’t control what the user enters in
their address bar. Maybe the user will type google.com or maybe they will
type www.google.com, or they might try and access some named service like
maps.google.com. They may even type in an IP address. Often
two or more of the sites may be hosted on the same server at the same IP
address. To keep things nice and consistent, many web developers may
redirect the connection to the pretty named site. This is how google does
it if you enter http://google.com into the address bar:

HTTP/1.1 301 Moved Permanently

Location:
http://www.google.com/

While this works great with
non-SSL sites, this will fail with HTTPS unless you plan correctly. The
reason that this fails is that the SSL negotiation happens BEFORE HTTP data can
be passed. You can’t have an HTTP redirect if the SSL negotiation
fails. This is the reason that many think that you cannot use host header
based virtual sites with SSL. They are wrong, you just have to know what
you are doing and get the right certificate.

We have two different
options, if we want a certificate to work with multiple names.

Wild Cards Certificates

One answer to
the problem of name matching is to just get one certificate that can answer for
all hosts in your domain. RFC 5425 defines a wildcard that can match any
host in a domain, so the subject common name *.google.com would work for
www.google.com, google.com, and mail.google.com. While some clients seem
to allow the wildcard to match domains as well as hosts, the RFC forbids this,
so avoid it. For example, *.google.com should not work for
www2.maps.google.com, as only the leftmost name is the host portion.
Everything to the right is domain, and the * is not supposed to figure into
domain matching. The domain here is maps.google.com.

Subject Alternate Name

The other
option, especially if you are using host headers for sites that do not share
the same domain name, is to use the subject alternate name or SAN. The
SAN is a list of names that can also be checked by a client. One
certificate could then be used for www.google.com and
www.gmail.com. While clients are supposed to be able to parse both
the subject common name and the SAN list, some do this poorly, therefore it is wise
to add the subject common name to the list of SANs as well.

Picking the Right Name

As stated above,
the client will match the name it thinks it is connecting to to the names on
the certificate. In the case of a web browser, this is the name in the
address or URL bar. This means that if your site is behind a load
balance, then the common name must match the DNS name that resolves to the load
balancer. The client has no idea there may be a cluster and load
balancer.

Figure 5 Typical Web Cluster

In Figure 5, the proper
common name for the certificate would be www.google.com.

In some corporate
environments, users may bypass the load balancer in certain
circumstances. In this case, the certificate may need SANs or to use a
wild card.

Time Validity

Certificates contain a Valid
From and Valid To date. The client will check to verify that the date of
the connection attempt is between the two dates. Don’t let your
certificates expire.

Certificate Revocation List Distribution Points

Generally it is thought that
only PKI operators need to be concerned with CRLs and CDPs (CRL Distribution
Points). This is not the case. A CRL is a list of certificates that
the issuing CA has determined are not to be trusted. Not all certificates
list a CRL, obviously if one does not exist, you don’t care about them.
If the client determines that the certificate it is inspecting is listed on the
CRL, it should reject the connection. The reason you should be concerned
with CDPs, as an application owner, is that your clients will attempt to read
them. Different clients react differently when they fail to access the
CRL. Some clients will fail open and allow the connection and some will
deny it. If you know your clients are behind highly restrictive
firewalls, you may need to make sure that they have access not just to your
site, but also to the CDP. This is a common place to fail in a B2B or corporate
environment.

Basic Constraints, Key Usage, and Enhanced Key Usage

A good PKI book or web site
would never lump these all together, but this is neither of those. Each
of these is a field that can describe how the certificate or the associated
private key can be used. Some clients may choose to make decisions based
on these. IE for instance will allow and SSL connection to a server with
no EKUs listed, but if the EKU section is used and the server authentication
usages is not listed, it will deny the connection.

If the Basic Constrains
listed in the certificate of the CA that signed your SSL certificate doesn’t
list it as a CA, the client should abort the connection. This keeps
someone from using a certificate and private key as if it were a CA. In
the past, vendors forgot to check this, and anyone who bought a legitimate SSL
certificate could use it to sign other certificates that were otherwise cryptographically
sound.

Key Usage should list Digital
Signature and Key Encipherment; if Key Usage is defined without Key
Encipherment the client should abort the connection. If Key Usage is not
listed, all uses are allowed.

The Chain

The server’s SSL certificate
does not exist in a vacuum. It is at the bottom of a PKI hierarchy.
Without the context of the PKI the SSL certificate is useless. The beauty
of using PKI to secure (authenticate, negotiate keys and provide integrity
checks) an SSL connection is that this can be done with a relatively small
number of trusted third parties. One of the big reasons to leverage PKI
is to avoid the risks and overhead of increasing the number of parties you, or
your clients, have to trust.

A quick reminder about key
management; there is NO WAY to exchange key material securely, over an
un-trusted network, without an out of band trust being established.
Diffie–Hellman does not provide authentication, so there is no way to know
which party you are negotiating with. RSA key pairs are useless as well,
as the public key must be distributed securely in order for it to be
trusted. This is why OSs, Java, web browsers, and such have keystores as
part of their distributions. Ponder this; did the browser or java package
you downloaded come over via SSL or could an attacker have poisoned the
keystore in the package?

The root CA is the most
important element in a PKI. The trust it conveys is significant and makes
it easy to extend trust to a huge number of parties. In order to extend
this trust, subordinate CAs can propagate that trust. It is common for a
well built PKI to be two or three tiers.

Figure 6 Typical 3 Tier PKI

To increase security and
usability, one should never ask a client to trust a subordinate CA or SSL
certificate explicitly. Trust should always extend implicitly from the
Root CA. Any time you ask an end user to make the decision to trust
a certificate, YOU HAVE FAILED.

It’s alright to configure dev
and test wrong, right? NO!! None of your testing is valid if you don’t
configure your dev and test correctly. Additionally, dev and test are
your chance to make sure you fully understand how to configure your systems
correctly. In the parlance of sports, “You play how you practice”.
So don’t practice failing. It is well worth the extra 10 minutes to
get this right.

This hierarchy is built so
that a security failure at a lower level can be eliminated by simply revoking a
certificate and effectively pruning an entire branch of the PKI. Figure 6
shows the chain belonging to webmail.T-Mobile.com, whereas Figure 7 shows the
entire PKI. If the A1 CA, in Figure 7, were compromised, the root could
simply revoke the A1’s certificate and A1 and everything below it would no
longer be trusted. This has no negative effect on L1C or L1B or anything
below them.

Figure 7 Big PKI

This type of design reduces
the overall damage that can be done by a successful attacker. Usually the
root is offline in a vault, so it can’t be hacked.

If the client only trusts the
Root, but the SSL certificate is signed by an issuing CA, like L1C in Figure 7,
how does the client bridge the gap and get the trust extended to it? The
chain MUST be sent by the server as part of the SSL negotiation.

This is another area of
common misconfiguration.

Figure 8 Wireshark sees 3 certificates

We see in Figure 8, a
properly configured server sending the SSL certificate, the Issuer CA
certificate, and the intermediate CA certificate. If we look at the
Intermediate CA certificate, it will list its issuer as the Root, Entrust,
which the clients hopefully trust. This is enough data for the client to
“build” the chain and cryptographically verify that no monkey business has
occurred.

As part of building the
chain, the client will perform all the checks above, time validity, not
revoked, etc, on each certificate in the chain. If any check fails, the
client should abort the connection. If the chain successfully builds to
the root, then the client can allow the session and security is assured.

Forcing SSL

Another common mistake is not
forcing SSL and simply enabling it. Some site owners and designers
wrongly assume that the only time they need SSL is during authentication.
If the user’s authenticated session has any value at all, then the entire
session must be SSL protected. HTTP sessions are trivial for an attacker
to hijack. Additionally, configurations on a site that mix HTTP and HTTPS
are prone to human error induced problems. Rather than trusting
that links include HTTPS or that only certain pages are set to require SSL,
instead set the entire site to require SSL. All the arguments about
the extra processor load of SSL haven’t been relevant since 2003 or so.

Why All the Failures?

It would be nice if there was
an easy to identify, universally hated, body like congress that could be blamed
for SSL failures. Vendors meet this requirement well. SSL
certificate vendors point their fingers at the application and OS vendors and
the application and OS vendors blame the SSL certificate vendors and the open
source community. The overall problem is that there is a knowledge gap
between the vendors and the admins who setup the servers and
applications. Very few admins know what the chain is, how to send it, or
how to verify that it is being sent.

Verifying the Chain is Sent

Getting the chain sent, in
the first place varies by application and platform, so it is best covered
later. Checking that the chain is sent is easy. Simply install
openSSL and run this command:

openssl s_client -connect
host:port

This returns a very handy
output full of useful information, including:

0 is the SSL certificate and
the numbers go up. The subject and issuer DNs are listed to make an
easy visual verification of the chain. You can see that this application
does not send the root. If you wish to verify the entire chain
cryptographically, you can use the –CAfile argument and reference your Root CA
certificate, in PEM format.

This would include the following
line, assuming the chain checked out:

Verify return code: 0 (ok)

Sending the Chain

This section is the real
reason this document was written. There is very little guidance on the
web telling admins to send the chain or how to send the chain, and often vendor
documents brush over this and or don’t even cover it.

Windows

This section may be a bit of
a generalization, as many applications can run on windows that use their own
SSL code. If the application uses the OS’s SSL, CAPI or CNG, then the OS
will attempt to build and send chain based on what it finds in the machine
store, or the store of the account running the server or application.
This will apply to all MS server applications, such as IIS, Exchange, Active
Directory, etc. Unless the developer overrides the default behaviors,
this will apply to .NET apps as well. To make sure that the OS can find
the certificates in the chain, make sure that they are all in the same “My” or
“Personal” store that holds the SSL certificate. Windows can find the
certificates in the other stores as well, but it is nice to know there they are
and be clear which certificates are which.

Figure 9 Windows Machine Stores

Java

No matter which OS your JVM
is running on, or the version of JVM you are running, there is one keystore
that is part of the build. The cacerts file found under lib/security
holds the list of trusted CAs. Java keystore files use passwords.
The default password for your cacerts file is ‘changeit’. This is the
list of CAs that you JVM will trust by default. If a client side Java
application will use SSL, this is the store that the trust will be based on,
unless the developer made a decision to use a separate store. The Java
keystore format is proprietary, and is sometimes referred to as JKS. Sun
provides the executable ‘keytool’ to create, edit, and inspect keystore
files. Newer versions of keytool and Java seem to make it harder to mess
up creating the keystore, but in older versions mistakes are very common and
allowed by keytool. The JKS file format is a password protected wrapper
that should contain all the CA certificates in your chain. Inside that
wrapper is another password protected wrapper that holds the private key
corresponding to your SSL certificate. For best results, when
creating your keystore file, import the CA certificates starting from the root
and working down, installing the signed SSL certificate last.

Create the Keystore and Key Pair

The following attributes must be specified. Remember
what the password is since it will be needed when setting up server and
importing signed certificates.

Password: password

First and Last Name: (must be the FQDN of the server or
the F5)

Organizational Unit: (ex. used Operations in particular
example)

Organization: (ex. used Company USA.)

City or Locality: (ex. used Seattle)

State or Province: (ex. used Washington)

Two Letter Country Code: (ex. used US for United States)

Once attributes have been inputted, user is asked to verify
information.

Select "y" and hit

The user is then asked for key password for < sslcert
>, use same password type above and hit

Keystore has now been created.

Create the Certificate Signing Request

From the command line enter

keytool -certreq -alias sslcert -keystore keystore.jks
-file ssl.req

Get the file ssl.req to the CA admin or vendor for signing.

Once the Certificate has been
signed, the admin or vendor will return the certificate file. In the
example, this is certnew.cer.

Import the CA Hierarchy Certificates

Before importing the signed
certificate into your keystore you will need to establish the trust by
importing the CA Certificates. Always start with the root CA and work
your way down. This example will show a three tier PKI.

openSSL

openSSL is a fantastic tool
and set of libraries that is leveraged by many applications. OpenSSL’s
native format for dealing with certificates, certificate signing request, and
keys is the PEM format. OpenSSL is the most versatile tool for dealing
with certificates and keys, as it can convert to and from almost any format,
except the Java keystore.

A typical PEM formatted
certificate looks like this:

-----BEGIN CERTIFICATE-----

MIIFuzCCBKOgAwIBAgIETBsbdDANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMC

VVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0

…

FPagpeXsGSr0ipNqRy6m1T2mAO3Ok2B4hK6pBnhtf2n7oY1LB/e1rSIHTivJ9eaN

rOCwFjRf+bcELIi2manBx9zI0Qy8JES3vzXBG3xMkJ4Vv3QI93Uk4qn3jodtCPNj

DZZTUPbnHqRshXE6zSB7S6BvRbsmaYps9JF1+j5e3Q==

-----END CERTIFICATE-----

The chain file is just the
individual CA certificates in one file, end to end, starting from the issuing
CA, moving up to the root, like so:

-----BEGIN CERTIFICATE-----

MIIFuzCCBKOgAwIBAgIETBsbdDANBgkqhkiG9w0BAQUFADCBsTELMAkGA1UEBhMC

VVMxFjAUBgNVBAoTDUVudHJ1c3QsIEluYy4xOTA3BgNVBAsTMHd3dy5lbnRydXN0

…

rOCwFjRf+bcELIi2manBx9zI0Qy8JES3vzXBG3xMkJ4Vv3QI93Uk4qn3jodtCPNj

DZZTUPbnHqRshXE6zSB7S6BvRbsmaYps9JF1+j5e3Q==

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

MIIE8jCCA9qgAwIBAgIEOGPp/DANBgkqhkiG9w0BAQUFADCBtDEUMBIGA1UEChML

RW50cnVzdC5uZXQxQDA+BgNVBAsUN3d3dy5lbnRydXN0Lm5ldC9DUFNfMjA0OCBp

…

PFS+/i/qaZ0cHimbltjI/lGQ8SSmkAaz8Cmi/3gud1xFIdlEADHzvjJP9QoyDfz8

uhZ2VrLWSJLyi6Y+t6xcaeoLP2ZFuQ==

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

MIIEnzCCBAigAwIBAgIERp6RGjANBgkqhkiG9w0BAQUFADCBwzELMAkGA1UEBhMC

VVMxFDASBgNVBAoTC0VudHJ1c3QubmV0MTswOQYDVQQLEzJ3d3cuZW50cnVzdC5u

…

RfPB+JbFi1WkzGuDFiAy2r77r5u3n+F+hJ+ePFCnP1zCvouGuAiS7vhCKw0T43aF

SApKv9ClOwqwVLht4wj5NI0LjosSzBcaM4eVyJ4K3FBTF3s=

-----END CERTIFICATE-----

Note that the begin and end
certificate statements are left in place.

To create a key pair and
certificate signing request do the following:

This will output your key
files and CSR. The vendor or PKI admin should sign the CSR and return
your SSL certificate as well as the CA certificate in your chain.

With the returned files and
openSSL, the sky is the limit.

Apache

First off, there are a couple
of ways to configure HTTPS on apache. The main methods are via mod_ssl
and Apache-SSL. Apache-SSL has not been updated in a while and should be
avoided.

Mod_ssl uses openSSL
libraries, so you will need openSSL to do anything fun. The native file
format for openSSL is called PEM or base64, which are text files, rather than
the DER format, that is binary.

The file that will contain
your SSL settings is the httpd.conf file. There are three settings that
you will care about.

SSLCertificateFile - This is the actual SSL certificate

SSLCertificateKeyFile - this is the associated private key
file. If there is a pass phrase on the key, it must be entered at
startup. This may not be appropriate for some environments.

SSLCertificateChainFile - This is chain file as described above.

F5 Local Traffic Manager – BigIP

First of all, you probably
should not be terminating SSL on your F5. “Need to know” or “reduced
surface area” says that you don’t want the F5 techs to be able to see your
decrypted traffic. There are few good cases for decrypting SSL at the
F5. If you need to inspect host headers or URLs, this may be necessary.

F5 uses openSSL so the
section above will help get the needed key and cert material.

In the SSL Client Profile,
there is an option to set the Chain file. Simply copy over the chain file
in the openSSL format and apply the setting.

F5 LTM

The LTM does not have extra
ports setup by default. If you do not setup the HTTP port, the user will
get a server not found error. Consider creating a VIP and iRule that
redirects the user to your HTTPS site.

The iRule would look like
this:

when HTTP_REQUEST {

HTTP::redirect
"https://[HTTP::host][HTTP::uri]"

}

This simply redirects the
browser, rewriting the URL with and HTTPS replacing the HTTP.

Not Covered

A few things may seem to be
missing here. This document focuses on PKI related configuration.
Different apps and OSs also have other SSL/TLS settings that are not directly
related to PKI.

Cipher Suites

As part of the SSL
negotiation, the client and server agree on a ciphersuite that is common to
them both, hopefully the strongest common suite. As attacks and computing
power change, the “best” ciphers change. It is called a suites, as
it defines ciphers users in four areas, authentication, key exchange, messaged
integrity check (MAC), and the bulk (symmetric) cipher used, including mode,
for block ciphers. There is a wide array of combinations of these parameters.
As the suite is only as strong as its weakest link, most ciphersuites should be
eliminated from your configuration. As of December 2011, you should
eliminate any suite that sets any of the four areas to null. You should
eliminate any suite that relies on MD5 for its MAC. You should eliminate any
ciphersuite that uses a block cipher in CBC mode. One should also
eliminate suites that utilize a symmetric cipher with a key shorter than 128
bit.

Apache

Mixed Content

I won't go into how to fix
it, or configure your systems, cause you don't pay me... but, creating a
page that uses SSL, but that has elements in the page that are not
over SSL is called creating mixed content. This is common for those who
use CDNs, as their images are all served up via HTTP.

Mixed content gives the user
fun symbols that indicate badness, but how bad?? We can't expect users to
make the call on this kind of thing.

Summary

If you want to make the customer
experience good, and avoid litigation, make sure to configure SSL on your web
servers correctly. Any failure in your configuration may put the client
at risk, but it may be a risk that is assumed by you, due to contracts.

Friday, March 2, 2012

As my readers know, in May of 2011, I coined and began protecting the terms "Advanced Persistent Protection", "Advance Persistent Controls", and "Advanced Persistent Control Suite".

This week, my brilliant reader Sterling pointed out that I now have one peer who agrees with me, Enrique Salem, president and CEO of Symantec. In a post for SC Magazine, he makes four bullet points about Advanced Persistent Protection. I'm thinking three is the limit for reasonable use. If Symantec is going to start pimping my copyrighted phrase, they need to pay!

Symantec, I have no desire to litigate as long as you give me what I am due. I'm glad to take a per use fee or we can negotiate an exclusive long term deal if you'd prefer. You'd better hurry though, RSA has already started talks. I'd hate for this to sink your upcoming marketing campaign.