cert_cb and TLS tickets

cert_cb and TLS tickets

Hello,

During development of one feature for my TLS proxy bud, I have discovered that the cert_cb is invoked only for newly generated tickets/sessions. The reasoning behind this is clear, but I believe that it is most likely needs a revision. Here is my reasoning:

The major use case is choosing a certificate/private key either dynamically (based on various parameters of SSL structure) or asynchronously (by using SSL_ERROR_WANT_X509_LOOKUP). However when the TLS ticket is provided by the client, it will be parsed and loaded using the ticket key from the main context, without giving a way for application to override it for particular servername (from SNI). Furthermore, with the TLS ticket provided application can no longer chose to provide a different certificate in case of expiration or revocation.

The documentation says:

"It is B<always> called even is a certificate is already set so the callback

can modify or delete the existing certificate."

Additionally, when talking about things allowed in `cert_cb`:

"An application will typically call SSL_use_certificate() and

SSL_use_PrivateKey() to set the end entity certificate and private key.

It can add intermediate and optionally the root CA certificates using

SSL_add1_chain_cert()."

It only says that it is *typically* for applications to use these functions not limiting them to set and use different secure contextes on cert_cb. This is in fact what I've used in bud, and what we use in node.js too.

Proposal:

Invoke cert_cb before looking up session ids and parsing tls tickets. It is a safe change, and it should be done to match intention of this API documented in man pages.

Re: cert_cb and TLS tickets

On Fri, Dec 09, 2016 at 08:43:01PM +0100, Fedor Indutny wrote:

> During development of one feature for my TLS proxy bud, I have discovered
> that the cert_cb is invoked only for newly generated tickets/sessions. The
> reasoning behind this is clear, but I believe that it is most likely needs
> a revision. Here is my reasoning:

The callback is *correctly* only called when choosing the server
certificate. In *that* case, it is always called. When sessions
are resumed, there is no certificate to present, so no callback is
made.

> Furthermore, with the TLS ticket provided
> application can no longer chose to provide a different certificate in case
> of expiration or revocation.

You can implement a ticket key callback that when appropriate, will
decline tickets under suitable conditions, in which case a full
handshake will be performed.

Custom ticket callbacks that do session ticket key rotation are a
good idea in any case, the default tickets are not rotated, which
is not apppropriate for long-running processes.

Re: cert_cb and TLS tickets

> During development of one feature for my TLS proxy bud, I have discovered
> that the cert_cb is invoked only for newly generated tickets/sessions. The
> reasoning behind this is clear, but I believe that it is most likely needs
> a revision. Here is my reasoning:

The callback is *correctly* only called when choosing the server
certificate. In *that* case, it is always called. When sessions
are resumed, there is no certificate to present, so no callback is
made.

That's why I said that the intended logic is clear. However, this implementation makes the API unusable in at least one case. Even more, there are no alternatives to this API, meaning that there is no other way to delay certificate/key/context selection. Clearly this is the case for which the API was created.

> Furthermore, with the TLS ticket provided
> application can no longer chose to provide a different certificate in case
> of expiration or revocation.

You can implement a ticket key callback that when appropriate, will
decline tickets under suitable conditions, in which case a full
handshake will be performed.

Custom ticket callbacks that do session ticket key rotation are a
good idea in any case, the default tickets are not rotated, which
is not apppropriate for long-running processes.

Ticket rotation is already possible in both bud and node.js and is completely unrelated to this issue. Even with ticket rotation valid tickets will cause asynchronous SNI lookups to be skipped.

Let me put this into real example. "bud" does SNI balancing and asynchronous OCSP stapling. When there are no tickets - it will connect to some HTTP backend to fetch certificate, key, and list of cleartext backends. It will also asynchronously fetch OCSP response from either remote cache or OCSP authority.

However, when valid TLS ticket is received - OpenSSL will skip the `cert_cb` and will just do regular handshake as no `cert_cb` was set. This breaks balancing and OCSP stapling fetching, which are major use cases for both bud and node.js (which can do all of this too through external modules).

Re: cert_cb and TLS tickets

On Fri, Dec 09, 2016 at 09:47:19PM +0100, Fedor Indutny wrote:

> > The callback is *correctly* only called when choosing the server
> > certificate. In *that* case, it is always called. When sessions
> > are resumed, there is no certificate to present, so no callback is
> > made.
>
> That's why I said that the intended logic is clear. However, this
> implementation makes the API unusable in at least one case. Even more,
> there are no alternatives to this API, meaning that there is no other way
> to delay certificate/key/context selection. Clearly this is the case for
> which the API was created.

Resumed sessions don't select any certificates, so it makes no
sense to invoke certificate selection callbacks.

> Ticket rotation is already possible in both bud and node.js and is
> completely unrelated to this issue. Even with ticket rotation valid tickets
> will cause asynchronous SNI lookups to be skipped.

If you deny the key id, then a new session will be negotiated, and
certificate selection callbacks will be invoked.

> However, when valid TLS ticket is received - OpenSSL will skip the
> `cert_cb` and will just do regular handshake as no `cert_cb` was set. This
> breaks balancing and OCSP stapling fetching, which are major use cases for
> both bud and node.js (which can do all of this too through external
> modules).

There is no OCSP with resumed sessions, no certificates are sent
by the server or checked by the client. If you want certificates,
don't do session resumption.

Re: cert_cb and TLS tickets

> > The callback is *correctly* only called when choosing the server
> > certificate. In *that* case, it is always called. When sessions
> > are resumed, there is no certificate to present, so no callback is
> > made.
>
> That's why I said that the intended logic is clear. However, this
> implementation makes the API unusable in at least one case. Even more,
> there are no alternatives to this API, meaning that there is no other way
> to delay certificate/key/context selection. Clearly this is the case for
> which the API was created.

Resumed sessions don't select any certificates, so it makes no
sense to invoke certificate selection callbacks.

The problem with ticket keys is that they are assigned globally and there is no way to control what's happening independently in asynchronous manner. Thus API looks a bit odd: on one hand OpenSSL allows selecting certs asynchronously, on other hand TLS tickets are going to be processed synchronously anyway.

> Ticket rotation is already possible in both bud and node.js and is
> completely unrelated to this issue. Even with ticket rotation valid tickets
> will cause asynchronous SNI lookups to be skipped.

If you deny the key id, then a new session will be negotiated, and
certificate selection callbacks will be invoked.

This means disabling TLS ticket keys for both bud and node.js . It is possible, but will hurt performance significantly.

> However, when valid TLS ticket is received - OpenSSL will skip the
> `cert_cb` and will just do regular handshake as no `cert_cb` was set. This
> breaks balancing and OCSP stapling fetching, which are major use cases for
> both bud and node.js (which can do all of this too through external
> modules).

There is no OCSP with resumed sessions, no certificates are sent
by the server or checked by the client. If you want certificates,
don't do session resumption.

OpenSSL is behaving correctly. This is a fundamental feature of TLS.

While I agree about OCSP, I don't see how this fundamental feature helps in backend selection case. Clearly it is impossible to choose backends dynamically with such reduced API implementation.

All in all, what I am asking for is an internal change that will still adhere to the documentation. There is no fundamental feature of TLS that could prohibit this.

Re: cert_cb and TLS tickets

During development of one feature for my TLS proxy bud, I
have discovered that the cert_cb is invoked only for newly
generated tickets/sessions. The reasoning behind this is
clear, but I believe that it is most likely needs a revision.
Here is my reasoning:

The major use case is choosing a certificate/private key
either dynamically (based on various parameters of SSL
structure) or asynchronously (by
using SSL_ERROR_WANT_X509_LOOKUP). However when the TLS ticket
is provided by the client, it will be parsed and loaded using
the ticket key from the main context, without giving a way for
application to override it for particular servername (from
SNI). Furthermore, with the TLS ticket provided application
can no longer chose to provide a different certificate in case
of expiration or revocation.

If you had a callback that ran before session resumption (possibly
the existing SNI callback, possibly a new callback), would that
allow you to solve your problem? I would very much like to see such
an early callback so as to be able to do SNI processing before
resumption, possibly even before version negotiation. (And yes, I
should put my money where my mouth is and come up with a patch.)

Re: cert_cb and TLS tickets

During development of one feature for my TLS proxy bud, I
have discovered that the cert_cb is invoked only for newly
generated tickets/sessions. The reasoning behind this is
clear, but I believe that it is most likely needs a revision.
Here is my reasoning:

The major use case is choosing a certificate/private key
either dynamically (based on various parameters of SSL
structure) or asynchronously (by
using SSL_ERROR_WANT_X509_LOOKUP). However when the TLS ticket
is provided by the client, it will be parsed and loaded using
the ticket key from the main context, without giving a way for
application to override it for particular servername (from
SNI). Furthermore, with the TLS ticket provided application
can no longer chose to provide a different certificate in case
of expiration or revocation.

If you had a callback that ran before session resumption (possibly
the existing SNI callback, possibly a new callback), would that
allow you to solve your problem? I would very much like to see such
an early callback so as to be able to do SNI processing before
resumption, possibly even before version negotiation. (And yes, I
should put my money where my mouth is and come up with a patch.)

That's exactly what I am asking for. Putting it before session resumption will be enough for my use case, though.

During development of one feature for my TLS proxy bud, I
have discovered that the cert_cb is invoked only for newly
generated tickets/sessions. The reasoning behind this is
clear, but I believe that it is most likely needs a revision.
Here is my reasoning:

The major use case is choosing a certificate/private key
either dynamically (based on various parameters of SSL
structure) or asynchronously (by
using SSL_ERROR_WANT_X509_LOOKUP). However when the TLS ticket
is provided by the client, it will be parsed and loaded using
the ticket key from the main context, without giving a way for
application to override it for particular servername (from
SNI). Furthermore, with the TLS ticket provided application
can no longer chose to provide a different certificate in case
of expiration or revocation.

If you had a callback that ran before session resumption (possibly
the existing SNI callback, possibly a new callback), would that
allow you to solve your problem? I would very much like to see such
an early callback so as to be able to do SNI processing before
resumption, possibly even before version negotiation. (And yes, I
should put my money where my mouth is and come up with a patch.)

That's exactly what I am asking for. Putting it before session resumption will be enough for my use case, though.

Re: cert_cb and TLS tickets

I was mostly under the impression that Matt Caswell was planning
to add a generic "early callback" that gets called just after
extensions are read but before they are processed, and was waiting
to see what that looked like and whether the same API could be
reasonably backported to 1.1.0 (not necessarily in the upstream
repo, of course). But I also haven't had a chance to work through
the extensions processing revamp that landed this week, yet.

Just moving the existing servername_callback earlier is rather
non-trivial, since it expects to be able to reach into the session
in various ways, and of course there's no session yet. Some
things can be worked around, like SSL_get_servername(), but I'm
not sure I can enumerate all the things that existing
servername_callbacks are supposed to be able to do, so it's hard
to be confident that I can move the servername_callback that early
without introducing regressions somewhere.

-Ben

On 12/09/2016 04:41 PM, Fedor Indutny
wrote:

Oh, just to restate it. I'm willing to submit the
patch if we agree on what exactly it should do.

During development of one feature for
my TLS proxy bud, I have discovered that
the cert_cb is invoked only for newly
generated tickets/sessions. The
reasoning behind this is clear, but I
believe that it is most likely needs a
revision. Here is my reasoning:

The major use case is choosing a
certificate/private key either
dynamically (based on various parameters
of SSL structure) or asynchronously (by
using SSL_ERROR_WANT_X509_LOOKUP).
However when the TLS ticket is provided
by the client, it will be parsed and
loaded using the ticket key from the
main context, without giving a way for
application to override it for
particular servername (from SNI).
Furthermore, with the TLS ticket
provided application can no longer chose
to provide a different certificate in
case of expiration or revocation.

If you had a callback that ran before
session resumption (possibly the existing SNI
callback, possibly a new callback), would that
allow you to solve your problem? I would very
much like to see such an early callback so as to
be able to do SNI processing before resumption,
possibly even before version negotiation. (And
yes, I should put my money where my mouth is and
come up with a patch.)

That's exactly what I am asking for. Putting it
before session resumption will be enough for my use
case, though.

Re: cert_cb and TLS tickets

What seems most sensible is to have a callback once the hello is parsed, allow the callback to change almost anything it wants. I would be very disappointed if we did something that was too specific to some special use cases.

Re: cert_cb and TLS tickets

On Sat, Dec 10, 2016 at 11:13:48AM +0100, Fedor Indutny wrote:
> This totally makes sense. Unfortunately, adding a new API method for this
> means that I'll have to re-introduce ClientHello parser in bud, and make a
> wider use of it in Node.js again.

FWIW, BoringSSL offers an early callback that is passed a semi-parsed CH, and
an API to extract specific extensions from it (though this returns the raw
unparsed extension body). Something similar could be adopted for OpenSSL.

Whether this should be called in the CH post process phase (immediately before
cert_cb) or much earlier (like BoringSSL) is likely to affect the implementation
though (e.g. I'm not sure if the CH buffer is still available in the post
process).

Might be worth noting that BoringSSL changed the CH processing recently, by
moving the session resumption logic after cert_cb, which means cert_cb is now
called every time, but without a SSL_SESSION being available. So calling the
cert_cb unconditionally is not unheard of.

On Sat, Dec 10, 2016 at 11:13:48AM +0100, Fedor Indutny wrote:
> This totally makes sense. Unfortunately, adding a new API method for this
> means that I'll have to re-introduce ClientHello parser in bud, and make a
> wider use of it in Node.js again.

FWIW, BoringSSL offers an early callback that is passed a semi-parsed CH, and
an API to extract specific extensions from it (though this returns the raw
unparsed extension body). Something similar could be adopted for OpenSSL.

Whether this should be called in the CH post process phase (immediately before
cert_cb) or much earlier (like BoringSSL) is likely to affect the implementation
though (e.g. I'm not sure if the CH buffer is still available in the post
process).

Might be worth noting that BoringSSL changed the CH processing recently, by
moving the session resumption logic after cert_cb, which means cert_cb is now
called every time, but without a SSL_SESSION being available. So calling the
cert_cb unconditionally is not unheard of.

The motivation was to flip our session resumption and cipher suite logic. cert_cb must be called before cipher selection, and extensions must be processed before cert_cb, hence the full reorder. (We'd already replaced OpenSSL's callbacks with BoringSSL's "early callback" mentioned elsewhere in the thread.) But I agree this is also a more sensible ordering.

Suppose a server which prefers GCM > CBC receives:

ClientHello{cipher_suites={GCM, CBC}, ticket=session{CBC}}

The preferred cipher suite is GCM, but OpenSSL will opt to resume the session and end up negotiating CBC. This may happen if the client or server changed but retained ticket continuity. (Common for servers. Or client might persist sessions.) After those changes, BoringSSL will pick the cipher first and only resume if the cipher matches.

The suboptimal negotiation is temporary in TLS 1.2, so it's not that important. This assumes TLS 1.2 sessions are short-lived and TLS 1.2 ticket renewal is rare (or at least doesn't extend lifetimes).

In TLS 1.3, we have psk_dhe_ke, renewal actually refreshes keys, resumption is more valuable (0-RTT, while 1.2 1-RTT can be done with False Start), and there may later be an extension to renew with fresh identity assertion. I anticipate 1.3 resumption to be more continuous. This suboptimal negotiation now persists. Better to pay a resumption miss once and break the cycle.

Fortunately, the "cert_cb before ciphers" rule is only true in TLS 1.2 where cipher suites include the key type. TLS 1.3's decoupled negotiation means OpenSSL can do reordered negotiation in 1.3 without breaking cert_cb compatibility but keep 1.2's negotiation old-style. We did it for both because it's easier to reason about, cleaner, and "mostly compatible".

On Sat, Dec 10, 2016 at 11:13:48AM +0100, Fedor Indutny wrote:
> This totally makes sense. Unfortunately, adding a new API method for this
> means that I'll have to re-introduce ClientHello parser in bud, and make a
> wider use of it in Node.js again.

FWIW, BoringSSL offers an early callback that is passed a semi-parsed CH, and
an API to extract specific extensions from it (though this returns the raw
unparsed extension body). Something similar could be adopted for OpenSSL.

Whether this should be called in the CH post process phase (immediately before
cert_cb) or much earlier (like BoringSSL) is likely to affect the implementation
though (e.g. I'm not sure if the CH buffer is still available in the post
process).

Might be worth noting that BoringSSL changed the CH processing recently, by
moving the session resumption logic after cert_cb, which means cert_cb is now
called every time, but without a SSL_SESSION being available. So calling the
cert_cb unconditionally is not unheard of.