Debugging JWT validation problems between an OWIN app and IdentityServer4

This post describes an issue I ran into at work recently, as part of an effort to migrate our identity application from IdentityServer3 to IdentityServer4. One of our services was unable to validate the JWT sent as a bearer token, even though other services were able to validate it. I'll provide some background to the migration, a more detailed description of the problem, and the solution.

tl;dr; The problematic service was attempting to call a "validation endpoint" to validate the JWT, instead of using local validation. This works fine with IdentityServer3, but the custom access token validation endpoint has been removed in IdentityServer4. Forcing local validation by calling ValidationMode.Local when adding the middleware with app.UseIdentityServerBearerTokenAuthentication() fixed the issue.

An overview of the system

The system I'm working on, as is common these days, consists of a variety of different services working together. On the front end, we have a JavaScript SPA. This makes HTTP calls to several different server-side apps to fetch the data that it displays to the user. In this case, the services are ASP.NET (not Core) apps using OWIN to provide a Web API. These services may in turn make requests to other back-end APIs, but we'll ignore those for now. In this post I'm going to focus on two services, called the AdminAPI service and the DataAPI service:

For authentication, we have an application running IdentityServer3 which is again non-core ASP.NET (sidenote: I still haven't worked out what we're calling non-core ASP.NET these days!). Users authenticate with the IdentityServer3 app, which returns a JSON Web Token (JWT). The client app sends the JWT in the Authorization header when making requests to the AdminAPI and the DataAPI.

Before the AdminAPI or the DataAPI accept the JWT sent in the Authorization header, they must first validate the JWT. This ensures the token hasn't been tampered with and can be trusted.

A brief background on JWT tokens and Identity

In order to understand the problem, we need to have an understanding of how JWT tokens work, so I'll provide a brief outline here. Feel free to skip this section if this is old news, or check out https://jwt.io/introduction for a more in-depth description.

A JWT provides a mechanism for the IdentityServer app to transfer information to another app (e.g. the AdminAPI or DataAPI) through an insecure medium (the JavaScript app in the browser) in such a way that the data can't be tampered with. It's important they can't be tampered with as they're often used for authentication and authorisation - you don't want users to be able to impersonate other users, or grant themselves additional privileges.

A JWT consists of three parts:

Header - A description of the type of token (JWT) and the algorithms used to secure the token

Payload - The information to be transferred. This typically includes a set of claims, which describe the entity (i.e. the user), and potentially other details such as the expiry date of the token, who issued it etc.

Signature - A cryptographic signature that describes the header and the payload. If either the header or payload are modified, the signature will no longer be correct, so the JWT can be discarded as fraudulent.

In our case, the signature for the JWT is created using an X.509 certificate using asymmetric cryptography. The signature is generated using the private key of the certificate, which is only known to IdentityServer and is not exposed. However, anyone can validate the signature using the public certificate, which IdentityServer makes available at well-known URLs.

Upgrading IdentityServer

I had been tasked with porting the existing ASP.NET IdentityServer3 app to an ASP.NET Core IdentityServer4 app. IdentityServer3 and IdentityServer4 both use the OpenID Connect and OAuth 2 protocols, so from the point of view of the consumers of the app, upgrading IdentityServer in this way should be seamless.

The good news is that for the most part, the upgrade really was painless. IdentityServer 4 has a few different abstractions to IdentityServer3, so you may have to tweak some things and implement some different interfaces, but the changes are relatively minor and make sense. Scott Brady has a great post on IdentityServer 4, or you could watch Dominick Baier explain some of the changes himself on Channel 9.

Trouble in paradise

With the IdentiyServer app ported to .NET Core, all that remained was to test the integration with the AdminAPI and DataAPI. Initial impressions were very positive - the client app would authenticate with the IdentityServer app to retrieve a JWT, and would send this in requests to the AdminAPI. The AdminAPI validated the signature in the JWT token, and used the claims it contained to execute the action. All looking good.

The problem was the DataAPI. When the client app navigated to a given page, it would send a request to the DataAPI with the same JWT as it sent to the AdminAPI. However, the DataAPI failed to validate the signature.

How could that be? Both APIs were configured with the same IdentityServer as the authority, and the same JWT was being sent to both APIs. I tweaked the DataAPI configuration to make sure it was identical to the AdminAPI and logged as much as I could, but try as I might, I couldn't find any differences between the two APIs. Both APIs were even running on the same server, in the same web site, using the same IIS app pool.

Yet the AdminAPI could validate the token, and the DataAPI could not.

IdentityServer3.AccessTokenValidation

At this point, I'll back up slightly, and describe exactly how the AdminAPI and DataAPI validate the JWTs.

We are using the IdentityServer3.AccessTokenValidation library to validate the JWTs. This extracts the identity contained in the JWT to authenticate the incoming request, and assigns it to the IPrincipal of the request. This library includes an OWIN middleware that you can add to your IAppBuilder pipeline something like the following (from the DataAPI):

This snippet shows all the configuration required to validate incoming tokens, extract the identity in the JWT payload, and assign the principal for the current thread. In this example, the IdentityServer app is hosted at https://www.test.domain/identity, and incoming JWTs must have the "DataAPI" scope to be considered valid

If you're not familiar with IdentityServer, it might surprise you that no other configuration is required. No client IDs, no secrets, no certificates. Instead, thanks to the use of open standards (OpenID Connect), the validation middleware can contact your IdentityServer app to obtain all the information it needs.

When the validation middleware needs to validate an incoming JWT, it calls a well-known URL on IdentityServer (literally well-known; the URL path is /.well-known/openid-configuration). This returns a JSON document indicating the capabilities of the server, and the location of a variety of useful links. The following is a fragment of a discovery document as an example:

To actually validate an incoming token, the middleware uses one of two approaches:

Local - The middleware uses the discovery document and the jwks_uri link to dynamically download the public certificate required to validate the JWTs.

ValidationEndpoint - The middleware sends the JWT to IdentityServer and asks it to validate the token.

You can explicitly choose which validation mode the middleware should use, but it defaults to Both. As stated by Brock Allen on Stack Overflow:

"both" will dynamically determine which of the two approaches described above [to use] based upon some heuristics on the incoming access token presented to the Web API.

That gives us the background, so lets get back to the problem in hand, why one of our apps could validate the JWT, and the other could not.

Back to the problem, and the solution

After much trial and error, I finally discovered that the problem I was having was due to the "both" heuristic, and the validation mode it was choosing. In the AdminAPI (which was able to validate the JWTs issued by IdentityServer4) the middleware was choosing the Local validation mode. It would retrieve the public certificate of the X.509 cert used to sign the token by using the OpenID Connect discovery document, and could verify the signature.

The DataAPI on the other hand, was trying to use ValidationEndpoint validation of the JWT. For some reason, the heuristic decided that local validation wasn't possible, and so was trying to send the JWT to IdentityServer4 for validation.

Unfortunately, the custom access token validation endpoint available in IdentityServer3 was removed in IdentityServer4. Every time the DataAPI attempted to validate the JWT, it was getting a 404 from the IdentityServer4 app, so the validation was failing.

The simple solution was to force the middleware to always use Local validation, by updating the ValidationMode in the middleware options:

As soon as the DataAPI was updated with the above change, it was able to validate the JWTs created using IdentityServer4, and t app started working again.

An obvious follow-up to this issue would be to figure out why the the DataAPI was choosing ValidationEndpoint validation instead of Local validation. I'm sure the answer lies somewhere in this source code file, but for the life of me I can't figure it out; given it was the same token, and same middleware configuration in both cases it should have been the same validation type as far as I can see!

Ultimately, it just needs to work, so I've moved on.

No, it doesn't irritate me not knowing why it happens.

Honest.

Summary

Upgrading from IdentityServer3 to IdentityServer4, and in the process switching from an ASP.NET app to ASP.NET Core, is not something that should be taken lightly, but overall the process went smoothly. In particular, .NET Core 2.0 made the port much easier.

The only issue was that a consumer of IdentityServer4 was attempting to use ValidationEndpoint to validate tokens, when using the IdentityServer3.AccessTokenValidation library for authentication. IdentityServer4 has removed the custom access token validation endpoint used by this method, so attempts to validate JWTs will fail when it's used.

Instead, you can force the middleware to use Local validation instead. This downloads the public certificate from IdentityServer4, and validates the signature locally, without having to call custom endpoints.