.NET Core, OSX, libcurl, and OpenSSL

.NET Core makes it convenient to develop and test C# code across platforms. On my current project, this means we can do much of our work on Macs without ever firing up a Windows VM.

Even the best abstraction layers occasionally leak, though. Here’s a story of an OSX-specific issue we encountered, what we learned, and how to resolve it.

The Problem

So there I was, running an existing .NET Framework project on .NET Core for the first time (using .NET Core 1.1). The project uses a third-party library to communicate with a REST API. Like many problems in software development, it started with an error message:

System.PlatformNotSupportedException: The libcurl library in use (7.54.0) and its SSL backend ("SecureTransport") do not support custom handling of certificates. A libcurl built with OpenSSL is required.

libcurl provides an HTTP client, and it can be built with one of several TLS backends. It looked like we needed to be using OpenSSL instead of SecureTransport (which is Apple’s TLS Implementation).

For us, the exception came from within a third-party library, but you might encounter it if you were doing something advanced with HttpClient. For example, a contributor on the GitHub issue provided a short repro that attempts to check a certificate revocation list.

Get libcurl

So how can we get libcurl built with OpenSSL? The easiest way is to install Homebrew’s curlpackage, which provides both the curl command line tool and the libcurl library. It, too, uses SecureTransport by default, but we can choose OpenSSL instead:

brew install curl --with-openssl

After it’s installed, you’ll get a warning:

This formula is keg-only, which means it was not symlinked into /usr/local, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble.

Let’s see what kind of trouble, shall we?

A Bad Solution

One suggestion on the issue thread is to disregard the warning and force-link libcurl (brew link --force curl). This would promote curl and libcurl to paths within /usr/local. Then feed the library path to the loader by setting an environment variable: DYLD_LIBRARY_PATH=/usr/local/lib.

Side note: DYLD_LIBRARY_PATH is OSX’s version of LD_PRELOAD, which is awesome, and you can read more about it here.

With this environment variable in place, when the loader goes looking for libraries, it will look in /usr/local/lib first. This fixes our libcurl issue by prioritizing Brew’s version over the system version. Sweet!

But what if there were some other libraries in there that we didn’t want? Like, what if we needed Apple’s special version of libJPEG because it provides extra symbols that Homebrew’s libJPEG doesn’t offer?

Oops. This also happens with libtiff (__cg_TIFFClientOpen) and libpng (__cg_png_create_info_struct). For a practical repro of the conflict, brew install imagemagick, which depends on all three. If you ever need to analyze problems like this more thoroughly, you can view library and symbol dependencies with the otool and nm commands.

To fix this, we could uninstall or unlink those libs, but that feels like setting another hack on a leaning pile of hacks.

A Better Solution

Instead of prioritizing all of Homebrew’s libraries, it’s safer to take just what we need: DYLD_LIBRARY_PATH=/usr/local/opt/curl/lib. No force link necessary. Tada!

You could also use this technique to provide Homebrew’s OpenSSL to dotnet, instead of manually symlinking the libs as the current installation guide instructs.

The Best Solution

If you’re stuck on 1.x, the Better Solution above offers a feasible workaround that doesn’t require you to ignore your package manager’s warnings and introduce risk to other software on your system. If you’re able, though, now’s a good time to try out .NET Core 2.0. It should be releasing very soon!

I was going along this path but was using the Linux environment variables by mistake, thanks for clearing this up! I wanted to point out that if you’re using self signed certificates on a server for development, you may still need this even in .NET Core 2.0.

Here’s my scenario:

I’m building a web api that is relying on IdentityServer4 to issue tokens for user authorization. After getting the token server to work, I secured one of my api endpoints (I had to specifically specify the authentication scheme in the Authorize attribute) all was working fine! Until I tried to use my self-signed cert to secure communications (this will change in production of course). The authentication pipeline for the token makes a web request to the token authority. Within that code, a HttpClient object is being created and used. By default, the HttpClient will use libcurl which will not allow self-signed certs even if you have added them to your system keychain. There was a workaround that was added to .NET Core 2.0 for this which was to set your custom callback validator to a specific statically exposed delegate on the HttpClientHandler.

This does solve the problem if you have access to the HttpClient object and its handler. In my case, that is buried deep in the authentication handler pipeline of .NET Core. Thus, I had to use this workaround.

So far so good. I’m still trying to figure out the best way to handle non-web interactions with IdentityServer 4. We’ll see how it turns out. My goal is to have my own backend that supports asp.net identity, but also allow users to use FB or other oauth2 compatible sites to login as well.

I recently had my system crash due to bad memory and now self signed cert is not cooperating the way it was. I wish macOS libraries and utilities would trust the self signed certs the same way it does any other. It just doesn’t want to do that.