This comment has been minimized.

edited

Do you know of a way to print/examine the trust policies of a certificate? After some experimentation, we have found that the certificates that aren't showing up seem to say "This certificate has custom trust settings" in the Keychain UI. However, expanding the "Trust" section doesn't reveal any specifics:

I'm not sure how the certificate was added/got to this state. It seems that if we manually update the state to "Always Trust" in the Keychain UI, then the certificate is returned. However, we'd like to understand this further, since most apps seem to trust the certificate even with these "Custom" trust settings but Go with CGo does not, which is causing issues for us.

*To clarify, I suspect that the issue has to do with the trust settings marked for the certificate rather than with the certificate itself

This comment has been minimized.

Unfortunately, I'm not sure about the provenance of how the cert was added. However, I do suspect that how it was added/how it was upgraded from a previous add is at the root of the issue here.

I have a certificate in my System store called "MSCA-ROOT-01-CA". I'm not sure how it was added, but the screenshot earlier in the issue shows that it displays as "This certificate has custom trust settings", although the UI shows "Always Trust" for all the parts.

I added some debugging code to root_cgo_darwin.go (edits at the end of this post), and the resulting output for this cert is:

Based my reading of the code, this certificate is in the "Admin" domain and has a non-NULL but empty trust setting. Because the trust setting is empty and it isn't a system cert, it decides not to trust it.

This logically makes sense to me, but I guess the resulting behavior isn't consistent with other applications (other applications seem willing to use this certificate for verification).

I'm not sure how this entry was created for me, but that's clearly the issue. Two of these certificates are valid and are added, and thus increment the "common" count by 2. One of these certificates is expired. This one appears only in the CGo code, which accounts for the "1 CGo only".

Here are some of the certificates that show up only for non-CGo (out of the 13):

None of the other entries show up in my plist. In the keychain, these show up as "no value specified" for trust:

The one curious thing is the extra certificate that shows up for CGo only after the local modification. That certificate is an old CA certificate I have that is expired:

It's showing up because before the modification it was in my admin-trust.plist with map[string]string{} as the trustSettings, so the change now includes it. I'm guessing the non-CGo code automatically prunes it based on expiration.

I don't think this should be an issue since even if the cert is added as a root cert, any code that does validation should properly check the expiration status.

This comment has been minimized.

edited

As far as the certificates with "kSecTrustSettingsResult":"-2147409654". I think this is why they're not marked as trusted. -2147409654 is not a valid value. The relevant Go code only checks a couple of the values (see SecTrustSettingsResult).

On those 13-2 certificates I wonder again if the plist/trust policies were generated properly. Just search for DigiCert SHA2 High Assurance Server CA gives a different cert fingerprint than what your policy has. See: https://crt.sh/?id=2900424 (SHA1 prefix A031C467) https://crt.sh/?id=2900424.

This comment has been minimized.

Yes, this behavior is on a machine that's managed by a company and uses tools to do so -- something along the way there writing an invalid policy entry is definitely a possibility.

I guess the difference in observed behavior is that most macOS applications (or Apple's cert API itself?) are more lenient on their verification here? Even though strict validation may technically be correct, if it results in an observable difference in behavior between native macOS apps and Go apps with CGo enabled that stills seems like it could be an issue (and I don't really have any good way of knowing how common or uncommon this scenario may be more broadly).

This comment has been minimized.

edited

Sorry you're right -- the only place where I know that to be true is for the trustSettings being an empty map (one of those certificates was a CA that was verifying all connections, but wasn't picked up by CGo before the proposed change).

Yes, I think that theory makes sense -- as shows in the UI for the certificates (the last screenshot), all of those certificates seem to be intermediate CAs.

If that's the case, I should probably update my PR to set trustRoot=1 (rather than trustAsRoot=1) so that the post-filtering step still happens -- I think that should eliminate that case of the one extra certificate being present only in CGo after the change (will verify locally now)

Edit: it didn't fix the issue I was seeing because that cert was a root cert (but just expired). However, functionally I think the change is correct, so updated PR.

This comment has been minimized.

edited

Ah interesting -- so that approach does use trustAsRoot for the certificates with empty entries (rather than `trustRoot), so it does add intermediate certs, which this one doesn't after my last update.

The only added certificate that I regularly use with Go is a custom root certificate, so I would not be impacted by this either way. I don't have a strong opinion on whether or not intermediates should be considered as root -- however, I do think it would be nice for this behavior to be consistent (at a minimum between CGo and non-CGo on Darwin, and in an ideal world across all platforms).

If there's a particular test you want me to run around validation with intermediate certs, if you outline it I can give it a stab and update with results.

This comment has been minimized.

What's the process to get more eyes on this? Whether it's via my PR or another mechanism, it would be nice to determine a path forward as there are many people on our team who seem to be encountering this issue

Finally, the cgo path is checking if any policy (ssl or any other explicitly set) has a kSecTrustSettingsResult value (ignoring the defaults, see above), with the last one in the array winning, omitting the following certificate (!!).

It's fairly late in the freeze, but I'm inclined to fix these, and maybe even backport them, because ignoring the policy types can lead to inclusion of roots that are not supposed to be trusted for TLS, and although crypto/x509 is not TLS-specific, it is meant to serve the WebPKI. @agl agree?

This comment has been minimized.

Just a comment: I have spent 2 hours today to debug this and unfortunately there seems no way to persuade/usr/bin/security to add the correct result type. If you specify „trustRoot“ (or nothing as „trustRoot“ is the default), security will just remove „Result Type“.

In Keychain.app the empty and explicitly set result types are not distinguishable, only security dump-trust-settings will show this.

Was completely confusing as Safari, curl and subversion all worked like a charm.

This comment has been minimized.

edited

If the test fails, please copy the whole output in a comment here. Do check that there's nothing you consider sensitive in it (it only lists names of certificates in your keychain, which might however include names of S/MIME senders) and if you'd prefer to report privately email filippo at golang.org.

This comment has been minimized.

edited

Unfortunately this change is hard to test and interacts with a very complex API, so I did not feel safe shipping it in 1.11 at the last minute. We will be targeting the next minor release after more testing.

As a temporary workaround, in some cases you can make the trust settings explicit by opening Keychain Access, finding the releavnt certificate, and marking it as Always Trust. It might even already show as such in the UI, because empty trust settings are the same as Always Trust, but Go doesn't understand that. Just toggling it in the UI back and forth should make the settings explicit.

It did not work for me unfortunately, although the trust settings are now printed by @FiloSottile test:

The cgo path was not taking policies into account, using the last
security setting in the array whatever it was. Also, it was not aware of
the defaults for empty security settings, and for security settings
without a result type. Finally, certificates restricted to a hostname
were considered roots.
The API docs for this code are partial and not very clear, so this is a
best effort, really.
Updates #24652
Change-Id: I8fa2fe4706f44f3d963b32e0615d149e997b537d
Reviewed-on: https://go-review.googlesource.com/c/128056
Run-TryBot: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Adam Langley <agl@google.com>
Reviewed-by: Adam Langley <agl@golang.org>

Certificates without any trust settings might still be in the keychain
(for example if they used to have some, or if they are intermediates for
offline verification), but they are not to be trusted. The only ones we
can trust unconditionally are the ones in the system roots store.
Moreover, the verify-cert invocation was not specifying the ssl policy,
defaulting instead to the basic one. We have no way of communicating
different usages in a CertPool, so stick to the WebPKI use-case as the
primary one for crypto/x509.
Updates #24652
Change-Id: Ife8b3d2f4026daa1223aa81fac44aeeb4f96528a
Reviewed-on: https://go-review.googlesource.com/c/128116
Reviewed-by: Adam Langley <agl@google.com>
Reviewed-by: Adam Langley <agl@golang.org>

This comment has been minimized.

@cvigo Hmm, that's weird because the tests pass, suggesting cgo and non-cgo agree, and the debug output suggests "Global Root CA" was added to the pool. Is the chain well-formed, with all the necessary intermediates?

This comment has been minimized.

Global Root CA was set by me as "Always Trust", the intermediate Global Issuing CA Infrastructure and the final Cert were, therefore, implicitly trusted and reported as valid by Google Chrome and curl, but rejected by go get and go mod

Now I changed the intermediate Global Issuing CA Infrastructure from "Use Syetem Defaults" to explicit "Always Trust" and everything works, including go 1.11.3

I thought the trustworthiness would be inherited from the root certificate down the tree and, actually, that is the case for browsers including curl, but Go is behaving differently.

This comment has been minimized.

It is indeed supposed to be inherited, so that's not intended behavior, if the server sends a full chain including the intermediate. There might also be something about the root CA that makes it disqualified from forming chains, if you can share it at filippo@golang.org I can look into that.

This comment has been minimized.

It is indeed supposed to be inherited, so that's not intended behavior, if the server sends a full chain including the intermediate. There might also be something about the root CA that makes it disqualified from forming chains, if you can share it at filippo@golang.org I can look into that.