Using the tcnative library compiled against a FIPS-compatible build of OpenSSL on Windows results in a FIPS fingerprint error when the FIPSMode attribute in the AprLifecycleListener's Listener element in server.xml is set to "on". As a result, Tomcat fails to start.
When the attribute is set to "off", no errors occur, Tomcat starts up successfully, and SSL services perform normally.
The error that results in the catalina log file is:
SEVERE: Failed to initialize the SSLEngine.
java.lang.Exception: error:2D06B06F:FIPS routines:FIPS_check_incore_fingerprint:fingerprint does not match
The problem appears to be that during startup, Tomcat's copy of the OpenSSL library libeay32.dll gets rebased from its desired memory address (by default, 0xFB00000) to a different address. Normally this wouldn't be an issue, but when operating in FIPS mode, the library executes a hash of itself which fails if it is not at its expected address. (See http://www.openssl.org/docs/fips/UserGuide-2.0.pdf, page 44.)
This was determined by using Process Explorer to examine the tomcat7.exe process following a successful startup of Tomcat with FIPSMode set to "off". The libeay32.dll library in that process displayed an "Image Base" address of 0xFB00000, indicating its desired base memory address, an a "Base" address of 0x63E20000 (or something else on other servers), indicating the actual base memory address being used for the library.
Additionally, the same procedure was used on a copy of the openssl.exe utility program, which successfully used FIPS mode with the same library used above. In that case, Process Explorer showed that the openssl.exe process' copy of libeay32.dll had the same value for the "Image Base" and "Base" addresses.
Working around this problem involved recompiling the OpenSSL library to request a different base memory address. (I used 0x6FB00000.)
Would it be possible to determine why, during startup, the libeay32.dll module's base memory address is being changed from the one it expects, and fix the problem? If not, it will need to be documented that in order to use a FIPS-compatible build of OpenSSL with TCNative in FIPS mode, the OpenSSL library will need to be recompiled with an alternate base memory address specified. (See the above PDF page for instructions for how to do so.)
This issue was experienced using Tomcat 7.0.32, tcnative 1.1.27, APR 1.4.6, and OpenSSL 1.0.1c with FIPS module 2.0.4, all on Windows Server 2008.

I think this turns out to be a problem with the OpenSSL build, not the tcnative build. So, there's nothing that tcnative can do about it, especially because the ASF doesn't distribute a FIPS-enabled version of OpenSSL alongside tcnative (nor can they, as I understand).
It is well worth noting in our build instructions that running with a FIPS-enabled library can run into these issues and the the solution is (I think) carefully building and linking OpenSSL such that all addresses are fixed.

While Tomcat does provide a binary openssl.exe for Microsoft Windows users, it is not FIPS-enabled. The only way to get a "proper" FIPS-enabled binary is to build it yourself for your own uses.
If we can find a procedure that will work with a modern MS Visual Studio, then we should certainly document it. I don't have access to a Windows toolchain, so I can't really be of any help.

We now have a documented and tested process for building a statically linked tc-native with a FIPS enabled OpenSSL on 64-bit Windows.
I'm not concerned by the need to change the base address when building a dynamically linked library. The OpenSSL docs cover that.

This is ASF Bugzilla: the Apache Software Foundation bug system. In case
of problems with the functioning of ASF Bugzilla, please contact
bugzilla-admin@apache.org.
Please Note: this e-mail address is only for reporting problems
with ASF Bugzilla. Mail about any other subject will be silently
ignored.