Issue #9659 has been updated by Usaku NAKAMURA.
Status changed from Closed to Feedback
----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-47794
* Author: Jared Jennings
* Status: Feedback
* Priority: Normal
* Assignee:
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: DONTNEED, 2.1: DONTNEED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.
On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:
~~~
$ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
Aborted (core dumped)
~~~
The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.
I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:
* source:ext/digest/digest.c@43668#L551
* source:ext/digest/digest.c@43668#L589
* source:ext/digest/digest.c@43668#L627
---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)
--
https://bugs.ruby-lang.org/