Description of problem:
GNUTLS has a function gnutls_hash:
int gnutls_hash(gnutls_hash_hd_t handle, const void * ptext, size_t ptext_len);
it lets you perform md5 or any of the other hashes on a chunk of data;
this is implemented by a call to the nettle libraries implementation:
void
md5_update(struct md5_ctx *ctx,
unsigned length, <<<<<-
const uint8_t *data);
Note that here someone unwisely used an 'unsigned' length.
Thus if you pass a large ptext_len into gnutls_hash, the length gets
truncated and thus only produces a hash on part of the data.
In my case I passed a 4GB chunk which ended up hashing 0 bytes!
Nettle fixed this upstream in:
b8bfc32f (Niels Möller 2013-04-26 10:28:57 +0200)
Use size_t rather than unsigned for all hash-related functions.
I'm not sure if gnutls has any fixes.
I hit this during development, I'm not aware of anything in particular
that would hit it.
Here's a backtrace that shows the truncation:
Breakpoint 1, gnutls_hash (handle=0x7f5a74029020, text=0x7f5a83e00000, textlen=4294967296) at crypto-api.c:503
503 return _gnutls_hash((digest_hd_st *) handle, text, textlen);
(gdb) b wrap_nettle_hash_update
Breakpoint 2, wrap_nettle_hash_update (_ctx=0x7f5a74045b10, text=0x7f5a83e00000, textsize=4294967296) at mac.c:305
305 {
(gdb) list
Breakpoint 3, nettle_md5_update (ctx=0x7f5a74045b10, length=0, data=0x7f5a83e00000 "S\377") at md5.c:62
62 {
I guess you could fixit in the wrap_nettle_hash_update (that's part of gnutls), but I guess you would also have to go on a hunting trip for anything
else that makes the same mistake. Or update nettle.
Version-Release number of selected component (if applicable):
gnutls-3.3.8-12.el7_1.1.x86_64
nettle-2.7.1-4.el7.x86_64
How reproducible:
100%
Steps to Reproduce:
1. allocate 4GB of RAM
2. Initialise gnutls
3. gnutls_hash on the 4GB chunk using md5
Actual results:
The md5 result for a zero length input.
Expected results:
The md5 result for your actual data.
Additional info:

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2016-2218.html

Note

You need to
log in
before you can comment on or make changes to this bug.