CVE-2009-1378: OpenSSL DTLS fragment handling memory DoS

This bug was reported by Robin Seggelmann and it was discovered by Daniel Mentz. It affects OpenSSL prior to 1.0.0-beta2 release. Here is the vulnerable code from the latest OpenSSL release which is 0.9.8k:

The bug is simple. An attacker can send numerous SEQ handshake packets to to trigger many allocations with size of frag_len which is calculated at lines 568-575. The allocation takes place at line 578 of dtls1_process_out_of_seq_message() function. To fix this, the following patch was used at ssl/d1_both.c which is where this function is located:

Those new checks ensure that only messages with sequence numbers less than 10 in advance will be buffered by that routine. A PoC trigger code for this bug was published by Jon Oberheide and you can find it here.