Sun xVM VirtualBox Guest Additions Memory Consumption (on guest OS)

This bug was recently disclosed by Sun Microsystems and it affects 1.6, 2.0.0, 2.0.2, 2.0.4, 2.0.6, 2.0.8, 2.0.10, 2.1, 2.2, 3.0.0, 3.0.2, 3.0.4, 3.0.6 and 3.0.8 releases of “Guest Additions” software provided by Sun xVM VirtualBox.
Guest Additions is a software available for Windows, Linux and Solaris and it is used in the installed guest operating systems for performance and integration purposes.
So, back to the actual code. As we can read in src/VBox/Additions/common/VBoxGuest/VBoxGuest.cpp of 3.0.8 release of VirtualBox we can find the following…

The aboce IOCTL handler is used by the aforementioned guest addition utilities software to perform a request on the hypervisor VMM. The request’s information are stored in a ‘VMMDevRequestHeader’ object which is passed as a pointer using ‘pReqHdr’ variable.
After initializing some variables with the request type and size, a check takes place to ensure that the minimum default size for that request is larger than the size contained in the request header. After that, a second size check takes place to ensure that the requested data fit in the requested size. Assuming that both conditions are true, VirtualBox will attempt to copy the request in heap memory using VbglGRAlloc() to allocate the required space and memcpy() to copy the actual data of the request. However, the request might still request more memory than it actually needs since it only checks that the space is more than the minimum. Because of this missing check, an attacker can create a malicious call to that IOCTL with a header size large enough to consume a lot of kernel memory and bring the guest OS in an unstable state.
To fix this, the above function was changed like this:

What it does is a series of checks the validate the request header’s contents. Specifically, it checks the following:
– The request is not NULL
– Data have at least the size of a valid request object
– The request’s size is larger than the data to be copied
– The data’s size cannot be less than the request’s expected size
– If the data’s size is equal to the expected one and the request’s size is not different of the expected one, return with success
– if the request type is VMMDevReq_LogString, VMMDevReq_VideoSetVisibleRegion, VMMDevReq_SetPointerShape as well as some 64-bit OS specific ones which are: VMMDevReq_HGCMCall32, VMMDevReq_HGCMCall64 and VMMDevReq_HGCMCall, check that the data aren’t larger than VMMDEV_MAX_VMMDEVREQ_SIZE constant.
Using this, the IOCTL handler will catch any invalid request that could lead in memory consumption and eventually a DoS situation in the guest OS when Guest Additions software is installed.
Similar checks were also added in VBoxGuestDeviceControl() function located in src/VBox/Additions/WINNT/VBoxGuest/VBoxGuest.cpp like this: