I am experiencing the same behaviour with Vista x64 and version 2.0.2 and 2.0.4, so it's not the same as #2060, since that was solved starting from 2.0.2.

I went from Vista x86 to Vista x64, and have observed the behavior in both cases. It happens erratically, I haven't found a pattern yet, but always during intensive disk activity.

My vdi is 230 GB, 60 of which are used, the rest is empty.

I'm trying to compact the vdi file, but keep stumbling on this bug.
When I use sdelete -c to fill the unused space with zeros, it happens very often, I'd say 8 out of 10 times. It happens also when compacting the vdi file with the virtual machine off but the virtual disk manager working. So far I haven't managed to successfully complete the sdelete and the compact phase in a row, so I constantly go back to a backup copy of my vdi and start over. That should tell you it's all but a rare behaviour, at least on my system.

Try creating a large vdi (at least 250GB) and using sdelete -c on it. Unless the bug is somewhat exploited by something specific in my sistem and antonxx's, you should be able to reproduce it.

But again, sometimes sdelete goes to 100% without the RAM usage moving, sometimes it starts going up immediately. One thing though is for sure: when the RAM usage starts increasing, it doesn't stop. The VBox processes in the task manager show normal RAM usage even when the total physical RAM used is skyrocketing. As soon as VBoxSvc is stopped, either by closing VirtualBox or by terminating it, the RAM is released back to system.

When the odd behaviour starts, it takes about 30-40 seconds to go from 25% of physical RAM used to 100% on a 8GB system.

One thing I forgot to mention: I think it's important that the vdi file is actually very large on the disk, not just an autoresizing vdi with a large maximum size but a small disk footprint. My vdi has a maximum size od 230GB, and an actual size of 212GB, because it was full of files that got deleted, 60GB o which are left.

So if you want to try and reproduce the bug, I suggest you copy a very large file (>250GB) inside the volume. During the copy the bug could already show up. If it doesn't, delete the file. That should leave you with a vdi that is larger than 250GB on disk. At that point use sdelete -c on it. If the bug is still not showing up, try and compact the vdi file. On my system that procedure is 100% guaranteed to make the odd behaviour appear several times, preventing the procedure from completing successfully.

alephzain, which version 2.0.4 are you referring to? This version is still not released and there are even no nonofficial 2.0.4 builds. But what is your exact problem with 2.0.2? What is your guest? And where do you execute sdelete? On the .vdi file or within the guest (I assume the latter but please be more specific)? And please add a VBox.log file of such a session to this defect.

alephzain, which version 2.0.4 are you referring to? This version is still not released and there are even no nonofficial 2.0.4 builds. But what is your exact problem with 2.0.2? What is your guest? And where do you execute sdelete? On the .vdi file or within the guest (I assume the latter but please be more specific)? And please add a VBox.log file of such a session to this defect.

The version 2.0.4 I am referring to is the one at #2212 comment 35.
I reopened this ticket, so the problem I'm experiencing is "Virtual disk manager uses all ram on vista x64".
I run sdelete -c inside the guest operating system, on the disk corresponding to the .vdi file I was referring to. I've observed this problem both with Vista x86 and XP x86 guests.
I don't have logs anymore, because I was forced to reinstall my host from scratch and I've lost them. And now it's a lot more difficult to reproduce the previous behavior (see below).

I did. That's what I meant when I said it does the same in version 2.04 as well.

Update: I have a P35 chipset based system, and I had Intel Matrix Storage Manager 8.5 installed. Now I have configured my host system to run in AHCI mode instead of RAID and I'm using Microsoft AHCI 1.0 driver instead of Intel's and the memory leak is gone or a lot harder to reproduce. I haven't observed it once in nine attempts.
The memory leak was still in VirtualBox, because all the memory was freed when closing the VBoxSvc.exe process. Also, I didn't have any memory leak if I ran sdelete on a host's disk directly. It seems Intel drivers have to do with it as well though.
Personally I'm fine with the new AHCI setup, so this problem is not affecting me anymore. I'd still suggest, if any of the developers can, to try the steps above on a P35 chipset based system configured in RAID with Intel drivers and a Vista host, since I bet someone else can be affected.