You can't tell a program to use less memory than it actually needs and expect it to work, so Windows has to give out requested memory if it is available, so there can be no software which can limit other software in the way you suggest.

However, Azureus and Java do have options for controlling the amount of memory they use - have a look at Options/Files/Performance Options. There are some notes about memory usage there.

If that doesn't help the only other option is to buy more memory.

.

Political correctness is based on the principle that it's possible to pick up a turd by the clean end.