Where, exactly, are you seeing this 4% value? It's likely not telling you what you think it is. Just paying attention to how much RAM the foreground application is using tells you very little, since it's the background rendering processes that are consuming most of the RAM when you're rendering.

Also, as Dave says, the goal is not to use all of your RAM; the goal is to render fast and well. If RAM isn't the bottleneck (e.g., because you're rendering something that is computationally expensive and thus the CPU is what's being taxed), then some other part of your system is working flat out while your RAM is not much needed.