Double click on LargeSystemCache and set to 0 to reduce the amount of memory used for file caching.

Click OK

Close the registry editor

If you start the Network control panel applet and select the Services tab you can select Server and click Properties. Select "Maximize Throughput for Network Applications" to use less memory (this actually sets LargeSystemCache to 0).

System internals have released CacheSet (http://www.sysinternals.com) which allows you to more specifically set memory used for caching.