(1) if your CPU is multicore, you will benefit a lot from running garbage collection in parallel with the -Xincgc flag (incremental garbage collection).

(2) disable explicit calls to the garbage collection, which may result in very long pauses. Simply add the -XX:+DisableExplicitGC flag.

See below on how to set such flags for each operating system.

A note for plugin developers: it is a current myth that calling System.gc() in java applications has any effect on making more memory available to the program. Such effect finished with java version 1.2, nearly 10 years ago. Without calling System.gc(), when there are no more pointers to a specific chunk of memory, any new memory allocations that require a larger memory than is currently reported by IJ.maxMemory() - IJ.currentMemory() will in fact use unclaimed memory regions which are still counted within IJ.currentMemory(). So the reading of IJ.currentMemory() is MISLEADING.

In addition, calling Runtime.runFinalization() will incur in a severe performance drag, bringing ImageJ to a halt until the method returns; in the case of having loaded plenty of images which are no longer referenced to, such halt can be several seconds or even minutes. There is no need whatsoever to call such method, since automatic garbage collection will take proper care to finalize objects when necessary.

All your plugin should care about regarding memory is to not keep pointers to chunks of memory, such as images, that are no longer needed. If you really need to load a lot of images, consider storing their file paths only, and reload them on demand without ever keeping pointers to many images at the same time. The VirtualStack classes will do this for you automatically.

When using VirtualStack, every time you call getProcessor() on it the image file behind that processor will be loaded; it is advisable then to store the ImageProcessor pointer temporarily to avoid repeated calls to getProcessor(), and work with the pixels array directly. When moving to the next slice of the virtual stack and thus the next call to getProcessor(), just release or overwrite the pointer to the previous one.

Avoid swapping: all modern operating systems, and particularly MacOSX, have a tendency to swap a lot. If you are using ImageJ heavily such as in working with lots of images at the same time, refrain from switching to another application while doing so.

Use the newest java available: when it comes to imaging, java 1.6.0 is a huge improvement over any other java versions. The internal graphics engine has been nearly fully rewritten to automatically hardware-accelerate many imaging tasks.
Be sure to check the menu “Plugins - Utilities - ImageJ Properties…” to check which “java.version” you are using. If it's 1.5.0 or lower, upgrade the java system and make sure that the ImageJ launcher script is using the newly downloaded one by checking the ImageJ Properties again.

Memory: don't use ever more than 70% of what the machine has. Otherwise, paging may kick in rather quickly and bring ImageJ to a near halt.
How to adjust memory: from within ImageJ, go to the menu “Edit - Options - Memory and Threads” … and set the number of megabytes to 70% of your current RAM. You may also set the value directly on the VMOptions key in Info.plist (see below).
How to find out how much RAM your system has: click on the apple menu and choose “About this mac”. In the small window that opens, the amount of memory is listed, in either gigabytes (Gb) or megabytes (Mb).
Also: be aware that most MacOSX are 32-bit, and thus the java virtual machine supports up to 1700 megabytes of memory only.
As of December 19, 2007, there is a java 1.6.0 64-bit package for MacOSX 10.5 Leopard ONLY, which is downloadable from Apple Developer Connection website. This release will enable you to use more than just 1.7 Gb of maximum memory.

Editing the ImageJ application preferences file:
To set the above options, control-click on the ImageJ icon in the Finder, select “Show package contents” and edit the Info.plist file. Search at the bottom of the file for the key “VMOptions” and add it there, after any -Xmx statement.
For example, for a machine with 2Gb of RAM:

Memory: the memory setting is automatically set for you when launching ImageJ, and is already optimized for your system. Like in MacOSX, manually setting the maximum memory flag -Xmx to a value over 70% of the total available memory will result in excessive swapping and thus slow ImageJ execution.