IndexWriteConfig ignored?

We limit the memory consumption of our IndexWriters by setting RAMBufferSizeMB to 5MB (IndexWriterConfig.setRAMBufferSizeMB). Inspecting a heapdump unveils that I still have writers wich consume/retain more than 35MB! How come? Any help/advice appreciated Clemens