Yes, there is some truth, however, my experience is that I can not notice any difference. Here is what the "experts" say:

Install the larger configuration modules in the first bank and lower configuration modules in the second bank. In other words, the 2 x 4 GB modules should be installed in bank 1 and the 2 x 2 GB modules in bank 2. In pairs of course.

But as I stated above, I can't tell the difference (with my 2011 iMac). If you installed the modules opposite from what's recommended, don't worry about it.