From what you are saying, you are getting consistent but different speeds when reading (copying) from RAM cache to your RAM buffer, depending on the electrical connection between the SD card and Arduino - which should not matter at all since the SPI bus is not used (and which actually shouldn't matter even if it was given the synchronous SPI bus, but that's another matter).