Instead of PREFIX, you can use INSTALL_BASE, which I believe is now the perferred option for Module::Build and ExtUtils::MakeMaker (so long as you have up-to-date versions of both). I haven't had a problem with this, altho I haven't installed all of CPAN with it.

You still run into the problem of setting your @INC, since things seem to randomly install in a subdirectory called either 'site_perl', 'perl5' or '5.x.x', meaning you have to add 3 entries to your @INC. This can be annoying if your paths can change depending on the box you're working on.