My Gigabyte GTX 460 1GB's core and memory clocks lower to 50 MHz/135 MHz respectively when I'm idling at my desktop. The problem is that it makes my whole desktop feel slow, for example scrolling is sloppy as is resizing windows and some other things.
When I keep scrolling or messing around for a few seconds, it switches to faster clocks and everything is great for around 10-15 seconds.

I'm fine with PowerMizer lowering the clock speeds, but not so low that it slows down 2D stuff that badly. I used a GTS 250 before I got the GTX 460 and it clocked to around 2/3 of its native speed and it was completly unnoticable.

Is there any way to have PowerMizer's power saving enabled but bump the lowest performance level up a bit?

With the tools in nvidia-settings it is impossible atm for GTX 4xx cards. I don't know about nvclock but you could look into it. It is possible however that later driver versions will support Coolbits again for this series.

Another option would be to flash your card's BIOS. I did it with my GTX 470 for 3D clocks (607/3348 --> 710/3400) with Nibitor 5.8 (Windows tool, but works in Virtualbox and I think Wine too).

Thanks for your very helpful response, Ulukai. I did just that, use NiBiTor 5.8 to increase the lowest P-States clocks by going to Tools -> Fermi Clocks and changed all the 3-labeled row's values to those of the 7-labeled row. Flashed it using Gigabyte's @BIOS VGA tool (in Windows). Works great.