Eyedropper won't select through outline anymore

I've been doing a gradient mesh based on a jpeg, but all of a sudden the eye dropper picks up the invisible gradient patches (I'm in outline mode) and even the color of the control handles!
I can't seem to find how to turn it off. Does anyone know?