Graphene's high mobility and Fermi velocity, combined with its constant light absorption in the visible to far-infrared range, make it an ideal material to fabricate high-speed and ultra-broadband photodetectors. However, the precise mechanism of photodetection is still debated. Here, we report wavelength and polarization dependent measurements of metal-graphene-metal photodetectors. This allows us to quantify and control the relative contributions of both photo-thermo- and photoelectric effects, both contributing to the overall photoresponse. This paves the way for a more efficient photodetector design for ultra-fast operating speeds.