As additional perturbative degrees of freedom, it is known that magnetic fields of inflationary origin can source curvature perturbations on super-Hubble scales. By requiring the magnetic generated curvature to remain smaller than its inflationary adiabatic counterpart during inflation and reheating, we derive new constraints on the maximal field value today, the reheating energy scale and its equation of state parameter. These bounds end up being stronger by a few order of magnitude than those associated with a possible backreaction of the magnetic field onto the background. Our results are readily applicable to any slow-roll single field inflationary models and any magnetic field having its energy density scaling as aγ during inflation. As an illustrative example, massive inflation is found to remain compatible with a magnetic field today B0 = 5 × 10−15 G for some values of γ only if a matter dominated reheating takes place at energies larger than 105 GeV. Conversely, assuming γ = −1, massive inflation followed by a matter dominated reheating cannot explain large scale magnetic fields larger than 10−20 G today.