In recent years integrated waveguide devices have emerged as an attractive platform for scalable quantum tech- nologies. In contrast to earlier free-space investigations, one must consider additional effects induced by the media. In amorphous materials, spontaneous Raman scattered photons act as a noise source. In crystalline materials two-photon absorption (TPA) and free carrier absorption (FCA) are present at large intensities. While initial observations noted TPA affected experiments in integrated semiconductor devices, at present the nuanced roles of these processes in the quantum regime is unclear. Here, using single photons generated via spontaneous four-wave mixing (SFWM) in silicon, we experimentally demonstrate that cross-TPA (XTPA) between a classical pump beam and generated single photons imposes an intrinsic limit on heralded single photon generation, even in the single pair regime. Our newly developed model is in excellent agreement with experimental results.