Solar Power - Photovoltiac cell confusion

I have been trying to understand how PV cells work in solar panels and have become a bit confused, here is what I have read so far:

The PV cell is basically a PN junction diode with a conductive connection on the P and N material between this connection there is a load, a lamp for example, the Photon from the light strikes the PV cell and causes a electron to form in the N material and a hole to form in the P material, the electron wants to get to the hole so it goes through the conductive material which is joining the two P and N materials, through the load and into the P material where it joins together with the hole.

What I dont understand is, there are many electrons in the N material and many holes in the P material, what is stopping these from meeting, I mean the only combination of holes and electrons seems to be coming from the photons and not the electrons and holes which were formed from the doping, I would have thought that if you connected a wire from the P side to the N side all the electrons and holes would meet and cause the cell or diode to change back to its initial semiconductor state.