No antenna can perfectly focus electromagnetic waves. The parabolic reflector antenna has almost all of the energy in front of the reflector (towards the main beam). However, the rear lobe is still there - although the radiation pattern is often decreased by about 60 dB (so a millionth of the power goes behind it) relative to the main lobe.

The idea of the feed horn reflecting all the energy of straight line rays is just a mental model. In reality, the waves will have some bend around objects (diffraction). In general, the energy that leaves (feed) horn antenna that hits the edge of the parabolic reflector will detract and radiate behind the antenna. While the horn antenna focuses most of the energy at the reflector, it will produce some energy that hits the edge and ends up directed towards the backward direction.

So yes, even if you assume the reflector is perfectly conducting, you will still have backward radiation.