Hosts belonging to the same species suffer dramatically different impacts from their natural enemies. This has been explained by host neighbourhood, that is, by surrounding host-species diversity or spatial separation between hosts. However, even spatially neighbouring hosts may be separated by many million years of evolutionary history, potentially reducing the establishment of natural enemies and their impact. We tested whether phylogenetic isolation of oak hosts from neighbouring trees within a forest canopy reduces phytophagy. We found that an increase in phylogenetic isolation by 100 million years corresponded to a 10-fold decline in phytophagy. This was not due to poorer living conditions for phytophages on phylogenetically isolated oaks. Neither species diversity of neighbouring trees nor spatial distance to the closest oak affected phytophagy. We suggest that reduced pressure by natural enemies is a major advantage for individuals within a host species that leave their ancestral niche and grow among distantly related species.