Well, its not 20", but my Brompton gets occasionally ridden on the local sustrans route along a canal.

The little wheels don't roll as well as 26"/700c obviously, so its a bit more tiring to keep the pace up.

I run my Brompton tyres (skinny 16" reinforced Schwalbe marathons) at over 80PSI, and after the three or so miles from the station i have had enough; vibration is nasty on the rough stone chip surface, which i dont think would be as bad on larger wheels or fatter & softer tyres.

So it doesnt seem to hurt the bike, but it isnt fun. Smaller tyre contact patch and the resulting grip is also one to watch out for, which becomes more of an issue on a mixed surface.