The distinction is hard to make in a verbal context, because so often we don't fully enunciate our words and just rely on our listeners to understand what is said, so "could've" is almost exactly the same as "could of". Which I believe is where the problem arises, especially in a society where true literacy is nearly gone, and butchering grammar is perfectly acceptable, "because only rich snobs care about such nonsense."

Honestly, I don't think I mind it in speech. Besides, there are so many different pronunciations that it doesn't matter. In the very city in which I live, people from different suburbs have starkly different accents!

Unless purposefully used to illustrate a character's dialect (in which case anything goes), it really throws me off. (And I have English as second language) What makes it even worse is that only English-as-first writers seem to make the mistake.