Even if you removed it while it was being read, which it's not when you're playing a game, you'd just get an error or the application you were running would crash. It wouldn't physically hurt anything.

removing a cd/dvd/sd/hd from a console/pc should never damage it physicaly
removing while reading can confuse/crash the reading app and cause corrupt files written on other media by that app
removing while writing will most likely corrupt the file being written and might corrupt the whole filesystem on that media