This happened to me once on one of my older o3ds's. The screw got broken while inside the 3ds and couldn't be unscrewed by any kind of screwdriver.
The only solution the tech found was to melt the plastic VERY CAREFULLY with a solder machine (and of course, breaking permanently that screw slot). But i had to do it for a motherboard switch...

Easiest way is to use a small tip drill. I used a dremel in the past on high settings and the screw head popped off. Just don't put too much downward pressure and just get the inside of the screw head to thin out and it pops off.