I won't speak for others but owning a fast car is not always about speed.
One of the things that sold me on the 911 was the feeling I got just driving it. The whole experience along with that glued to the road feeling made me want it.

On the serious note. It's a common Direction that many m3 owners travel when purchasing a newer vehicle. M3 to Porsche is exactly what my next move will be. After I hit 40.