Nothing unexpected there, only the performance loss/gain is more than it should be due to the cache issues causing thrashing. Dedicate the caches to just one core and suddenly the cache thrashing isn't so bad.