I have been using AE for many years now but still seem to bump in to strange behaviour that I can't explain once in a while.

This time it's about a composition that has several 3d-layers (shape layers) but no camera. When I try to scale the 3d-layers to make them larger by dragging a corner with selection tool, it actually shrinks the layer. It feels like everything is working in the opposite way it should. I also have spread the layers in z-space and for some reason layers with a bigger z-value seem to be _closer_ to the screen when usually positive values on z-axis will move the object further away and negative values closer to the camera. Is this simply happening because I don't have a camera in the scene or what could be causing this confusion?

If you have offset the anchorpoint Z value to a negative number like -500, then you scale up the layer it will move away from the default camera. If the z offset is positive then the layer will move toward the camera as you scale the layer.

I think that's what you've seeing. Make sure that your anchor point Z value is 0 and normal behavior should return.