I am new to game development and I was trying to build a turret model, and control it through lua script in shiva3d. I can rotate it around the y axis which is no problem, but when it comes to pitching, I am not sure how to do that in a way which is independent of the Y rotation, for example,

in the default position, rotating with a positive value around X aims upwards, which is what I expect.

when I rotate it by 180 degrees around Y, the positive rotation around X aims downwards towards the ground instead of upwards.

on the other hand, if I rotate by 90 degrees, the turret rotates around it self as if its rolling...

I am not sure what to look for in order to understand how rotation in a 3d world should happen, so any pointers on that would be much appreciated

It sounds like you're doing the rotations in the wrong order. The pitch rotation should be applied first, then the yaw rotation. That way, the turret is always in "default position" when pitch is applied, so the pitch rotation will do the right thing. Then you apply the yaw rotation to get it pointing the right horizontal direction.