Issue with 6DOF camera motion

Hi all, I'm hoping someone can help. I have a weird issue with a 6DOF camera I'm trying to implement. I'm trying to simulate flight in space, so the ship can rotate and travel in any direction. I'm using a Quatenion for rotation, I have a simple scene set up, my camera is at 0, 0, 0, I have a skybox with starfield as background and one object, a planet at 0, 0, -10, right in front of the camera, the planet rotates locally. I can pitch or yaw the camera and the scene rotates around perfectly in any direction, with no gimbal lock.

Now I want to introduce motion, and fly the camera anywhere and have it move in the direction it's pointing. The simulation of moving the camera is done by translating the object(s) (single object in this test), depending on the vector in which the camera is pointing. Which works, as long as yaw and pitch aren't going in opposite headings.

Let me try to explain, say I start and fly by the object to the left, so I yaw slightly left and fly by, and to turn around to again face the planet, I pitch 180 degrees by pulling back on the stick and again I'm flying towards it. This works as expected. Or say instead of the pitching the camera to face the planet, I yaw back around, this works as well.

Now say instead this time I fly by the planet to the left, again yaw to the left, then once past the planet, yaw back to face it and fly by the planet again, so now we're pointing in the opposite direction and when past the planet this time I pull back to pitch around 180 degrees to face it, but now the planet is moving in some weird direction, sideways to the camera for example. If instead I would have yawed the camera to face the planet again, and fly by the planet and then pitch to turn around, then all would be working ok

Hard to explain, basically it seems to work ok if when I yaw the ship and make sure to yaw it again pointing in the original direction before doing an opposing pitch to turn back. Or alternately, pitch and pitch back to the original heading before doing an opposing yaw to turn around. Hopefully this makes sense.

Here's my code, I'm on IOS and using GLkit libraries to simplify this code and get it working. This code runs every frame. Thanks for any help you can provide!

// _position is the original position of the planet, in this case 0, 0, -10, and we move it to the new position to simulate camera motion.
GLKVector3 newvec = GLKVector3Add(_position, translation_vector);
modelViewMatrix = GLKMatrix4Translate(modelViewMatrix, newvec.x, newvec.y, newvec.z);