I want to simulate isometric engines in 3d. I know how to rotate the map by using trigonometric functions (thanks to somebody here). Now I want to know how to determine the correct axis to modify to simulate map scroll up/down/left/right, like the UFO Alien Invasion engine.

Not that I've done anything like this, but I'd assume it would be just like strafing left and right with a camera, and moving it up and down

(BTW, by up and down do you mean +y/-y or +z/-z)?

To move up and down you'd just need to modify the y axis.

You'll need a vector that points straight into the scene (direction vector) to move forwards and backwards, and you'll need either a vector that points left or right (a leftVector or a rightVector). A leftVector is not a vector in the negative x direction, a rightVector is not a vector pointing in the positive direction and a direction vector is not one that points down the z axis.

To calculate the direction vector you'll need a point that the map is at (the currentPosition of the map), and any point that is straight down the scene (this will be your lookAt point), then subtract lookAt from currentPosition and you'll have your direction vector which you then should normalize.

To get the right vector, you take the cross product between the up vector and the direction vector and then you'll have the right vector.

now to move left (using the right vector) you set an amountToMove variable that will say how much to move right. You then

I'm not really sure if you are working in 3 axes or 2 axes. I assume you're working with 3D coordinates and trying to get a Final Fantasy Tactics type of engine going, If not, then of course all the above wont work as it is.

Im working in 3d. As I said, rotating the map (moving the camera in a circular trajectotry around a point) is already implemented. But I think that for an RPG, that fixed point of view isnt adequate. I want to freely scroll the map so the player can explore visible areas beyond the current screen limits.

once you have the camera setup so that it looks down on the map in the angle you want all you have to do is translate the cameras position on the x-z plane. you could additionally alter it's height based on the maps height at the cam's position

I've done something like this before. You have to do a calculation, when translating, that involves the current rotation. Meaning that if you translate "left", you have to figure out the rotation, because that might not just be -x. I'll give you a clue, use sin, cos and the two axis that you're translating on.