I'm trying to improve my matrices calculations, and I'm having trouble with figuring how to deal with quaternions.

I have an Orientation class which has Vector3s Position/Center/Scale, and a Quaternion for Rotation.

I've read everywhere of how quaternions are faster than matrix, and how I should hide the euler angles and use quaternions for rotations internally, and only use euler angles if I have to expose or if I have an interface, and that's why I implemented them, but I'm having trouble actually doing this.

This Orientation class is inherited to my SceneObject class, which is inherited by all "useable" objects in code.

I'm coding an engine not just for myself, but for other programmers as well, so I'm trying to get the classes they'll use as easy and fast as possible.

I wasn't able to get the proper values from the Quaternion to set my Camera Matrix, and these are the two main problems I'm having, I don't know to set up a Camera Matrix, and how to avoid using the euler angles and just stick to quaternion math.

For now I'm using the GLM's lookAt function, and I have a Quaternion for the camera rotation, and when I call Camera.RotateX() or Camera.RotateY(), I multiply the Vector3 cameraTarget and this rotation quaternion, but I think this could be better.

I'm calculating the rotation first, to get the vector, and then do the camera matrix calculation with the vector.

Isn't there a way I can keep track of the camera's values and generate/calculate everything only once?

And is there a way I can keep the rotation values exposed (it doesn't need to be euler angles) from the quaternion class and be able to calculate it only once?

Currently if I keep the Quaternion's SetRotation() function, I calculate a new quaternion and multiply it everytime it is called. The value could change due to various factors, and not straightly at once, in the same update tick, ex. changes once because of physics, another because of player input, and one more time because of an object's event. Instead of calculating 3 times, is there a way to do the calculation only once?

The easy response is, "Your math skills are lacking. You need to learn more math." Recall that up until a few decades ago it was only the esoteric mathematicians living in universities that ever touched this kind of thing. It has been simplified and made accessible to many game developers who use it without really understanding the math, just hoping that what they read online does what they want. To be successful in graphics you need to deeply understand the mathematics behind it.

I'll try to add more details.

I've read everywhere of how quaternions are faster than matrix, and how I should hide the euler angles and use quaternions for rotations internally, and only use euler angles if I have to expose or if I have an interface, and that's why I implemented them, but I'm having trouble actually doing this.

Euler angles, quaternions, and transformation matrices are all used for different things.

Euler angles have their uses and drawbacks. They are very nice when you need to show values to a user that they can understand. They can represent an orientation, but they suffer from problems. They are ambiguous (multiple values represent the same orientation), suffer from Gimbal lock, require additional mathematics to rotate smoothly, and so on.

Quaternions also have their uses and drawbacks. They are unambiguous and produce aesthetically pleasing rotations with little computation. They are easy to manipulate programatically. Unfortunately they are not very friendly for humans, and they can look odd to animators who are manipulating control points and curves.

Transformation matrices also have pros and cons. They represent far more than just an orientation. They also represent location, scale, and shear. They provide very easy access to the three basis vectors, and if you need to do any serious math those vectors can be incredibly useful. They can represent affine transformations like matrix-based scaling to produce a very compute-cheap squash and stretch, and they can represent non-affine transformations for special effects. They are critical to most of the rendering formulas. They suffer a major drawback, thanks to floating point error they quickly lose affine properties and cannot be used for numeric accumulation.

Quaternians are only 'faster' than a transformation matrix for some operations, where they only have four/three values compared to the matrix's sixteen. If you need to perform actions that convert between quats and a matrix, there will be a significant cost.

Don't pick a representation because you have been told it is faster. Chose the representation that gives the most correct results.

This Orientation class is inherited to my SceneObject class, which is inherited by all "useable" objects in code.

This does not make sense to me. When someone says "Give me a quaternion" you hand them a SceneObject.

I can understand that a scene object has an orientation. A function like "GetOrientationAsQuaternion()" that returns a quaternion might sense. There is even value in having a family of functions that convert between all three orientation types.

Use composition, not inheritance.

I wasn't able to get the proper values from the Quaternion to set my Camera Matrix, and these are the two main problems I'm having, I don't know to set up a Camera Matrix, and how to avoid using the euler angles and just stick to quaternion math.

The formulas are readily available through your favorite search engine. Note that you cannot "just stick to quaternion math" because certain uses require different formats.

For now I'm using the GLM's lookAt function, and I have a Quaternion for the camera rotation, and when I call Camera.RotateX() or Camera.RotateY(), I multiply the Vector3 cameraTarget and this rotation quaternion, but I think this could be better. I'm calculating the rotation first, to get the vector, and then do the camera matrix calculation with the vector. Isn't there a way I can keep track of the camera's values and generate/calculate everything only once?

And is there a way I can keep the rotation values exposed (it doesn't need to be euler angles) from the quaternion class and be able to calculate it only once?

Possibly, but do what makes the most sense for your game.

Since you are using LookAt style functions, I'll assume this means you have some form of follow camera. This type of camera doesn't really need to store the orientation. Instead the correct value can be trivially computed directly whenever either the camera position moves or the look at point moves.

If you chose to store the orientation as a quaternion, then do so. To make axis-based rotations, create a quaternion that encodes the desired rotation and add it. Your quaternion class can easily have constructor functions that accpet Euler angles. It can also have a function that converts a quat into a set of Eulers. (For hopefully obvious reasons it will need to pick just one of the infinitely many combinations of Euler angles that could represent it.) And it could have some convenience functions like RotateX() or RotateY() that create the quat and do the addition. This kind of thing generally is done in game object processing, not an area of code that is a bottleneck.

Currently if I keep the Quaternion's SetRotation() function, I calculate a new quaternion and multiply it everytime it is called. The value could change due to various factors, and not straightly at once, in the same update tick, ex. changes once because of physics, another because of player input, and one more time because of an object's event. Instead of calculating 3 times, is there a way to do the calculation only once?

If you don't want it to change three times, don't change it three times. But if you do need to change it three times, then do so.

If this is a follow camera, it doesn't need to be changed three times. Bump the location because physics says so or because an object's event says so. Bump the camera angle or information because the player says so. After each change mark the camera info as dirty. The next time you actually need to use the value for rendering it will need to be recomputed (because it is dirty) but computing a camera LookAt matrix is a fairly trivial operation.

If it isn't a follow camera, then yes, you may need to bump it three times. So what? That isn't going to be a bottleneck, so don't worry about it. Do the work you need to do.

I agree that my math skills are terribly lacking, I only started to pick it up (besides the 4 basic operations) after I started this engine, but even now that I'm a little better than before, when I google something and I find only the mathematical formula with all those abbreviations, I still have a really hard time understanding it. When I see a programming implementation I find it much much easier, like "oh so you just add this and multiply that!"

I thought I always had to store the rotation in the Quaternion, convert to Matrix, multiply, and get the resulting matrix, but I searched for some quaternion specific operations and it seems I don't really need to convert it back to matrix all the time!

I was managing my performance obsession until I started to add more objects and text on screen, and reading LSpiro's post about deferred rendering/lightning and Phong shading, I thought "wow, so that model with shading effects run at 2000 fps, how am I getting 3000~ with only one image and a text?" (and my PC is about 1y old only)

So I think there's lots of room for improvement, but if you say this won't be much of a bottleneck... I'll keep going improving bits by bits as I notice the issues.