Rigid Body Physics (OpenGL)

I am trying to build rigid body physics engine. I am using the glm library as my standard math library.The problems I am facing are as follows,

When I calculate torque with respect to force (see function calculateTorque()), the body gets distorted and skewed to the extent, where it rotates and increases in size and is not visible anymore. But when I remove the force, from the torque equation that is I just randomize (see calculateForces())the torque values it works fine.

When rotating, I think the body is skewing and increasing in size slightly, for both quaternions and matrices. (Not sure if the inertia calculation is right, the body is a cube)

Am also having problems with the collision response, the body gradually penetrates the surface, and it does not react properly with respect to the value of epsilon.

I cant figure out what is wrong, I really want this to work. I have been breaking my head for a week now, can't seem to pin point what the problem is exactly. If somebody figures something out, or notices something please let me know. It would really be appreciated. Always helps when more than one person is looking at it. Thank you, much appreciated.P.S: The AABB - Plane collision is implemented with the help of the book Real Time Collision Detection by Christer Ericson, and the Rigid Body physics is implemented with the help of Braff's implementation.