Your vectors aren't normalized. Normalize them first. That's required when using this equation.

Edit: (just to add an explanation for why the above matters)

To make it really clear, the dot product equation is:

A . B = ||A|| * ||B|| * cos(theta)

Where (A and B are your vectors, . is the dot product, and ||A|| means the magnitude of the vector A). In this form the equation, A and B are not required to be normalized (because the ||A|| * ||B|| part takes care of that... you'll see below).

Rearranging it, we see:

theta = acos((A . B) / (||A|| * ||B||))

Now, to normalize a vector A, you just do A / ||A||. I'll let An be the normlized version of A, where An = A / ||A||. Then you can plug it in as:

theta = acos(An . Bn)

Which is what you're trying to do, but your vectors aren't normalized, which is why you're getting problems.