I'm having some issues with trying to do some perspective textureing, i've been working on the problem for a few days and i'm doing something daft and cant work out what it is so i guess here i'm just after conformation that i'm indeed not going bonkers !

Ok, so heres the scene.

i have a frame of video for which i know all the camera parameters for the scene such that i can draw things in the right perspective for the frame of video on the video.
I can draw a rectangle in the image (2d Screen space) and i assume that the bottom of the rectangle i have drawn is on the ground plane meaning i can turn the screen space drawn rectangle into a 3D billboard. I save the bit of video within the region i have drawn as a texture.

So at this point just to clarify i have a bit of texture obtained from the video and a 3D quad that corresponds to my bit of texture.

I want to draw the texture with the correct perspective onto my 3D quad such that it matches the underlying video, i'm doing something daft here though as when i try this my texture sprites don't quite match the underlying video they were generated from when i draw them on the quads.

heres a bit of psuedo code of what i'm doing.

glMatrixMode(GL_TEXTURE);
glLoadIdentity();
glTranslated(to the centre of my texture in texture coords);
glScaled(to scale & bias my texture to its correct size)
gluPerspective(using my camera parameters that i know when the texture was generated);
gluLookAt(using my cameras position, look at point and up vector)

i then draw my quad with the 3D coordinates for both the quad and the texture coordinates.

If anyone has a clue about what i'm doing or trying to do and can either point me in some direction for further info or tell me what bonkers thing i'm doing wrong then i would be very greatful as this is makeing me crazy !