I doubt GLUT supports overlays on Mac OS X. Not exactly common functionality.

What do you mean by "overlay" in a web browser? AFAIK you'll be limited to writing a plug-in which is given a small rectangle of the page to draw in, and I know people often have trouble getting OpenGL to work here, though it's definitely possible.

I think you're outta luck. It's probably possible on Mac OS X by creating a child window of the browser window, making it transparent, drawing your OpenGL in there, manually passing events on to the main browser window, etc. Maybe the same kind of approach would work in Windows Vista under Aero, I don't know, but I think XP would be out of the question. Maybe the same kind of approach would work on X11 with Composite, but without it, I don't think it'd be possible.

Then your best bet (most flexible) is to create a browser with webkit and add the opengl above it. But you might be able to attach an external window to another, like a safari window, but I never did it, so I'm not sure.