Math problem involved in culling

I was trying to work on the code for deciding what objects are in view of the camera and ran into this problem. Given a quad with known points and grid with cells of a known size, i'm trying to find the cells that are overlapped by the quad... it's just like in this diagram

Does anyone know a quick way to solve this? Thanks... i know it must be hard, but any help is appreciated

First of all, what I'm about to describe should be precisely equivalent to the job of drawing a polygon. You may be able to get OpenGL to do it for you, but you may as well work it out yourself because you'll need to refer to the results a bit and (I think?) if you drew it with OpenGL it'd reside in video memory, which you don't want to read from (it's slow).

So look for some old-skool 3d techniques pages, just for rendering a flat polygon in a rectangular grid of pixels (like a screen), and then use basically the same techniques. IIRC, it goes something like:

Label your grid squares with their own gridX, gridY co-ordinates. Find the quad-corner which lies in the quad with the highest gridY co-ord. That's the highest row of grid squares that can possibly seen. Then find the gridX extent of the quad at each gridY level:

Continue the line from the "top" quad point towards the next quad point anticlockwise, and keep track of what grid cells you touch. For each gridY value, keep a gridXstart (lowest gridX for that gridY position), and update it for each row you go through. Clockwise from the top point will give you gridXend values for each row (highest gridX for that gridY position).

Once you've got all the way to the gridY level specified by the lowest point, you have a start and end horizontal position for each row covered by the quad. When you are drawing a polygon, you would draw this information to the framebuffer; you may prefer to store the information in a more appropriate format for your purposes, and may even be able to use the raw start/end data for each height quite efficiently.

Regardless, the basic principle is the same: and some very clever people have worked on ways of drawing polys very fast indeed in software (particularly sincle you'll only need this once per frame). The algorithm I've detailed may not be perfect; I suggest you do some research and abstract the bits about calculating which pixels to draw for your purposes.

hth: if this sounds like gibberish, post again and I'll try to explain more carefully.

ps: you may want to substitute "bottom" for "top" and "up" for "down" in the above description, if your co-ordinate system is the right way up

pps: it also assumes the quad is convex, which it should be - if you want arbitarary shaped culling you could break the quad into triangles.

oh, and another thing: you might want to actually cull things outside a slightly-too-large polygon, because if an object is right at the edge of an uncovered cell it could theoretically be "in view" by overlapping into the way of the camera. Making the quad bigger'll mean you have to deal with a few more redundant objects, but you shouldn't get too much nasty popping at the edge of the screen.

I think I see what you mean... basically, go right to left on each grid line and find the end of the line (lowest gridX coord that is overlapped by the quad on that line only) for the entire quad? Sounds easy enough...

This probably isn't fast enough, but is it fast nonetheless? Are there any more complicated methods that are faster?

As I said, on an abstract level it's the same as rendering a quad. There may be faster ways of rendering a polygon; there may be faster ways of doing your precise task that are utterly different to the one I've suggested. My way shouldn't be too slow, though, unless your grid cells are truly microscopic, and I bet there'll be some clever optimisations to be made if you look hard enough .

As for the order you look at the cells in: what order is best will depend upon how you have stored your cells. Try to avoid having to jump around in memory too much and you should be fine.

Remember, OpenGL will do polygon culling for you. A simple test that'll eliminate most of the objects that are definitely out of view is probably preferable to a deep and involved solution that's unnecessarily accurate - and it'll certainly be quicker to implement.