Painting a virtual first-down line on a football field is an excellent example of this process; the concept of painting a first-down line across the field on people's TV screens certainly sounds simple. As it turns out, implementing this is incredibly complex. It takes a tractor-trailer rig of equipment, including eight computers and at least four people, to accomplish this task

Here are some of the problems that must be solved in order for this system to work:

The system has to know the orientation of the field with respect to the camera so that it can paint the first-down line with the correct perspective from that camera's point of view.

The system has to know, in that same perspective framework, exactly where every yard line is.

Given that the cameraperson can move the camera, the system has to be able to sense the camera's movement (tilt, pan, zoom, focus) and understand the perspective change that results from that movement.

Given that the camera can pan while viewing the field, the system has to be able to recalculate the perspective at a rate of 30 frames per second as the camera moves.

A football field is not flat -- it crests very gently in the middle to help rainwater run off. So the line calculated by the system has to appropriately follow the curve of the field.

A football game is filmed by multiple cameras at different places in the stadium, so the system has to do all of this work for several cameras.

The system has to be able to sense when players, referees or the ball crosses the first-down line so it does not paint the line right on top of them.

The system also has to be aware of superimposed graphics that the network might overlay on the scene.

A key piece of hardware used in the system is a special camera mount that holds the television cameras. This mount encodes all of the camera's movement (such as tilt, pan, zoom and focus). The data the mount produces helps the computers understand exactly what each camera is doing in real ­Another key piece is a computerized 3-D model of the field. The computers know exactly where the cameras are located in the 3-D model and can orient the virtual first-down line on the field accordingly. The model also accounts for things like the crest of the field and the location of the yard lines on the field.

Color palettes are also critical to the system. The computers must be able to distinguish between grass, on which the line should be painted, and everything else (players, referees, the ball, etc.), on which it should not. Color palettes solve this problem. You can see the palettes at work in this frame:

The player does not have the line painted over his jersey because of the color palettes.