From the author of

Keylight is useful when preservation of edge detail is of utmost importance—most typically (but not exclusively) when working with footage that was shot specifically to be keyed against a screen of either blue or green. The use of Keylight in After Effects CS4 is more or less identical to that of previous versions. The first decision in Keylight is most important: sampling a color for the Screen Colour setting. (Keylight reveals its UK origins at Framestore CFC with that u in word Colour.)

In the best-case scenario, you create any necessary garbage mattes and then follow these steps:

Use the Screen Colour eyedropper to sample a typical background pixel. View defaults to Final Result so you get a matte instantly. Set the background to a bright color and solo the plate layer, or examine the alpha channel by pressing Alt-4 (Mac: Option-4).

If in doubt about the Screen Colour setting, turn off the effect and set another instance, repeating as necessary until you have one that eliminates the maximum unwanted background. (Figure 1 shows the original image; Figure 2 and Figure 3 show two possible results.) Then you can delete the rest.

Now, as needed, look for areas to refine.

Figure 1 Even with a well-shot, high-definition source, it's imperative to get the Screen Colour setting right to preserve all of the transparent detail in the shot.

With the Info panel active, sample a pixel in After Effects, either using an eyedropper tool or simply by moving your cursor around a viewer. The Info panel then displays the value that belongs to the exact pixel underneath the selection point of the cursor (the lower-left corner of the eyedropper, the upper-left corner of the pointer).

As in Photoshop, the After Effects eyedropper samples only the color state of a pixel; transparency of any given pixel is always evaluated as if 100%.

Switch View to Status. Opaque pixels are displayed as white, transparent pixels black, and those containing any amount of transparency are gray (see Figure 4). It's an exaggerated view of the alpha channel matte.

Figure 4 Sample a background pixel near the center of the image for a good initial key, as seen in the Status matte. The black areas are already completely transparent, the white areas opaque, and the gray pixels constitute the area of focus; they're the semitransparent regions.

Still in Status view, you have the option to try Screen Balance at settings of 5.0, the default 50.0, and 95.0, although Keylight will preselect it based on your background color selection. This setting controls the weighting between the primary matte color (blue, green, or red) and each of the secondaries (see Figure 5).

Figure 5 This initial result, with a Screen Balance setting of 95 (top), seems in Status view to be inferior to a setting of 5 (bottom), but a look at the color channel shows the default 95 setting to yield a subtler initial matte edge.

If the background isn't solid black, you can boost Screen Gain until the gray mostly disappears in Status view, although ideally you shouldn't raise this value at all. Use this setting as sparingly as possible (see Figure 6).

Figure 6 A light touch is best with Screen Gain; some artists never raise it above the default value of 100, although settings up to 105 or 110 are often acceptable. Don't use it to remove anything like the noise in the lower-left corner of this figure, which is easily solved instead with a garbage matte.

NOTE

Keylight's built-in spill suppression can and will enhance the appearance of graininess, particularly with 4:2:2 or another compressed source. The workaround is to apply the keyed layer as a track matte instead.

Optionally, set the Despill Bias using its eyedropper. Sample an area of the foreground that has no spill and should remain looking as is—typically a bright and saturated skin tone area. (I very rarely do this.)

The rare perfect footage is now completely keyed. If not perfect (see Figure 7), this is a decision point; how can you best divide this matte into multiple passes?

Figure 7 Nice edges—it's the core of this matte that needs to be a little firmer.

The Apple Shake-based implementation of Keylight adds inputs for a garbage matte and holdout matte. This method of isolating and focusing on the edge inspired me to devise the following workflow to achieve the same in an After Effects pre-comp, with as few extra steps as possible.

Three-Pass Method

Keylight's Inside Mask and Outside Mask controls let you draw a mask, set it to None, and then select it as an Inside Mask or Outside Mask. The three-pass method is the same principle, but it should save you from extensive hand-animating. This breakdown will work with any software keyer, but it's especially helpful to get the most out of Keylight.

Create a pre-comp containing the Keylight layer created with the previous steps. (Enable Move All Attributes in the Pre-compose dialog box.) Rename the layer Edge.

Duplicate this layer (Keylight still applied), rename it Background, and solo it. This is the top layer. For clarity's sake, turn off effects for Edge for the moment.

The goal with this layer is to create a background matte that's 100% transparent and outside of all edge detail by several pixels. To create this layer, follow these steps:

In Keylight Status or Combined Matte view (depending on your preference), raise Clip Black just enough to remove gray pixels from the background in Status view, and then lower the Clip White setting to just above the same number (see Figure 8).

If necessary, choke this matte further to eliminate tracking markers and other small bits of noise. You can lower the Screen Shrink/Grow setting (under Screen Matte in Keylight) or apply a Simple Choker effect and raise the Choke Matte number. Be brutal, complete, and quick; this can be crude (see Figure 9).

Figure 8 Screen Gain can always be left at 100 for the edge matte (ideal) if the background is matted separately.

Whether or not you've choked the matte, spread it until it's well outside the foreground edge. The easiest way to do so is to apply Simple Choker and lower the Choke Matte setting to a negative value, quite possibly in the double digits.

Garbage matte any items in the background that haven't disappeared, hand animating the matte as needed.

Finally, switch to Final Result view and set the layer's blending mode to Stencil Alpha (see Figure 10). You should see a green (or blue) buffer around the foreground edge.

Figure 10 Look closely at the edges. The matte has been expanded so that it doesn't touch them, and the layer is set to Stencil Alpha so that the matte is passed through.

Duplicate the original Edge layer again, solo the duplicate, move it to the bottom of the layer stack and rename it Core.

The goal with this layer is to create a core matte that fills in all holes inside the matte edge without coming close to overlapping that edge. To create this layer, follow these steps:

In Keylight Status or Combined Matte view (depending on your preference), lower Clip White until the foreground is completely opaque. Raise Clip Black to a value just below the Clip White number.

If necessary, spread this matte to eliminate foreground holes. You can raise the Screen Shrink/Grow setting (under Screen Matte in Keylight) or apply a Simple Choker effect and lower the Choke Matte to a negative value (see Figure 11).

Figure 11 The core matte, like the background matte, is chewy but solid, and is choked so as not to touch the edges.

Whether or not you've spread the matte, choke it until it's well outside the foreground edge. The easiest way to do so is to apply Simple Choker and raise the Choke Matte setting, quite possibly to a double-digit value.

Garbage matte any remaining holes in the foreground, hand animating the matte as needed. These would be the result of shiny, reflective costumes or props.

Finally, switch to Final Result view and un-solo the layer. You have isolated the edge to the maximum extent possible to permit the most subtle possible key (see Figure 12).

Figure 12 Thanks to the combination of Stencil Alpha for the background matte and Silhouette Alpha for the foreground, only the crucial part of the matte—the edge—remains. The core can be switched back to Normal blending for the final version.

Now you can work just on that edge layer—starting over with its key, if you want. Your goal should be to use three controls only to refine it: Screen Colour, Clip Black, and Clip White. If possible, avoid adjusting Screen Gain.

This isn't the furthest you can go to break down a color-keyed matte, but problematic footage requires further steps.