Rotoscoping introduction and resources

Rotoscoping (or just roto in
casual usage) is the drawing or painting on frames of a movie, using
visual elements in the movie as a reference. A common kind of rotoscoping
is tracing a path around an object in a movie and using that path
as a mask to separate the object from its background. This allows
you to work with the object and the background separately, so you
can do things like apply different effects to the object than to
its background or replace the background.

Note: If a background or foreground object is a consistent, distinct
color, you can use color keying instead of rotoscoping to remove
the background or object. If the footage was shot with color keying
in mind, color keying is much easier than rotoscoping. (See Keying introduction and resources.)

Rotoscoping in After Effects is mostly a matter of drawing masks,
animating the mask path, and then using these masks to define a
matte. Many additional tasks and techniques make this job easier,
such as using motion tracking on the object before you begin drawing
masks, and then using the motion tracking data to make a mask or
matte automatically follow the object.

Rotoscoping tips

Immediately after beginning to draw a mask, press
Alt+Shift+M (Windows) or Option+Shift+M (Mac OS) to turn on keyframing
for that mask and set a keyframe. This way, you won’t edit a mask
frame-by-frame for several minutes (or longer) and then realize
that you lost all of your work on previous frames because you forgot
to click the stopwatch to make the mask shape animated.

Draw your masks on a white solid layer with its Video (eyeball)
switch off, above the (locked) footage layer. This way, you run
no risk of accidentally moving the footage layer when you manipulate
the mask, and you can also much more easily apply tracking data
to the mask. (You apply the tracking data to the invisible solid
layer that holds the mask.) This also means that you don't lose
your cached RAM preview frames each time you manipulate the mask.
(See Toggle visibility or influence of a layer or property group and Lock or unlock a layer.)

When possible, transform (rotate, scale, move) the whole
mask or a subset of the mask vertices instead of moving the vertices
individually. This is both for efficiency and to avoid the chatter
that comes from inconsistent movement across frames. (See Move vertices in free-transform mode.)

Manual motion tracking is less time-consuming than manual
rotoscoping. The more effort you spend getting good tracking data
for various parts of your scene and object, the less time you'll
spend drawing and fine-tuning masks. (See Tracking and stabilizing motion.)

Online resources about rotoscoping

Pete O'Connell provides excerpts on his website from his Advanced Rotoscoping Techniques for After Effects DVD.
This tutorial series shows in detail how to rotoscope with After
Effects masks and motion tracking to separate foreground elements
from the background in preparation for compositing work.

Sean Kennedy provides several good tutorials on the SimplyCG
website, including some for rotoscoping in After Effects. Sean maintains
an index of these tutorials on his website.

Scott Squires provides a pair of movies on his Effects Corner
website that show how to rotoscope, both painting and masking:

Chris and Trish Meyer provide some tips on animating masks, including
using Smart Mask Interpolation, on the ProVideo Coalition website.

Alejandro Pérez provides a script on the AE Enhancers forum with which you
can use tracking data to position individual mask vertices.

Mathias Möhl provides the KeyTweak script on his website, with which you can modify
many keyframes on a property simultaneously. With KeyTweak, you
can modify a few keyframes manually, and the script modifies the
remaining keyframes in between accordingly. KeyTweak is especially
useful for Mask Path keyframes in a rotoscoping workflow.