For filling, in a small heavy saucepan, mix sugar and cornstarch. Whisk in eggnog. Cook and stir over medium heat until thickened and bubbly. Reduce heat to low; cook and stir 2 minutes longer. Remove from heat.

Divide mixture in half. In a small bowl, whisk cocoa and remaining water until blended; stir into half of the eggnog mixture. Stir rum extract into the remaining half. Refrigerate both mixtures, covered, until partially set.