You might want to look into the sources of the gameboy emulator, look into sys/fbdev/fbdev.c. It does a 5-step ordered dither via using a lookup table for 8bpp grayscale values. Should be easy to port into an mplayer fbdev output modification.