There are many manufacturer's web sites out there for lithium
polymer batteries, but very few places to actually buy them in
low volume. It seems however, that people are starting to use
thin polymer cells to power RC gliders and the
like. Interesting.
Skyborn
Electronics has batteries for sale; not the lowest price,
but if you are building your own MP3 player you probably are
not doing it to save money...

LCD

Earth LCD -
Surplus LCDs. This is where the little LCD shown in the
pictures is from. It is the Epson E1010 (look under Character
and Small Graphic LCDs). Format is 4x12 character with an LED
backlight, and it is really really cheap ($3).

AVR Projects - Related

yampp
- Originally I wanted to use this design pretty much
wholesale, including code. But with one thing or another I
ended up redoing the hardware design mostly from scratch
according the the VLSI datasheet. Also, Becky (who wrote the
FAT code) found that it was easier to implement from scratch
too. Oh well, we are not actually playing MP3s yet so the
yampp code might still come in handy. And at least the
existence of the yampp gave me the idea that this could be
done, and also indicated a good decoder to use.