Many people noted that both lenses are great and to either get both or one and then the other.
In terms of only one lens, here is the breakdown:
20mm Panny: 8 ppl
45 mm Oly: 10 ppl
14 mm Panny: 5 ppl

Some suggested waiting for the new 17mm Oly as well, and some also suggested the 9-18mm Oly as an alternative to both.

Major arguments for the 20mm Panny:

More versatile for general, all-day shooting,

Better for indoor close-range shooting.

45mm Oly:

Best for serious portraiters, though good for certain types of architecture shooting, etc.

Very sharp images and fast speed and great in low light (though not at close range).

So it seems that it comes down to whether I want a more specialized lens in addition to my kit lens for special shooting, or something to generally replace my kit lens for better quality everyday use.

I am still making my decision, but I may end up with the 45mm as a treat for myself, and deal with my kit lens for now until some more alternatives come out and then get a good pancake lens (either the 20mm or a newer model).