AFAIK, NASM does not generate 64-bit code on Linux x86-64 target.
This makes some nasm-compiled object files could not link with
the encoder. mppenc's speed falls from ~20x to ~16x
on my Athlon 64 3000+(S939) with profile "standard".

From what I have heard, YASM, a replacement of NASM, could generate 64-bit code.
Currently it uses an extension of NASM syntax to support x86-64 assembly.
IMO those SSE registers do not change much. With a little modification,
is it possible to go with YASM?

The old NASM code in mppenc is related to 3dnow! and is disabled since it doesn't affect speed. You need to manually change the source code to enable it. Have you done that? If not, your speed decrease may be due to compiling as 64-bit with "optimizations" which actually slow things down a lot.

Let me re-phrase what Shy said.
Mppenc's assembly code (fft4gasm.nas) contains 3DNow-optimized versions of some of functions, it's deprecated and does not appear to give significant improvement on current CPUs, at least with Microsoft compilers. Fft4gasm.nas will be most likely dropped from sourcecode package in next release.
Additionally, source code needs to be modified for functions from assembly module to be actually called. If you compile current source even with assembly module included, assembly functions are never called, unless you alter relevant line of fft4g.c.