I want to learn assembly

The problem isn't with learning it, it's with finding out which one to learn.

Assembly differs according to your CPU architecture, all right. Most PC CPUs are built with the x86 architecture? Fine.

From my understanding, there's still a multitude of different versions. NASM? MASM? What's the difference between these? How many more of these are there? Why are there more than one for one architecture? Which one should I learn?

Most CPU's in computers use x86. NASM and MASM and A86 are just different assemblers. If you've ever programmed in C, C++, Pascal, etc. before, it's like a compiler. There will be standard things on each one that will always work. The syntax for different assemblers should be pretty much the same. Naturally, each one will be a bit different. Dev-C++ and gcc are a bit different. At the core, though, they should be the same.
Even if they're more different than I expect, it will make the assembly code into something that the processor will execute. Whether you assemble it will NASM or MASM, it will do the same thing. Choose the one that's right for you and learn it.

Try using fasm , its open source , with much documentation.
Or if you would like to be near the processor , see the intel's developers manual , there you find the opcodes , so you wouldnt need to use any compile ,but write directly to file.