I understand that to install a plugin, I need to put the dll in the plugins folder. But within the plugins folder there are two subfolders: x86-ansi and x86-unicode. What's the difference between them? What's the difference between the dll's in them? And how do I know which folder to put a new dll in?

In NSIS 2 all plug-ins are x86-ansi so there was no need for subfolders. When you download a 3rd-party plug-in from the Wiki and it only contains one .DLL then it is most likely a x86-ansi plug-in. Unicode plug-ins are usually in a folder named Unicode in the archive but some of them are named nameW.dll instead (and need to be renamed after you extract it to x86-unicode).

The naming scheme of these folders is %CPU_Target%-%Character_Encoding%.

You cannot tell just by looking at .DLL which type it is. If you put one in the wrong folder the installer will crash or not perform correctly when that plug-in is called. If you download Dependency Walker you can see if the plug-in imports lots of functions with either a A or W suffix. "lstrcpyA" vs "lstrcpyW" etc. (W for "Wide" character which is the UTF-16 LE Unicode encoding on Windows)