Introduction:
I would like to start by saying that although I have only tested this method downgrading from 3.1.3 to 3.1.2, I would assume it works for any firmware and any device.
This downgrade doesn't require you to do anything to your iPod, such as go into recovery or DFU mode, its just a simple RESTORE button

Requirements:

an iDevice

Mac OSX (iPhone SDK/xCode is for OSX only, sadly. If you are running a windows look down at the RED notes.)

The Firmware file you would like to downgrade to

xCode (if it works on Windows it would be the Provisioning Application)

One last note:
Before I get into what the exact process is, please let me apologize in advanc for not knowing too much about this process because I kind of just stumbled upon it and decided to test and see what happens.

Tutorial:

1. First, open xCode, and go to Window -> Organizer

2. If you haven't already, set your device to be used for development

3. On the Software Version drop down box, it should show the current firmware version you are on. I have once located my 3.1.2 firmware file, but you must click the drop down box and choose "Other Version..." and located the IPSW file.

4. So after locating a lower firmware file (I am using 3.1.2 in this case), a note should show up on the bottom of the dropdown box saying that your device version will change after your next restore. You can now press the RESTORE button next to the dropdown box.

Conclusion:
So there you have it. The easiest way ever to downgrade your device. I am not quite sure of the exact requirements, or if this even works anymore for any device besides the iPod Touch 2G.. so if you guys could help me out with that I'd be glad to update this tutorial to be more exact with the requirements.

Hope this has helped you guys out (I know it helped me out, as I want to start developing iPhone apps =P)

[*]Also... if you have a windows pc ONLY, and want to try this.. I cannot say much more than Google: VMWare, and check your local torrent/w4r3z forum for an Image.

Try to get some of the new bootrom guys to give it a shot as well. I don't think it will work though on new bootrom devices because one of the deva here probably would have found it first. But if not, you are about to become famous.

Meh, I doubt it.. just a thought that occured to me, either way I'm sure if/when Apple sees this thread, or hears about it (provided it works) they will stop it from working.. Also, I'm sure by the time I get this working and people figure it out, Geohot or Chronic Dev will release something. :P
I figured it was worth a shot.. although I'm sure the usual noob who comes and complains will pop up in this thread sooner or later, haha.

Almost set up here.. my 16.5GB download is finally almost over, after 4 hours.. so let's hope for the best.

Sorry for the double post guys - Bad news.
Mac is not working for me at the moment (trying via VMWare) so I am just gonna hit the hay for the night. I will get back on this as soon as I wake up, and I DO expect results.

That sucks, I really hope maybe I have some better luck... I'll let you guys know how it goes for me, if I can even get in to OS X.. it's not cooperating at the moment.

Update:
It almost works, but same **** as Windows at the moment.. "The device isn't eligible for the requested build."
Maybe I can spoof the Hosts file like on windows.. anyways I'm out for now guys if anyone has any luck post here, or I will when I try again later.

The fact you said you got a "Not eligible for request build" shows that it's doing nothing different than normal, because it can't.

Firmwares are incomplete on their own. The SHSH that apple gives you completes it and makes it unique to your device, and since that can't be recreated, you're with the rest of us, unable to downgrade.

Yep, you are right man. I just thought it might work since it worked on the iPod, but I figured that it might not.
Just figured I'd post this here as a ground for anyone who wants to experiment with it and maybe find some way to completely cut communications between iTunes and Apple and/or spoof an SHSH string using the 3.1.3 ECID in order to make a 3.1.2 SHSH.