Dump your RomFS and ExeFS using braindump. This can take a long time, so be patient. You don't need the ExeFS for custom music, so once you're done, you may delete that if you want.

Put your SD into your computer and extract it using CTRtool. There's a guide on how to use CTRtool to extract your romfs in the braindump thread, as well. While it's extracting, if you haven't already, you may want to use this time to obtain your nus3bank files.

Brawl custom music has the option to download the files fit for smash 3DS already, if you open the dropdown box that says "Download or add to template for" and select Super Smash Bros. for Nintendo 3DS, then select the new drop-down box that appears right below it, and select the song you want to replace. It will then give you a download with the proper filename. Even if you have a converter, this can be useful for figuring out which songs are which. You can then choose to download it as a NUS3BANK file.

After this, open the folder that contains the extracted RomFS and place the song in the sound/bgm folder. Build the RomFS using what ever tool you may be using.

Create a hans folder on the root of your SD card if one isn't already there/you haven't created one already, place the RomFS in there, and name it the last 8 digits of the title ID (this changes from region to region) .romfs, for example, the American version would be 000EDF00.romfs. If you aren't sure where to find this, it will display it in the Hans menu.

From here, put the SD back into your 3DS, open *hax 2.5, open Hans, change "Romfs -> SD" to YES and then start the game. You should now have your custom music.

NOTE: Hans doesn't load update data for some reason, so you won't be able to use your save file or play online with custom music. No way around this until some sort of work-around is found/made.

EDIT: as of the latest version of Hans you no longer need to strip the first 0x1000 bytes, so I've removed that step.

After this, open the folder that contains the extracted RomFS and place the song in the sound/bgm folder. Build the RomFS using what ever tool you may be using, then open it with your hex editor. From here, delete the first 0x1000 bytes (0x0-0xFFF). Yes, this means REMOVING them from the file altogether, not 00'ing them. Some hex editors warn you if you attempt this, just ignore it.

Well. I know that @shinyquagsire23 is known for datamining the update data for this game, so he must be using some tool to do it somehow.

Click to expand...

Private tools for decryption off SD (but the same can be done using Decrypt9 + xorpads), 3dstool + ctrtool and a dtls.py I modified and PR'd to crediar's repository. I've got the process almost automated now.