Using git, how do I ignore a file in one branch but have it committed in another branch?

I've got a project that I'm deploying to heroku. The source code tree includes a bunch of mp3 files (the website will be for a recording project I was heavily involved with).

I'd like to put the source code for it up on github, but github has a 300 MB limit on their free accounts. I don't want to use 50 MB of my limit on a bunch of mp3 files. Obviously, I could add them to the

.gitignore

file to keep them out of my repo.

However, I deploy to heroku using

git push heroku

. The mp3 files must be present in the branch I push to heroku so that they get get deployed.

Ideally, I'd like to

.gitignore

the mp3 files in my local master branch so that when I push that to github, the mp3s are not included. Then I'd keep a local production branch that has the mp3s committed rather than ignored. To deploy, I would merge master into production, and then push the production branch to heroku.

Now I've got a branch with these files committed, as I want. However, when I switch back to my master branch, Foo.ignored is gone.

Anyone got any suggestions for a better way to set this up?

Edit: just to clarify, I want the mp3 files to be present in both branches so that when I run the site locally (using either branch) the site works. I just want the files ignored in one branch so when I push to github they are not pushed as well. Usually .gitignore works well for this kind of thing (i.e. keeping a local copy of a file that does not get included in a push to a remote), but when I switch to the branch with the files checked in, and then back to the branch with the files ignored, the files vanish.