Use git non invasively

I have a somewhat nasty build system that will choke as soon as it finds any unexpected files or directories. I'd like to add version control with git now without any change in the build process. (I can not/do not want to change this at the moment.)

Problem: Git will also leave some files/directories in that tree.

Is there a way to have git store any of its file externally and not put anything into the version controlled directories?