Unless something has changed, you can't mix host architectures for distcc builds (which is what Xcode uses under the hood, or something similar at least) because some of GCC's precomputed data that is shared between distcc workers is actually a serialized segment of RAM that isn't byte-ordering safe.

OP is absolutely right. distcc from Xcode (or from the commandline xcodebuild) are totally unreliable if Xcode runs in 64-bit mode. It did cause me quite a bit of trouble until I found the same advice and switched Xcode to 32-bit. It has been reliable ever since. (You can toggle from the Finder inspector window).