Is the buy a building part of the card an option, the text on the card makes it sound like you have to.

The general rule is that when you have two actions separated by an ampersand, you can perform them in either order and must perform both if you can. You could begin by making an exchange that leaves you without enough money to buy any available building, but otherwise you can’t refuse to buy just because you don’t feel like it.