What rule(s) determine the answer? The gender of the principal word in the acronym ("agency"/"agência" in this example)? The natural gender of the "word" (e.g. "nasa" would appear feminine), or something else?

Yes, they do. Generaly the same rules used to determine the gender of a word in Portuguese are applied; an example of these rules is checking wheter the word ends in a or in o. For instance, it seems natural to me to say a NASA, a NSA, but not a FBI and rather o FBI.

However, there are no fixed rules on which to use, so you'll just use which ever sounds better.