“Since they’re funded by taxpayer dollars (through the NIH, NSF, and so on), should scientists have to justify their research agendas to the public, rather than just grant-making bodies?

NO. No way.

The public has no context in which to understand most research programs and aren’t at all qualified to assess a grant proposal. This would be an invitation to the ignorant to proxmire good research. Can you imagine how the creationists would react to proposals in evolutionary biology? Or cat lovers to experimentation on animals?

On the other hand, I do think that researchers have an obligation to educate the public about how they are using federal funds. They don’t have to justify, but they should explain. It might be a useful condition of a grant to require that the recipient give an open lecture summarizing in terms a non-specialist can understand the results of their work, at the end of the grant period.