RobinS:
I had this happen a couple times inside a place and I painted it with sunshine concentrate and it definitely did the trick, and safe for the environment too.

52504:
Wow! Thanks a lot. I will be doing this. I have been wondering about what to do about that for years. Your right, they are wasps.

sharingsunshine:
You probably have paper wasps in the birdhouses. They commonly build there on the inside roof of the nest box. Use the end of a stick from the ground to knock off the wasp nest while wearing a pair of gloves to protect you from wasp stings. They are usually out and about during the afternoon which is a good time to do it. Generally they leave one to guard the nest .... but you may be able to find it without any guarding.

They generally build elsewhere if a nest is destroyed, but they might try again in the same spot. If the wasps repeatedly build in your nest box, coat the inside of the roof with a non-toxic substance like a thin application of Vaseline or Ivory soap: this will prevent them from building but won't bother the birds.