Well, it depends. If you remove him, he'll simply come back unless you get a net or something to go over the top. But, if you don't, you'll probably have a lot more frogs eventually. I'd probably let him stay, personally, unless you really hope to breed a lot of goldfish in there.

If you don't want any company coming over ( Frogs , birds , cats ) and eating your goldfish , I would recommend you get a cover for the pond or bring the fish in ( if space and tanks permit ). If your goldfish collection is worth saving , owe them the right to be safe ! Just my two cents !