Even if they served only whole vegetables from YOUR garden, with paper plates and plastic utensils.

The reason is sometimes difficult for gerim: We can't give the appearance that we are eating non-kosher. If a goy were to see a known Jew eating non-kosher food (or the appearance of such - like eating at a goyishe house), it would case a bad light on all Jews.

Maris Ayin... I couldn't. There is no knowing if their kitchen is kosher, or their utencils, etc. If it is perceived that I ate somewher non-kosher, then it could be misconstrued that this is what Jews do (since I'm considered an observant Jew).

So I wouldn't.

Best bet is to graciously bow out and explain in simple terms the halachah. The only things that should be going into our bodies are holy things.....