If that's all that people are doing to you, you should consider yourself some what lucky. When I was working on those achievements people would come in and destroy buildings, especially the banner towers. At least those two things you can undo.