"If a handful of people that care enough want to surreptitiously visit Skara Brae, it's not going to cause any problems."

So if there's no erosion of the monument then erosion of the rules is OK?

Thats the very crux of the issue and I strongly disagree.

It is simply impossible for me to say if "I" break the rules at Skara Brae that definitely won't increase the likelihood that a vandal may follow suit. Ergo, neither I nor anyone else should break the rules at Skara Brae - for precisely the same reason, increased risk of damage, that you say people shouldnt break the rules at Silbury.
The safest policy for both places is dont break the rules.

Sorry Nigel, I don't see the world in such black and white terms as you, so we'll have to agree to differ.