Weakness using Skeleton Key

I know why SE originally put weakness for a random amount of time will using a skeleton key but really it is already a pain to release the prisoners in Aht Urhgan but I shouldn't be punished for doing it. At the same time if you take the rare off the dropped door keys it would save me crafting skeleton keys.