You certainly do not have to wash hands just to change your pants or other clothing...

Just being within the pantry would not change that...

If you had some caustic cleaner that you had been using with bare hands then that could change the situation... ( you should not use directly on skin some products without precautions>>such as protective gloves >> of course)

For example I would caution you to not use bleach directly with bare hands but a laundry detergent would be OK to use but wash after for a couple of common examples...