Once you identify all of the inactive fields you will need to activate them so that they can be removed from the record permission field. Once they have been removed the fields can be set back to inactive.