I suggest that you mark any references in the group you want to delete (make them hits) and then switch to the Hits group. Now you can delete them from the library with the Delete key or trash icon.

This (and many other operations) would be easier if there was a way to clear the hits group without changing context to select that group. This technique of using hits as a temporary group is quite useful. It is less useful to have to first go to the hits group to clear it, and then remember exactly where you were and what you were doing.