You can leave the nurse bees if you are sure your queen is not on that frame, but it's hard to tell when you have several thousand bees to sort through. Probably safer to brush the bees off the frame. The swarm bees will take care of the brood.