This will sometimes happen to me, where the dw 5000 double bass pedal beaters, one will slip out of its holster, right after i tighten the lug with the drum key to hold the beater in, it falls out while playing sometimes

I used to have this with my pedals. they are no DW but they had the same problem. What I did is I used some blue threadlocker/lock-tight and that did the trick. It acts like a glue to prevent vibration from loosening the screw. Works well and I've never had a problem since, even between many many many beater switches. So I would recommend thread locker, but if you can't find that stuff then just tighten the crap out of the screws :p but don't strip the threads!