If dark matter is asymmetric, fermionic, and self-interacting, it may form black holes in pulsars at the galactic center. In this case, a measurable maximum attainable pulsar age would track the density of the dark matter halo, with the oldest pulsars being allowed in the least dense parts of the halo. This could explain a recent observation, that there are not as many pulsars in the galactic center as expected.