Mineral Deposits in Water Heater Inlet

While re-piping the cold water supply to my Water Heater, I noticed that at the the dielectric union at the water heater was very clogged with rust and other minerals. This would probably partly explain the terrible hot water pressure I get in my house. It was almost totally clogged with rust and black gunk. I cannot remove the Male portion of the Dielectric union (that is screwed into the top of the water heater tank), as its in their pretty tight. Is there any danger in simply taking a screwdriver and breaking up the mineral deposits so they fall into the water heater... or should I put more effort into replacing the nipple/dielectric on top of the water heater?

Funny thing is I already replaced the hot side and it actually looked cleaner than the cold. Must have been the black iron coupling someone had inexplicably joined with the galvanized pipe I just removed from the supply side.