When thrown, they will roll over the ground damaging any monster (up to 4 or 5 hits), much like a Shuriken or Bones. They cannot be picked up once thrown. They will continue to damage monsters once they stop rolling, making it useful for traps, although you would have to reset it often. They decay over time while on the ground and there is a limit as to how many can be placed at a time.

Spiky Balls are fairly effective against Skeletron, as they can damage his hands and head as Skeletron slides and rolls along the ground.

They are similarly useful against the Eater of Worlds as every segment will pass through the same space (unless it has split). In this way, the whole worm can be softened up. Spiky Balls are also as effective as Shuriken and Throwing Knives when fighting the Eye Of Cthulhu.
Their best use is as a defense against ground enemies: by making a funnel-like vertical duct that leads to the outside, the Spiky Balls can be safely thrown in front of a base's entrance to deal some damage and knockback to invaders. If you have lots of them you may use them as makeshift throwing weapons that later turn into traps (with enemies walking over them).

Update Info

In previous patches, Spiky Balls would dissolve in a thin layer of lava when other items would not, possibly due to their small sprite size. As of patch 1.1, they will no longer dissolve in a sufficiently thin layer of lava.