The other dwarves know that it's useless and only the nutty ones with nothing to loose or the ones driven by revenge are willing to go.

Also, maybe it's similar to the Fellowship. The less people they have going the easier they'll pass through unnoticed? "...For if joyful is the fountain that rises in the sun, its springs are in the wells of sorrow unfathomed at the foundations of the Earth"