I got the distinct impression that Gandalf and company wanted to avoid attracting the attention of the giants because one of the giants might decide to pick up a dwarf and drop-kick him. I suppose it could be considered an "amoral" danger, but it felt to me like something more than walking through a lightning storm. More like an ant walking by a kid with a magnifying glass at high noon.