BADRIYA, Iraq—Looking out from his hilltop post over the fighting for the Mosul Dam, the Kurdish special-forces commander fretted that his 5,000 troops didn't have the firepower to dislodge Islamist insurgents.

Less than 10 miles away, about 80 fighters from the radical Sunni group Islamic State were holed up inside a secondary dam. The area was so heavily booby-trapped and mined that Kurdish forces struggled to advance.