You will fail a dungeon should 3 deaths occur while playing it. In a co-op party, these deaths are shared among the party -- if one player dies once and another dies twice, the entire party fails the dungeon and receives no rewards. You do, however, keep any loot or gold collected prior to failing.