I realise it's really hard for the other three of you, and I understand that you feel betrayed.
But I don't think there was any malice on his part. I think he was confident he could do it, and then found out he couldn't.
I would urge you to work on it together. See what makes him insecure or jealous about "sharing" his wife. See if there are things she could do that would make him feel better about it.
As much as she must feel pushed away by his attitude, she might need to give him more attention and affection to help him through.

I know that from you're point of view, he's the party pooper and you're not too happy with him. But I hope this won't turn into an "us vs him" kind of situation. In cases like this, it's important to reinforce the relationships. Get together and make it clear you want to work through this. But also make it clear you don't want to just go back to being friends. You want to give the whole thing a fair try.
Is he fine with "sharing" his wife with your wife, or does he have problems with it too?

I think he needs to feel confident that he's not going to lose his wife over it. Only then can he start being comfortable with the idea, as he was hoping he would be.

I think his point of view here would be extremely interesting. Any way he can share it?