So after 5 tries I finally reach the end of the womb and kill mom's heart.After she dies a giant golden chest and a devil room appear.
I check out the devil room and notice a trapdoor but I'm like I'll go in there after I open that golden chest. However when I opened it the game ended so now I'm kind of curious as to what would've happened if I entered it.
Also does that room always appear or is it random like the rest of the devil rooms?

So I just beat Mom's Heart for the first time and had the Devil's Room pop up, but before walking in, I opened the chest. FUUUUUUUUUUUUU. It really sucks that picking up the chest ends the game.

I was getting so lucky in powerups (full stats, full health, brimstone, skatole, attack fly, meat cube, etc). So frustrated because I could have rocked Sheol and now, knowing this game, I won't have that kind of luck for another 20 hours of play.

Going through the Devil Room door should prompt the "You unlocked this," "This appeared!" and such messages, which are the things you'd unlock for beating the game x number of times. They also unlock just for leaving the heart room through the door you entered from, if you wanted to explore or pick up items you haven't gotten or whatever.
Afterward you can go to the Stats page from the main menu and view the ending if you wanna see what happens to Isaac.

tl;dr ALWAYS go in the Devil Room after the heart, you have nothing to lose.