Cake house?

I've never made a gingerbread house because, from what I understand, in order to get structural integrity, the "gingerbread" basically has to be like spiced drywall. It's kind of like those awful weaponized decorated cookies that corporate clients send in "thank-you" baskets, and that poison half the support staff (the only ones silly enough to try and eat them).

But I digress. Wanting to make something festive and yet edible, I thought: why not make a house out of cake? Just make some rectangular layers, cut 'em, stack 'em, make a roof, cover with icing and festoon with candies. Woohoo! That would work, right? Has anybody actually done it? Any tips, gotchas, etc.?