The Raw Shark Texts by Steven Hall. It is similar both in that it uses odd formatting (one word on a page, blank pages, text that forms an image, etc.) and in that the story itself is "weird." Good read. I read it specifically because I was looking for something like House of Leaves and it didn't let me down.

It's love it or hate it, but check out Black Box: A novel in 840 chapters. by Nick Walker. It's got a very strange format, an interesting story and a handful of good characters. It's not as daunting as the title sounds either, those 840 chapters are over in a little over 300 pages.