Gaming on server hardware? (ML350 w/ video and sound cards added)

I have found threads, and they are never specific enough. This is impossible to google, because you find so much about gaming servers for hosting.

First, my reason. I want this for personal use. I want a server that I can run ESXi on, and it's very specific on what you can run it on, due to drivers. I could try and piece together a white list compatible ESXi box, but it'd be hit or miss. The other side is that I want to be able to play some games, but don't want to have to invest in a second machine, and I don't want the noise and space of a second machine (wife factor). I just want to add something like a Radeon 2850 and a sound card. I am not looking for it to perform better than a dedicated desktop system, but within 10% or so on average would be nice.

Fully buffered RAM should not hold it back. Xeon processors don't seem like they'd be bad for gaming, and most of that load falls on the GPU. I plan on running a RAID 0+1 setup for the space allocated for Vista. And then we come to Vista, am I going to see missing drivers here?

I just want to know what problems you all think I might encounter with this setup, or dealbreakers. Or if anyone has heard of someone doing this, properly (not with onbard video)?