Radeon R9 295X2 CrossFire at 4K - Quad Hawaii GPU Powerhouse

You need a bit of power for this

PC gamers. We do some dumb shit sometimes. Those on the outside looking in, forced to play on static hardware with fixed image quality and low expandability, turn up their noses and question why we do the things we do. It’s not an unfair reaction, they just don’t know what they are missing out on.

For example, what if you decided to upgrade your graphics hardware to improve performance and allow you to up the image quality on your games to unheard of levels? Rather than using a graphics configuration with performance found in a modern APU you could decide to run not one but FOUR discrete GPUs in a single machine. You could water cool them for optimal temperature and sound levels. This allows you to power not 1920x1080 (or 900p), not 2560x1400 but 4K gaming – 3840x2160.

All for the low, low price of $3000. Well, crap, I guess those console gamers have a right to question the sanity of SOME enthusiasts.

After the release of AMD’s latest flagship graphics card, the Radeon R9 295X2 8GB dual-GPU beast, our mind immediately started to wander to what magic could happen (and what might go wrong) if you combined a pair of them in a single system. Sure, two Hawaii GPUs running in tandem produced the “fastest gaming graphics card you can buy” but surely four GPUs would be even better.

The truth is though, that isn’t always the case. Multi-GPU is hard, just ask AMD or NVIDIA. The software and hardware demands placed on the driver team to coordinate data sharing, timing control, etc. are extremely high even when you are working with just two GPUs in series. Moving to three or four GPUs complicates the story even further and as a result it has been typical for us to note low performance scaling, increased frame time jitter and stutter and sometimes even complete incompatibility.

During our initial briefing covering the Radeon R9 295X2 with AMD there was a system photo that showed a pair of the cards inside a MAINGEAR box. As one of AMD’s biggest system builder partners, MAINGEAR and AMD were clearly insinuating that these configurations would be made available for those with the financial resources to pay for it. Even though we are talking about a very small subset of the PC gaming enthusiast base, these kinds of halo products are what bring PC gamers together to look and drool.

As it happens I was able to get a second R9 295X2 sample in our offices for a couple of quick days of testing.

Working with Kyle and Brent over at HardOCP, we decided to do some hardware sharing in order to give both outlets the ability to judge and measure Quad CrossFire independently. The results are impressive and awe inspiring.

Let’s talk quickly about the specifications and raw power found in this kind of setup. With a total of four Hawaii GPUs amongst the two R9 295X2 graphics cards, we have a total of 11,264 stream processors at work capable of 23 TFLOPS of theoretical compute power! Between all GPUs there is 16GB of addressable GPU memory, more than most enthusiasts’ desktop system memory capacity.

With each R9 295X2 claiming a 500 watt TDP, we are looking at 1000 watts of power draw from the GPUs alone in this machine and in my testing it actually exceeds that.

Setup Concerns

Moving past the concerns of cost, which is plenty large enough on its own, power consumption issues cropped up. Our standard GPU test bed uses a Corsair AX1200i power supply that has been more than enough for our single, dual and triple graphics card testing over the last year or so. As we quickly found out though, the pair of Radeon R9 295X2 cards were able to bust past the limits of this PSU requiring us to be more creative.

Since we did not have any power supply available above 1200 watts (yet), we went with the ol’ double power supply trick. The Corsair AX1200i powered the motherboard, platform, SSD and primary graphics card while we attached a secondary Antec 750 watt High Current PSU to power the second R9 295X2. By shorting out a couple of pins in the ATX connection you can power on the PSU without it being attached to a motherboard and this allowed us to maintain a stable and functioning test system.

Clearly this is not ideal for consumers and no one should have to use two different power supplies to get a single machine running. Corsair did ship us one of the upcoming AX1500i units that will have no problems with the pair of R9 295X2 cards. However, we did not get it in time for this publication. Extreme gaming calls for some extreme power supplies, it seems.

As for a system build using a pair of R9 295X2 cards you need to find a case that can handle the mounting of two individual 120mm radiators, likely one on the back and one up top. If you plan on water cooling your CPU as well you’ll need yet another mounting location.

at 28nm indeed is not useful if you are speechless...what did you expect from 2x dual GPUs like these? I don't think that's a "normal" setup...
then wait for 20/16nm if you think about power saving for a quadcrossfire rig... ;-)

Smaller components mean less power usage, because smaller just uses less power. It is also an important part of the architect of the chip itself. Smaller components mean more components they can place on one chip. Which means more compute units in each chips.

Games typically use a technique called AFR to increase performance. In a two way GPU system, GPU A will render the even frames and GPU B will render the odd number ones. The ideal result is an ABABABABABABABABABAB patters of what GPU renders what frame. Scaling to three GPUs produces an ABCABCABC pattern and four goes to ABCDABCDABCD.

There are a couple of problems with this technique and GPU scaling. First is that there needs to be enough CPU power to provide the frame rate increase. Nowadays games are typically GPU limited but a CPU limitation could crop up in 3 way and 4 way configurations. Secondly, there is an API limitation to how many frames at one time can be processed. DirectX supports a maximum number of 6 concurrently frames being processed simultaneously. Thirdly Windows 7 has a limit of 8 GPU's in a system (I suspect Windows 8.x has the same limitation but haven't personally checked). Most distributions of Linux have a similar 8 GPU limit as Windows but there are kernel patches that'll enable more in a system. Fourth is that systems that use BIOS to boot will have issues with 8 or more cards due to legacy 32 bit memory allocations for GPU's. 64 bit EFI does not have this issue.

nVidia previously used a technique called split frame rendering where the top half of a frame is rendered by one GPU with the bottom being taken care of by another. This solves some of the issues outlined above. CPU load doesn't necessarily have to increase linearly but there is a bit of overhead in the drivers to perform load this load balancing. The DirectX limitation is also by passed directly so the real limitation becomes how many GPUs a single system can boot with. Since the number of frames being worked on is the same as a single GPU system, there is no microstuttering like you could encounter with AFR. SFR has several of it own issues though. nVidia has hidden away SFR support in their drivers so some developers tools are necessary to even enable it. This is for good reason as it is buggy and in some cases doesn't work at all. Last I checked, SFR didn't scale as well in 2 way GPU scenarios. I have not seen any modern tests using SFR and 4 way GPU's but really old benchmarks had 4 way AFR and 4 way SFR relatively similar in terms of scaling (about 3x performnace as a single card ideally). AFR and SFR can also be combined. AMD doesn't have a direct equivalent to SFR. I do recall some talk of a tile based solution where each GPU would render a checkerboard pattern but I believe nothing came of this.

I guess they must be actually using AFR for 4 gpu rendering. This seems like it would cause some artifacts or stuttering, attempting to render 4 temporally distinct frames simultaneously. Does this essentially induce 4 extra frames of latency? If so, is this enough to notice? Would the 6 frame DirectX limitation actually cause a problem? I could imagine needing to start set-up for another 4 frames while previous 4 frames are still processing (8 active frames).

Tiling the rendering load could scale to 4 or more gpus better, but the load balancing is not simple. Most images encountered in games can not be simplistically divided, since the load would be significantly different. The top tiles might only be sky, lower tiles might be low res scenery texture/geometry, while one tile may get high-res character texture/geometry. It may be simpler to just use stripes (NVidia SLI ?) rather than attempting to split into arbitrary tiles since you need something which works for 3 gpus and 4 gpus.

It is somewhat amazing that we are seeing good scaling in some of these games already. It would be interesting to know what is being done differently between those that scale and those that do not. Are some of them specifically optimized for up to 4 GPUs in the render engine?

Only really having access to 4 GB seems to be causing some performance limitations (see HardOCP testing). It would be nice to see the GPUs able to share memory rather than completely independent memory systems, but this will not be available for a while yet. Nvidia seems to be working on this with their NVlink technology; I don't know what AMD is doing. Sharing GPU memory requires really high bandwidth interconnect, but it is doable. If one used 4 of AMDs 32-bit HT links, you could get over 100 GB/s and this is not the latest tech. For chips really close together, the speed could probably be increased. You would probably reduce the width of the memory bus on each chip, and replace it with interconnect to neighboring chips. It could use 2 GB attached to each GPU. You may be able to put quad-gpus on a single card this way, but the power consumption/heat output would be limiting.

Anyway, I wouldn't buy one of these. My AC bill is already high enough without adding a 1500 W space heater. Back when I lived in a cold climate, I used 2 1500 W oil-filled radiant heaters to heat quite a bit of my house. If I were to use a 1500 W system in California, I would need to rig up some fans and ducts to blow the hot air out the window.

I decided to save that discussion for another time as it would really have complicated things. We wanted to find the CrossFire performance factor without mudding it up with Mantle stuff that may not be perfected yet.

1261W at the wall times the ~89% efficiency gives 1122W real consumption that is about 93,5% of it's capacity that should not be the slightest problem for a PSU of this ones quality! (There are plenty of other PSUs that I would not even consider buying, but this one SHOULD handle this setup 24/7 at max load for years!)

The r9 295X2 has some pretty stringent power restrictions to it. Another website I know of did this with a 1350W PS and still had issues. These two cards together, with their unique specifications, really need a 1500W PS.

Well in theory 1200watt PSU would be enough, but not everyone has the same setup, some machines pull more power cause cpu used and even how many hdd's they have. Point being should get a bit more beefy power support then 1200 watts if you plan to run 2 of these cards. Also comes in to play is the circuit the computer is on in your house but that is another matter.

My point was, whether there was an issue, or they just went with 2 PSUs based on the [H]'s experience... Don't take me wrong, Enermax is not bad, but I would not consider it as an option for myself :P... This particular Corsair however is a different cup of coffee...

In Europe, we do not have (huge) limitations on power in flats... The default here is 16A per 230V circuit and a 25A common fuse in front of it(them), so in theory a 3.5kW PSU (the actual legal limit on a single phase appliance power in a home) would be doable and the lights and maybe a TV would still be on in the flat :D

I wanted to follow up a previous comment someone posted asking about three way with a 295X2 and a 290X. I suppose theoretically that should work but would it actually work in real world gaming applications?

I would love to see a quick setup with that configuration and how it scales.

Given the weak scaling of 2 x 295X2s in some games, it might make more sense just to setup the 295X2 with a 290X for extreme performance and a little less price and power consumption.

lol this card is killing nvidia xD.
they delayed the TitanZ trying to figure out a way to beat it.
when im guessing everything was ready, R&D done, the cooler probably bought and even mounted on the pcb, packagings... thats alot of money wasted for last minute delay to rework all that, the TitanZ perf must have been ridiculously low compared to the R9 295x2, to accept to lose so much money.

probably yes, but then the Nvidia brand would take a deadly blow, that will go from highest perf GPU, to the most greedy margins scammers.
Nvidia have been making alot of mistakes these last couple years, and very bad ones at that, from G-sync, to this stupid luxury new brand Titan, that failed in 3 consecutive instances, and AMD isnt making things easy for them, from Mantle, freesync,Bundles to R9 295...just imagine, that if AMD didnt bring out this card, we would have today a Nvidia card 20-30% less performant at twice the price, i am really glad that AMD is sticking into desktop GPU market otherwise Nvidia would have sucked us out dry.
they really should give it up and go back to GTX, because honestly i dont see a place for this card in the market, it would just hurt them.
because if they do like AMD with watercooling that would hurt their image for copying AMD, if they stay on air they have to find some ingenius way to keep it stable while overclocked, which ii doubt very much.
the most likely is that they will put a good cooling that wont be able to keep the frequency, but then pay some sites to bench with cool cards before it warms up, and they also have to drop the price, because AMD planned on the 3000$,
with the hashtag #2betterthan1, witch comparaison for same price is crossfire r9 295x2, so the price need to be lower than 2k$, otherwise AMD would just drop the price to match the 1/2 ratio

and ofc "nuff said" is a sound counter argument, that's what cracks me up about ppl like you, if i was deluded you would state why unless ofc you have nothing to say without sounding ridiculous...nuff said lol

That Corsair AX1500 would have come in handy. Thats a lot of gpu power you got there Ryan, epic review. At least AMD has made great improvement in their drivers to support 4 Gpus, although still not perfect.

Now, im still gaming on a 1680x1050 monitor, and your on 4k, you have the best job Ryan!

hi bro i just have a question is possible make one tri crossfire of 295x , so i do ear some over the new standard CrossFireX XDMA was possible make a cross with more of 4 GPUS .. cheers i just have that doubt