Here's what I propose. Use all the same tools and methods to find the data.

Now, when it comes time to present the data, simply convert the numbers into familiar FPS numbers. Lets say a 99th Percentile Frame Time was 17.54. Just display that as 57 FPS. Call it "99th Percentile Frame Rate".

So many people I talk to on other forums and even with my friends just don't understand that these numbers are essentially measuring the same exact thing, but more accurately. If even partially understood in the first place, it then becomes an additional test that is completely unrelated to frames per second. I don't think any amount of editorializing is going to drive this home, but maybe I'm being too impatient.

Flying Fox wrote:The problem is, fps is by definition an average over a period of time. But the individual frame times are basically a number "at an instant" (loosely). IMO it is even more confusing?

You're absolutely right, but I think that people who understand it know what you mean. The people who don't understand it are left scratching their heads because they've been thinking in terms of FPS for 10+ years. It's a hard transition to make and to fully grasp.

Even understanding the editorials and meaning I think imparts only a portion of what all of this equates to. By all accounts, it almost seems like it's simply measuring consistency. It really isn't, it's measuring a percentile, and EXPOSES inconsistency. It wasn't until I went deep into testing all of this myself with CPU overclocking performance that I even fully grasped the meaning of it all, and I find myself to be a fairly intelligent fellow (who doesn't on the internet though, right?).

Every time someone talks about in game performance it is 'but my frames dropped to 25-30fps in X, Y, Z parts'. This testing method faces two uphill battles in that it's a testing method that people can't have passively displayed in a tool tip near the top of their screen, and it's using terms and units of measurements that they are completely unaccustomed to.

NeoGAF, a fairly popular gaming forum with a great cross section of people. There's industry insiders, games developers, NVIDIA/AMD employees, hardware enthusiasts, and gaming enthusiasts. Look at the general responses in this thread: http://www.neogaf.com/forum/showthread.php?t=512976

It's really really hard to grasp, despite how logical it may seem to people who understand it.

I guess I just feel a sense of urgency to get this testing method to become standard. The Titan reviews are a perfect example of what I'm talking about. You see hundreds of people making arguments and huge monetary decisions over reviews that are nearly worthless. It makes me physically cringe to watch this happen. You pop in to say something about frame latency, and it just goes over the tops of their heads. It's a foreign language.

I'd rather it be understood and become the norm using terms that aren't quite accurate (FPS vs. ms) than have another 6 months go by where this is swept under the rug as some sort of off the cuff testing that has nothing to do with game performance.

Sure, you could display 99nth percentile frame time as frame rate. But why? It's still not the same as the average or min/max FPS metrics that everyone is used to anyway. As such, you may risk more confusion between average FPS and 99nth percentile FPS.

Plus, 99th percentile frame time is only the tip of the iceberg. The graphs that TR includes (albeit in conjunction with the single figure metrics) are where the real meat is.

Flying Fox wrote:The problem is, fps is by definition an average over a period of time. But the individual frame times are basically a number "at an instant" (loosely). IMO it is even more confusing?

Even though frames per second is a measurement over a time period, its not to say that frame times could not be converted into frames per second. For instance, if it takes a person 10 seconds to run 10 meters it is not difficult to convert this into a speed (meters per second). The amount of time it takes a frame to render is roughly equivalent.

I think that the same is true of frame times, and while it might slightly misleading, I think that knowing X frame time is equal to Y frames per second if it were to continue over a period is a useful measurement especially if multiple frame times were converted to to fps second. For example if a reader saw two lists of frame times converted to frames per second and saw that one was consistent and the other vary dramatically, he or she could still draw the same conclusions as would be drawn from frame time. Further, seeing a list that contained lots of "15fps" measurements paired with lots of "120fps" measurements would clue the reader in that the card was under performing in a way that seeing the frame times were not simply because frame times are not such an intuitive measurement.

All in all, what the frame time measurement is trying to measure is how consistently a GPU is outputting frames and I think any effort to make this new measurement conform to reader expectations would go a long way in expanding the use of frame time as a metric.

So, in other words, you're so wedded to FPS that you'd prefer Scott's latency graphs to be converted from time per frame into FPS because all you understand is FPS, even though each GPU review gives the explanation of how to convert time per frame into FPS.

Captain Ned wrote:So, in other words, you're so wedded to FPS that you'd prefer Scott's latency graphs to be converted from time per frame into FPS because all you understand is FPS, even though each GPU review gives the explanation of how to convert time per frame into FPS.

No, I'm not wedded to it in any way. I'm talking about the clear difficulty in getting people to understand what any of it means. Yeah, it's clearly written, but there's a very visible block that people are having in understanding that this is a more accurate measure of performance rather than an alternate measure of performance. Using the different measurement I think contributes to this.

mkenyon wrote:No, I'm not wedded to it in any way. I'm talking about the clear difficulty in getting people to understand what any of it means. Yeah, it's clearly written, but there's a very visible block that people are having in understanding that this is a more accurate measure of performance rather than an alternate measure of performance. Using the different measurement I think contributes to this.

OK, how do you explain to someone that their GPU went from 120 FPS to 10 FPS, but only for 100 milliseconds? FPS is dead and gone and should have been dead and gone the first time anyone twigged to frame latency (I have to believe that ATi/AMD and nVidia knew this long before Scott went public with it) and I think the current measure of 99th percentile latency combined with time over x milliseconds perfectly describes how current GPUs do their job.

You're talking about adjusting the paradigm under which an entire hobby and entire industry functions. (ノдヽ) For so many years -- before I was born, even -- the games industry has been about "dat FPS". Even as far back as Doom, people were concerned about the framerate of their game; starting the game with -devparm enabled, among other things, an FPS ticker. 3dfx famously used 60 FPS in the marketing for their Voodoo5 series of graphics cards.

A big change like that takes a lot of time -- multiple years -- and some people just aren't going to get it at first. Heck, some people will never get it at all; look at the gun fogeys who still refer to 7.62 NATO as .308, even though they aren't the same thing. ヽ（´ー｀）┌ I don't think this kind of "transitional" step is going to do anything but confuse the issue further.

mkenyon wrote:You didn't address anything I said above though. In fact, you wrote most of it off with a fairly snarky retort that betrays lack of comprehension.

The problem is that your solution, i.e. converting 99th percentile frame time to FPS, deliberately obfuscates the original measurement just so that some can read the review without having to think, and also destroys the entire data stream used to create the frame time histogram. Your proposal throws away the high-latency outliers that are the reason everyone swears at their gaming PC from time to time. The 99th percentile is the Good Stuff. What makes things maddening is how badly GPUs deal with the stuff after the 99th percentile and converting 99th into FPS ignores that completely.

Scott's post here makes it clear that he's working with other review sites on this and also makes it clear that there will no longer be one single number to pull out of a GPU review to allow noobs to make bad decisions. You're trying to boil GPU reviews down to a single number. Sorry, but you're going to have to read and grok the reviews before buying instead of looking for a single number to make your decisions for you.

Yup, I agree. If (as seems apparent) raw FPS is a poor measure of gaming performance, an entirely new metric is needed, and it needs to be clear that it is a new metric. Don't try to force the new metric to conform to the old paradigm, it will add to the confusion rather than reducing it.

mkenyon wrote:You didn't address anything I said above though. In fact, you wrote most of it off with a fairly snarky retort that betrays lack of comprehension.

The problem is that your solution, i.e. converting 99th percentile frame time to FPS, deliberately obfuscates the original measurement just so that some can read the review without having to think, and also destroys the entire data stream used to create the frame time histogram. Your proposal throws away the high-latency outliers that are the reason everyone swears at their gaming PC from time to time. The 99th percentile is the Good Stuff. What makes things maddening is how badly GPUs deal with the stuff after the 99th percentile and converting 99th into FPS ignores that completely.

Scott's post here makes it clear that he's working with other review sites on this and also makes it clear that there will no longer be one single number to pull out of a GPU review to allow noobs to make bad decisions. You're trying to boil GPU reviews down to a single number. Sorry, but you're going to have to read and grok the reviews before buying instead of looking for a single number to make your decisions for you.

Okay, I don't think you understand at all what I'm saying. The bolded above is offensive, and quite honestly very snobbish of you to inject when I make no such request for a single number. You're looking to shoot down an argument that I'm not making, while throwing in some questionably ad hominem type attacks into your point. This is how questions are greeted by moderators? Yikes.

just brew it! wrote:Yup, I agree. If (as seems apparent) raw FPS is a poor measure of gaming performance, an entirely new metric is needed, and it needs to be clear that it is a new metric. Don't try to force the new metric to conform to the old paradigm, it will add to the confusion rather than reducing it.

I think this makes sense, but the Inside the Second articles are a lot to digest. There's so many points and revelations in there that some finer details might be obscured or fail to hit home. Frame time testing is so much more accurate, yet exists in a sort of bubble because the data is both hard to grasp and fairly difficult to test on your own outside of what everyone has always seen subjectively as stutter. It took me a weekend of fiddling around with excel, a number of questions on this forum, and an email to Scott to get the data all in line with the correct metrics.

I think that while clearly demonstrating that this is a different methodology with different measurements is a convincing point. However, the double edge of this is that people are thinking that this is an additional suite of tests to frames per second, when really they are a replacement for a more accurate measurement of what Min/Max/Average has for so long failed to capture adequately. I'm not sure that this point has been driven home hard enough, which is why I was suggesting a switch to FPS.

You can measure the MPH of a car at any given moment. Within a second of heavy acceleration between 1-30 MPH, the car could at one point be measured at going 22.3 MPH. That isn't inaccurate nor is it misleading. I don't think it is here either.

For example, on the 'Time Spent Above' graphs, the data could be 'Time Spent Below X/Y/Z FPS'. The 99th Percentile is easily converted as outlined above. I do think, at first glance, the Frame Latency by Percentile graphs are most definitely best served by continuing on as is. While it is important to make a distinction between these tests and the previous second based polling of FPS, I think the different stats communicate that sufficiently without needing to also display the measurements in 'milliseconds to render frame'. It of course works great for people who get it, but there are just so many people that don't nor care to. Giving them something familiar to ease into the transition might make this an easier point to get across.

Ugh. Despite your trashing of my comments you still want an FPS number to fall back on.

FPS died the instant Scott posted the first frame latency article. Get used to it. The metrics are going to change and FPS will no longer be one of those metrics. The noobs will actually have to learn something.

Captain Ned wrote:Ugh. Despite your trashing of my comments you still want an FPS number to fall back on.

FPS died the instant Scott posted the first frame latency article. Get used to it. The metrics are going to change and FPS will no longer be one of those metrics. The noobs will actually have to learn something.

I don't think you understand the discussion at hand. Is there a point that I can elaborate on to help clear things up?

1. The analysis has changed. Even though you're trying to propose a "transition" from old to new, you risk causing further confusion. FPS was inadequate because it masked issues that gamers were experiencing that frame times make clearer. Separating the units makes it so that you KNOW that you are dealing with "smoothness" numbers rather than raw averages.

2. Your computer is rendering discrete frames. FPS averages are derived from averaging frame times over a period of time. This is somewhat unnecessary because you can timestamp each individual frame, but the flow of animation relies more heavily on even frames rather than number of frames. Getting 120 frames delivered in one second sounds good, but the perceived FPS would be a lot less if there were several high latency frames delivered in that time. Targeting metrics in ms convey this better than FPS, and you can get even better additions like "time spent beyond X ms," which tells you how much time was spent stuck looking at old frames when new ones were slow.

What we're getting to is that this isn't just a rendering speed issue, but a reconstruction issue. If you want to reconstruct smooth animation at the monitor, you have to weed out individual frames. Again, measuring in ms is better.

3. Even 99th Percentile Frame Time doesn't get the job done. The measure is light years ahead of traditional FPS analysis, but I still don't we have pinpointed a perfect benchmark that says "card X has this smoothness rating." Even if you add up all the industry work, you get a lot of good information and a lot of insight into the rendering process and correlations between metrics and results on screen, but we still don't have a metric that can definitively describe smoothness.

Trying to translate these new metrics into FPS lingo is going backwards. We need consumers, hardware developers, driver programmers, and game programmers to think about things in terms of frame delivery. Frame time metrics do this. Instead of trying to translate, we'd be much better served by educating the new analyses.

"A life is like a garden. Perfect moments can be had, but not preserved, except in memory. LLAP"

I feel like you guys are trying to tell me what the difference is between FPS based tests and frame time based tests. I don't have any confusion on this. I've even gone as far as to create mini guides to get people up to speed on the new methodology.

I'm simply talking about presentation of the findings in a format that is easier to digest. That doesn't change the data gathered, tested, and displayed. It is simply changing the information to help reiterate that this is a replacement for the standard FPS metrics that have been in place for years.

mkenyon wrote:I'm simply talking about presentation of the findings in a format that is easier to digest. That doesn't change the data gathered, tested, and displayed. It is simply changing the information to help reiterate that this is a replacement for the standard FPS metrics that have been in place for years.

Because you're still looking for a single-number metric by which you can rank GPUs. We're trying to tell you that that approach is no longer valid. FPS (or any other single-number metric) IS DEAD. Get that through your microcephaly. Oh, and keep insulting my intelligence. I love the abuse.

mkenyon wrote:I'm simply talking about presentation of the findings in a format that is easier to digest. That doesn't change the data gathered, tested, and displayed. It is simply changing the information to help reiterate that this is a replacement for the standard FPS metrics that have been in place for years.

Because you're still looking for a single-number metric by which you can rank GPUs. We're trying to tell you that that approach is no longer valid. FPS IS DEAD. Get that through your microcephaly. Oh, and keep insulting my intelligence. I love the abuse.

I don't think he was insulting your intelligence. He just has a different opinion.

I don't see how saying "here's something that you need to *pretend* is FPS" helps at all; it just causes confusion between the "old" FPS and the "new" FPS. You need to convince people that FPS is the wrong way of looking at things, and that there's something entirely different that is a better measure of performance.

mkenyon wrote:I'm simply talking about presentation of the findings in a format that is easier to digest. That doesn't change the data gathered, tested, and displayed. It is simply changing the information to help reiterate that this is a replacement for the standard FPS metrics that have been in place for years.

Because you're still looking for a single-number metric by which you can rank GPUs. We're trying to tell you that that approach is no longer valid. FPS (or any other single-number metric) IS DEAD. Get that through your microcephaly. Oh, and keep insulting my intelligence. I love the abuse.

Please show me where I ask for a single number metric in order to rank GPUs?

In fact, I said:

For example, on the 'Time Spent Above' graphs, the data could be 'Time Spent Below X/Y/Z FPS'. The 99th Percentile is easily converted as outlined above. I do think, at first glance, the Frame Latency by Percentile graphs are most definitely best served by continuing on as is. While it is important to make a distinction between these tests and the previous second based polling of FPS, I think the different stats communicate that sufficiently without needing to also display the measurements in 'milliseconds to render frame'. It of course works great for people who get it, but there are just so many people that don't nor care to. Giving them something familiar to ease into the transition might make this an easier point to get across.

That's not a single number, it's not a single test, that is in fact three that I use as a beginning point.

just brew it! wrote:I don't see how saying "here's something that you need to *pretend* is FPS" helps at all; it just causes confusion between the "old" FPS and the "new" FPS. You need to convince people that FPS is the wrong measure, and that there's something entirely different that is a better measure of performance.

Yeah, I get that and agree with it to a large extent. My only counter point which I'm not putting all my stock into is that I think by having the range of tests that we do look different enough and describe different enough things to eliminate any chance of confusion.

just brew it! wrote:I think if you want to stress that it is a new (and better) way of looking at things, you need to call it something else. Something that does not have "FPS" in its name.

This summarizes my thoughts as well. The reason not to use 'instantaneous framerate' (which is what frame time is) is that it's too close to 'frames per second'. Breaking with that paradigm is the whole point of looking at frame times, presenting it in a way that makes it familiar for the sake of being 'easy' would defeat the purpose of thinking about things differently. I wouldn't say FPS is flat out 'wrong' it's just not everything.

mkenyon wrote:So far, that hasn't panned out too well. I'd say an overwhelming majority of folks that I talk to about this view it as a way to capture something else entirely which isn't quite as important as FPS.

If it hasn't panned out well, that's the fault of people reading it not the measurement. The presentation methodology has been refined since it was first introduced. There are a few reasons it might not have panned out well and in all of them the onus is on the reader to improve. The first is simple vested interest, or what might be called fanboyism - people see 'their card' from 'their company' not doing well, or getting results they didn't expect based on previous inferior measurements, and they don't want to accept it so they fight it by saying things like 'it doesn't really measure framerate' and such. The second is that because it is much different than a simple average FPS it's harder to grasp even though it doesn't have to be. To be blunt, I think this is because people aren't smart enough to wrap their head around the presentation, or to understand what it represents despite explanations. They'd rather have a conclusion of X>Y, what's funny about that is even with FPS testing in different games that was never true anyway.

Flying Fox wrote:The problem is, fps is by definition an average over a period of time.

I'm a bit late to read this topic, but I have something to add, which is that I agree with kumori and mkenyon. The OP has a point that could be used in future TR reviews. Maybe have an extra button to click, to switch between millisecond and frame-per-second graph labels?

FPS can be used to show instantaneous performance. Think about it as a speedometer; if yours says 50 mph, does that mean you've been driving for an hour? No! It just means that if you keep it up, you will go 50 miles in an hour's time. Same with FPS. Does an FPS count mean it was sustained for a second? No! It just means that if it were, then X frames would be delivered in one second.

Superjawes also had an idea about a "smoothness rating" formula, that sounds like something TR might look into.