I don't think it's even possible to cheat in SSD benchmarks. I believe they made the SSD benchmark to simulate real-world performance rather than purely synthetic numbers. And I think I read Anand's tweet/comment somewhere that they do plan to make their secret, undisclosed (to phone manufacturers) benchmark in the future.Reply

Have you considered the open source route for a mobile benchmark? While having the source open would let manufacturers know exactly what tests and how they're done, it would enable peer review of the code and allow other site to do a custom build that'd evade basic application detection.Reply

Hope you guys can afford one. Maybe kickstarter it and offer nothing in return. Just use the money to build the benchmark. In either case I just hope you don't let anyone touch it so they can't cheat it.Reply

First, when discussing how devices feel in hand, I expect analyses of polar moment of inertia from now on.

I think one of the big problem is you need unique benchmark suites for phones, or browsers.The PC used to have some of these issues, but not so much anymore because most of the programs used to test computer CPUs are the same programs the user actually uses. When you tested Haswell it was with Excel, 7zip, Visual Studio, Photoshop and so on. Those are real programs that people use, so if a vendor "cheats" at those, well it just makes those programs actually faster for users. When benchmarking video cards at this point, its pretty much exclusively with video games (if there are synthetic benches in a video card review I just skip it). At this point its just understood that AMD and NVidia tweak their drivers to improve performance in big name games, but you know what - that means those games are actually performing faster, so its actually a good thing. (as long as they aren't degrading video quality like the old nvidia quake problems)

So really the problem appears to come down to a lack commonly used apps and games on phones? Because then "cheating" on those would just make life better for everyone, well except that the current Android cheating method kills battery life.Reply

I think the problem is twofold. First, there is a distinct lack of automation tools. Windows automation is a mature, well-developed market, but this is much less true in the mobile/ARM space. Not to mention any such benchmark could well require root privileges in order to launch other apps and profile them. Is it even possible to write something like fraps as a non-root app for Android or iOS?

The other problem is basically what you mention--a lack of obvious benchmarking targets. Desktop benchmarks hit obvious, common tasks that people do with computers, like file compression, video encoding/decoding, rendering, and code compilation. The only tasks besides gaming that I can see being remotely applicable to mobile from a desktop benchmark are photo editing and maybe spreadsheet calculations. And mobile game developers don't include benchmarking tools, so I have to refer you back to my first point, which is that I at least don't know how you would go about benchmarking mobile games.

I'd love to see Anand's and Brian's approach to a good mobile benchmark, but it's definitely not a simple problem.Reply