This is not the first time that such claims from this dxo company seem unbelievable. I don't have the physics background to dive into this. But this seems rather a very singular approach to what is clearly a sum of multiple parts. I don't own a Leica (unfortunately) and I don't own a Micro 4/3rd either. But the real life results that I see are pretty obvious in favor of the Leica system. And I'm just talking about the various test shots that I've seen and should not include the "Leica buyers are more serious about photography" factor which may or may not be true.

Maybe the Leica sensors are technically "inferior" in this or that way but I still like looking at the results some folks are getting. And I have to admit that I wish I could afford the M system. It would work well with my preferences.

You seem to be doing the error of believing that "if camera A has better DR than camera B, then camera A must produce better images. If it does not, then DR must be wrongfully measured".

This is not so. People make wonderful images using low-DR capturing.

Just like a 36 MP camera does not in itself make better pictures than an 18 MP camera. The photographer makes pictures, the camera facilitates. A high-resolution camera can enable some images that would be hard to do on a 18 MP camera.

DXO mark means nothing. They take their objective tests and interpret them in completely subjective ways, all (as it would seem) on Nikon's dime. Super biased, all rubbish, especially their ISO scores. Canon should be top of the mark for every camera they've put out in the ISO division, right next to Pentax, and Nikon and Sony should be right in the bin every time. But it's absolutely the opposite. If you've ever seen the DPReview studio comparison tool, I highly recommend cranking up the ISO on there and seeing for yourself what I'm talking about. D5200? Rubbish. D800? Rubbish at anything above or below ISO 200. 5DIII? A f@ck!ng mint. DXO scores the opposite in every case.

And I will reiterate again, that their ISO score is weighted toward the base ISO, and not at more moderate levels. And then there's the downsampling to 8 megapixels...certainly that helps the D800...a ton! It's also a meaningless method of scoring noise.

Would anyone like to bet me a new series ii 400 or 600mm supertele lens, that DXO will somehow not score the future high megapixel Canon 1 series camera's sensor, well below that of the Nikon D800? I mean, what's the closest it could get to their score for the D800E? 7 points? 12?

If it scores less than 10 points behind the D800E, then in reality, it's actually 10 points ahead...and why give the 800E one more point just for not having an anti aliasing filter? The sensors are THE SAME...

If average people like me can predict the outcome of a test, 6 months or more before the product is even announced in the public domain, while it is still a rumor...then you have to admit, it does smack of favoritism and intentionally skewed test results.

I mean, the little Nikon D5200 scores 2 points better than the 1Dx...GET F***ING REAL...That's like saying a Prius scores higher than a Ferrari 458 because it gets better gas mileage, or because the driver tends to obey the speed limit more closely. The test is just wrong, skewed to portray a "truth" that is too narrow in its scope to have the meaning that many think it does.

And I will reiterate again, that their ISO score is weighted toward the base ISO, and not at more moderate levels. And then there's the downsampling to 8 megapixels...certainly that helps the D800...a ton! It's also a meaningless method of scoring noise.

Would anyone like to bet me a new series ii 400 or 600mm supertele lens, that DXO will somehow not score the future high megapixel Canon 1 series camera's sensor, well below that of the Nikon D800? I mean, what's the closest it could get to their score for the D800E? 7 points? 12?

If it scores less than 10 points behind the D800E, then in reality, it's actually 10 points ahead...and why give the 800E one more point just for not having an anti aliasing filter? The sensors are THE SAME...

If average people like me can predict the outcome of a test, 6 months or more before the product is even announced in the public domain, while it is still a rumor...then you have to admit, it does smack of favoritism and intentionally skewed test results.

I mean, the little Nikon D5200 scores 2 points better than the 1Dx...GET F***ING REAL...That's like saying a Prius scores higher than a Ferrari 458 because it gets better gas mileage, or because the driver tends to obey the speed limit more closely. The test is just wrong, skewed to portray a "truth" that is too narrow in its scope to have the meaning that many think it does.

+1 ... actually several people at nikonrumors echo similar sentiment, recently DxO ranked some Sigma and Tamron zoom lenses ahead of some of the better Nikon zoom lenses ... also the Nikkor 70-200 f/4 scores higher than Nikkor 70-200 f/2.8 ... lot of the Nikon users are not very thrilled by it. I'm no expert at DxO scores but ranking Sigma 120-400 lens ahead of some of the Nikon lenses clearly shows that real world results and lab tested results are very different.

And I will reiterate again, that their ISO score is weighted toward the base ISO, and not at more moderate levels

Completely wrong, please review their method for ISO score. The dynamic range and color depth scores use base ISO. The ISO score is the highest ISO that meets a number of image quality criteria (the highest for which a certain noise level and dynamic range are maintained)

Quote

And then there's the downsampling to 8 megapixels...certainly that helps the D800...a ton! It's also a meaningless method of scoring noise.

Also not true. Normalizing to a common resolution is completely sensible unless you're always viewing 100% crops. The typical use case is to ultimately print images at the same size or otherwise rescale the image to the same size. To do otherwise is a "pixel peeping" approach. They have "screen DR" so that you can also see the per-pixel results, but these are not as meaningful.

Quote

If it scores less than 10 points behind the D800E, then in reality, it's actually 10 points ahead..

I don't think so -- perhaps you can show us your industry leading benchmark that demonstrates this.

Quote

.and why give the 800E one more point just for not having an anti aliasing filter? The sensors are THE SAME...

The anti-aliasing filter obviously has some impact on noise properties of the image. You wouldn't expect an enormous difference but then the test doesn't show an enormous distance so I don't really understand this complaint.

Quote

If average people like me can predict the outcome of a test, 6 months or more before the product is even announced in the public domain, while it is still a rumor...then you have to admit, it does smack of favoritism and intentionally skewed test results.

Maybe it just means that average people can predict how well the next sensor from each major manufacturer will perform. Given the relatively slow trajectory in sensor improvements, today's performance is a pretty good predictor of tomorrow's performance.

Canon's low ISO noise hurts them because the aggregate score is heavily weighted towards low ISO performance. The 1DX has substantially better high ISO performance but that is not the only factor the overall score takes into account. The 1DX also has a number of features that are important to serious photographers, but these don't enter into the test either.

Quote

That's like saying a Prius scores higher than a Ferrari 458 because it gets better gas mileage,

Well, what do you mean by "scores higher" ? Trying to assign a single numerical "score" to something complex is always a difficult task. Here's one -- should a Mazda Miata score "better" or "worse" than a Honda accord ?

The reason they do report a single aggregate score is that many users do not want to dig deeper. But for those who are, they publish all the underlying measurements.

Quote

or because the driver tends to obey the speed limit more closely. The test is just wrong, skewed to portray a "truth" that is too narrow in its scope to have the meaning that many think it does.

The test is just fine. You are confusing the test with the score. You may disagree with the way the factors are aggregated, but they do provide their measurements for you so that you can get the most out of their tests even if you don't agree with the weighting scheme (or if you want to understand why the results came out the way they did).

The notion that the test is intentionally biased in favor of Nikon is just silly. DxO devised their test score and methods before Canons struggles with low ISO performance became the elephant in the room.

+1 ... actually several people at nikonrumors echo similar sentiment, recently DxO ranked some Sigma and Tamron zoom lenses ahead of some of the better Nikon zoom lenses ... also the Nikkor 70-200 f/4 scores higher than Nikkor 70-200 f/2.8 ... lot of the Nikon users are not very thrilled by it. I'm no expert at DxO scores but ranking Sigma 120-400 lens ahead of some of the Nikon lenses clearly shows that real world results and lab tested results are very different.

Lenses are a different matter -- DxO are the leading tester when it comes to sensors, but there are a number of sources who test lenses and appear to do a better job than DxO. Having said that, I wouldn't necessarily trust the popular opinion at a camera "rumors" site (which tend to be heavily stacked with "fans") over an objective reviewer.

+1 ... actually several people at nikonrumors echo similar sentiment, recently DxO ranked some Sigma and Tamron zoom lenses ahead of some of the better Nikon zoom lenses ... also the Nikkor 70-200 f/4 scores higher than Nikkor 70-200 f/2.8 ... lot of the Nikon users are not very thrilled by it. I'm no expert at DxO scores but ranking Sigma 120-400 lens ahead of some of the Nikon lenses clearly shows that real world results and lab tested results are very different.

Lenses are a different matter -- DxO are the leading tester when it comes to sensors, but there are a number of sources who test lenses and appear to do a better job than DxO. Having said that, I wouldn't necessarily trust the popular opinion at a camera "rumors" site (which tend to be heavily stacked with "fans") over an objective reviewer.

Yes "Lenses are a different matter" but lenses are an important part of a DSLR, no lens, no photo! -- no one is disputing DxO being a leading tester of sensors, however real world photography requires lenses, but when a "leading tester" like DxO easily screws up on testing lenses, I too wouldn't necessarily trust "the leading tester's opinion, especially when what I need is a Camera and a lens to take real world photos (as I do not shoot brick walls and lab controlled environments ... my world is dynamic with light conditions changing from one moment to another and one place to another, thus requiring a camera and lens that can do the job I need, not just a camera which DxO says has a better sensor). By the way, the "rumors" site you refer to happens to have several intelligent and accomplished photographers from whom I have learned a lot, I know some folk would like to call them "fans" (in a manner that is not entirely respectful). DxO tests have not helped me to make better photos and neither did it answer any of my questions about photography, however the "fans" at CanonRumors have helped out on numerous occasions, it is a place where I come to learn, seek advice from people more skilled than I am, share ideas and if possible, once in a while assist someone who knows less than I with a little advice that hopefully is helpful so for me DxO test results are irrelevant, in saying that I mean no disrespect to DxO or the people who like/follow DxO ... However, I do engage in arguments with people who keep insisting that everyone must accept the DxO test results as the absolute fact As I said before, a good image requires a camera and lens when companies like DxO or any other company gives out highly questionable test results, I do not necessarily refer to them as "objective".

Yes "Lenses are a different matter" but lenses are an important part of a DSLR, no lens, no photo! -- no one is disputing DxO being a leading tester of sensors, however real world photography requires lenses, but when a "leading tester" like DxO easily screws up on testing lenses, I too wouldn't necessarily trust "the leading tester's opinion, especially when what I need is a Camera and a lens to take real world photos

On this we agree -- I don't give much weight to their opinions, but thankfully they do a good job at publishing their measurements which I can read and understand.

Your point that the sensor is merely part of the camera which is merely part of the camera system is understood and well taken. That is why we don't just have sensor benchmarks -- lens reviews (measurements and qualitative/subjective reviews) as well as reviews of the bodies themselves (both the dpreview approach and more subjective testing) all factor into appraising gear.

So I'm not trying to say that the sensor is everything, but I am pointing out that DxO do a pretty good job at benchmarking sensor performance. I also find hard cold numbers to be a refreshing alternative to exaggerated claims from manufacturers, fans, or new users trying to justify their purchase after the fact (instead of doing due diligence before the fact).

Quote

However, I do engage in arguments with people who keep insisting that everyone must accept the DxO test results as the absolute fact

Well, their measurements (unlike exaggerated claims from fans) are at least objective. You can argue about what the measurements mean (but even that requires some understanding, um, of what the measurements mean)

Quote

As I said before, a good image requires a camera and lens

this we agree on

Quote

when companies like DxO or any other company gives out highly questionable test results, I do not necessarily refer to them as "objective".

I don't see how this is related to the first point. Regarding questionable test results, it's not at all clear that DxO's all-over-the-map lens numbers point to any bias (lots of variance but not much bias! but it does shed some light on why their lens tests don't have the same stature as their sensor benchmarks). It's like seeing a leading quarterback play a bad golf game and then wonder if they really are a good quarterback. DxO's sensor benchmarks and methods have been studied and analyzed ad-nasueum and have stood up to scrutiny pretty well.

Yes "Lenses are a different matter" but lenses are an important part of a DSLR, no lens, no photo! -- no one is disputing DxO being a leading tester of sensors, however real world photography requires lenses, but when a "leading tester" like DxO easily screws up on testing lenses, I too wouldn't necessarily trust "the leading tester's opinion, especially when what I need is a Camera and a lens to take real world photos

On this we agree -- I don't give much weight to their opinions, but thankfully they do a good job at publishing their measurements which I can read and understand.

Your point that the sensor is merely part of the camera which is merely part of the camera system is understood and well taken. That is why we don't just have sensor benchmarks -- lens reviews (measurements and qualitative/subjective reviews) as well as reviews of the bodies themselves (both the dpreview approach and more subjective testing) all factor into appraising gear.

So I'm not trying to say that the sensor is everything, but I am pointing out that DxO do a pretty good job at benchmarking sensor performance. I also find hard cold numbers to be a refreshing alternative to exaggerated claims from manufacturers, fans, or new users trying to justify their purchase after the fact (instead of doing due diligence before the fact).

Quote

However, I do engage in arguments with people who keep insisting that everyone must accept the DxO test results as the absolute fact

Well, their measurements (unlike exaggerated claims from fans) are at least objective. You can argue about what the measurements mean (but even that requires some understanding, um, of what the measurements mean)

Quote

As I said before, a good image requires a camera and lens

this we agree on

Quote

when companies like DxO or any other company gives out highly questionable test results, I do not necessarily refer to them as "objective".

I don't see how this is related to the first point. Regarding questionable test results, it's not at all clear that DxO's all-over-the-map lens numbers point to any bias (lots of variance but not much bias! but it does shed some light on why their lens tests don't have the same stature as their sensor benchmarks). It's like seeing a leading quarterback play a bad golf game and then wonder if they really are a good quarterback. DxO's sensor benchmarks and methods have been studied and analyzed ad-nasueum and have stood up to scrutiny pretty well.

I think you misunderstood my comment about "questionable test results" to mean "biased", I am not accusing DxO of being biased ... my point is that DxO does their testing in a certain controlled environment, lab or whatever but when we make photos in real world scenarios there are far too many variables that are simply not in the realm of any one company to accurately predict and test ... Let me give u a real world scenario I face on a regular basis, I make photos in the Middle East where harsh desert environment and unpredictable dust storms are constantly present, plus I have to deal with massive cranes (at our land rigs) that kick up dust so bad that it feels like I'm in the middle of a dust storm ... when I go to the offshore rigs it is water splashes and humidity I have to deal with and the people I photograph do not wait to pose for me, coz they are extremely busy drilling crew who have no time to stop and pose, plus I have to be mindful of not getting in their way. So in my case my needs are for a robust camera and lens combos that can also quickly auto focus and get the job done ... I do not need or care about a camera with a sensor that has scored higher and/or something that is going to help me get 2 or more stops of additional DR etc ... and I'm pretty certain DxO (no matter how objective or accurate they might be) is not going to test the camera in the conditions I face coz it is just not possible for them to replicate millions of variable conditions. But we get some DxO fans who, without knowing my needs, (and the needs of other photographers) comment saying Nikon D5200 sensor has scored higher than 5D MK III, therefore it is a better camera and that I must accept their "facts" else I am an ignorant person etc (BTW I am not accusing of of that). So just like me there are lots of people in CR who do not care about DxO test results coz people have been making great photos before DxO came along and will continue to make great photos even after they are gone. Therefore, regardless of DxO "trashing the Leica M9 sensor" (as I said in my first post on this subject) a good camera does not become bad bcoz of what DxO says ... so those who know why they need the Leica M9 will buy it regardless of some company "trashes its sensor". Its too bad that some DxO fans think that it is "foolish/stupid" to buy an M9 without ever using it, just bcoz DxO said so. But no disrespect to your faith in DxO.

Yes "Lenses are a different matter" but lenses are an important part of a DSLR, no lens, no photo! -- no one is disputing DxO being a leading tester of sensors, however real world photography requires lenses, but when a "leading tester" like DxO easily screws up on testing lenses, I too wouldn't necessarily trust "the leading tester's opinion, especially when what I need is a Camera and a lens to take real world photos

On this we agree -- I don't give much weight to their opinions, but thankfully they do a good job at publishing their measurements which I can read and understand.

Your point that the sensor is merely part of the camera which is merely part of the camera system is understood and well taken. That is why we don't just have sensor benchmarks -- lens reviews (measurements and qualitative/subjective reviews) as well as reviews of the bodies themselves (both the dpreview approach and more subjective testing) all factor into appraising gear.

So I'm not trying to say that the sensor is everything, but I am pointing out that DxO do a pretty good job at benchmarking sensor performance. I also find hard cold numbers to be a refreshing alternative to exaggerated claims from manufacturers, fans, or new users trying to justify their purchase after the fact (instead of doing due diligence before the fact).

Quote

However, I do engage in arguments with people who keep insisting that everyone must accept the DxO test results as the absolute fact

Well, their measurements (unlike exaggerated claims from fans) are at least objective. You can argue about what the measurements mean (but even that requires some understanding, um, of what the measurements mean)

Quote

As I said before, a good image requires a camera and lens

this we agree on

Quote

when companies like DxO or any other company gives out highly questionable test results, I do not necessarily refer to them as "objective".

I don't see how this is related to the first point. Regarding questionable test results, it's not at all clear that DxO's all-over-the-map lens numbers point to any bias (lots of variance but not much bias! but it does shed some light on why their lens tests don't have the same stature as their sensor benchmarks). It's like seeing a leading quarterback play a bad golf game and then wonder if they really are a good quarterback. DxO's sensor benchmarks and methods have been studied and analyzed ad-nasueum and have stood up to scrutiny pretty well.

I think you misunderstood my comment about "questionable test results" to mean "biased", I am not accusing DxO of being biased ... my point is that DxO does their testing in a certain controlled environment, lab or whatever but when we make photos in real world scenarios there are far too many variables that are simply not in the realm of any one company to accurately predict and test ... Let me give u a real world scenario I face on a regular basis, I make photos in the Middle East where harsh desert environment and unpredictable dust storms are constantly present, plus I have to deal with massive cranes (at our land rigs) that kick up dust so bad that it feels like I'm in the middle of a dust storm ... when I go to the offshore rigs it is water splashes and humidity I have to deal with and the people I photograph do not wait to pose for me, coz they are extremely busy drilling crew who have no time to stop and pose, plus I have to be mindful of not getting in their way. So in my case my needs are for a robust camera and lens combos that can also quickly auto focus and get the job done ... I do not need or care about a camera with a sensor that has scored higher and/or something that is going to help me get 2 or more stops of additional DR etc ... and I'm pretty certain DxO (no matter how objective or accurate they might be) is not going to test the camera in the conditions I face coz it is just not possible for them to replicate millions of variable conditions. But we get some DxO fans who, without knowing my needs, (and the needs of other photographers) comment saying Nikon D5200 sensor has scored higher than 5D MK III, therefore it is a better camera and that I must accept their "facts" else I am an ignorant person etc (BTW I am not accusing of of that). So just like me there are lots of people in CR who do not care about DxO test results coz people have been making great photos before DxO came along and will continue to make great photos even after they are gone. Therefore, regardless of DxO "trashing the Leica M9 sensor" (as I said in my first post on this subject) a good camera does not become bad bcoz of what DxO says ... so those who know why they need the Leica M9 will buy it regardless of some company "trashing its sensor". Its too bad that some DxO fans think that it is "foolish/stupid" to buy an M9 without ever using it, just bcoz DxO said so. But no disrespect to your faith in DxO.

I think you misunderstood my comment about "questionable test results" to mean "biased", I am not accusing DxO of being biased ... my point is that DxO does their testing in a certain controlled environment, lab or whatever but when we make photos in real world scenarios there are far too many variables that are simply not in the realm of any one company to accurately predict and test

Well, yes, of course -- they are in the business of measuring sensor performance, no more, no less.

This -- sensor performance -- is a really big deal to some people. I agree that for some shooting scenarios it may not be terribly important. (Actually, for what you do, some of digitalrev's reviews are probably more relevant!)

Quote

But we get some DxO fans who, without knowing my needs, (and the needs of other photographers) comment saying Nikon D5200 sensor has scored higher than 5D MK III, therefore it is a better camera and that I must accept their "facts"

Well, that's horribly ill-informed on their part. But I haven't seen these fans (are you sure they're not Nikon "fans" who would drop DxO like a bad habit if Canon started scoring higher than Nikon ?)

Anyway, the DxO score only tells us that the D5200 scored higher on the aggregate score, which in this case means that the sensor has less noise at low ISO. It says very little else about the relative merits of the two cameras.

for some shooting scenarios it may not be terribly important. (Actually, for what you do, some of digitalrev's reviews are probably more relevant!)

Quote

Nah, digitalrev reviews are not really my thing, when I do watch it more for KAI's comedy and to check their store, coz for gear (sometimes) their prices are cheaper and they don't charge shipping to middle east and they also take care of customs.A few years ago, I used to rely on the-digital-picture for reviews, these days I prefer the reviews from lens rental

(are you sure they're not Nikon "fans" who would drop DxO like a bad habit if Canon started scoring higher than Nikon ?)

Quote

Ha ha ha ... quite possible By the way, although my primary gear is 5D MK III and a few Canon lenes, I am somewhat of a Nikon fan as well, I always try to keep at least some Nikon gear, currently it is the D7000+18-300 VR ... will be upgrading if D400 comes out, otherwise I'll upgrade to D7100 when there is a sale or if a refurbished one shows up for a good deal.

DXO mark means nothing. They take their objective tests and interpret them in completely subjective ways, all (as it would seem) on Nikon's dime. Super biased, all rubbish, especially their ISO scores. Canon should be top of the mark for every camera they've put out in the ISO division, right next to Pentax, and Nikon and Sony should be right in the bin every time. But it's absolutely the opposite. If you've ever seen the DPReview studio comparison tool, I highly recommend cranking up the ISO on there and seeing for yourself what I'm talking about. D5200? Rubbish. D800? Rubbish at anything above or below ISO 200. 5DIII? A f@ck!ng mint. DXO scores the opposite in every case.