I just typed some comments on the sigma 24-105, parts of which have some relevance to your question, a quick cut and paste:

Count me super dubious of DxO when it comes to lenses.

First, why do they rank lenses based solely upon best aperture and focal length, what sense is there in that?
Second, since they seem to rate so many zooms best at wide open, like, just for starters here, they claim the 24-70 f/4 IS does best at 24mm and f/4? What are they measuring like the 4 center most pixels in the frame? Because I know that none of the lenses they'ved claimed best at wide open have done so for me in my tests or real world or on basically any other test site.

And they have so many weird results like: That the 16-35 II has the crispest edges on FF at f/2.8. That the 70-200 2.8 IS II is the worst performer at 200mm f/2.8 of all the 70-200 f/2.8 lenses. That the 70-300 non-IS has better 300mm than the 300 f/4L and 70-300L. And so on.
In fact they claim that every single one of those zooms perform the very best at wide open.

That seems impossible, so maybe for their overall score they simple try to find where each zoom does best at wide open and don't count anything closed down?

But how does that make any sense at all?

Their overall sensor scores were slightly dubious, but their overall lens scores are utterly absurd. Ranking lenses by the spot they are sharpest at wide open and ignore every single other aspect? it doesn't even begin to make any sense. What if one lens happens to do well wide open at one extreme but does barely better stopped down and stinks over the entire rest of the range? Who cares, rank it #1.

I'll also note that you can get the canon 24-70 f/4 IS from Adorama at the moment (after mail in rebate) for the exact same price as this Sigma and the Canon uses less expensive filters and is oodles smaller and lighter.

Also note, that in their best wide open test, the new tamron and new canon lenses do best at 24mm, which was always the real catch when it came to these lenses on FF, THAT was the trickiest part of the focal range while the sigma does not, according to their tests at least, do best at 24mm, but at 35mm (as with the 24-105L) where it has always been easier to find somewhat better performance at FF.

The real trick was always to find something that could handle 24mm crisp edge to edge on FF for landscaps. All the old standard zooms and even some of the old regular primes struggled a bit with that at 24mm on FF. The magic of the new tamron and new canons is that they finally deliver improved 24mm performance on FF in a zoom (or even in a lens just regardless).

So they can have a lens do best wide open at the easiest spot to do well and perhaps even stink everywhere else or perhaps at the focal lengths you care much more about or apertures you care more about and yet it might be ranked dozens of lenses higher than some other lens that might do better at a tricky focal length, etc. etc.

And when you compare across brands you are using different sensors so, as all the other test sites say, you can't use the scores to compare lenses across systems.

ratsnest74 wrote:
ok, well who is the better source for such ratings? I hear so many people on this Canon forum saying Canon has the better lenses and I believed this until a Nikon shooter pointed this site out to me.

It's because they get lots of weird lens results in general and then on top they use an extremely dubious method for giving each lens it's overall score. Lens A could do a little better at 50mm f/4 and far worse than lens B at 24-40mm f/4 and 60-200mm f/4 and worse across the board at f/7.1 and have more distortion and more CA and yet it would score higher on their overall ranking.

And in this case you are comparing not just different lenses on the same system but on different systems, which is not valid if you try to talk about the lenses themselves, at that point you need to talk about entire systems. Nikon may have weaker AA filters and more MP.

Hah so they test lenses in dim conditions. Those MTF programs are dicey as it is and then they toss it noisy images.

Basically they are really test lens+sensor combinations but present the data, at first glance, to seem more like pure lens tests.

Also considering that under tricky lighting it's so often you get a hint of residual blur and all sorts of noise entering in and that pixel peeping sharpness is usually not the key for such shooting, I find it ultra dubious that they decide to test 'lenses' and score then overall only in dim, noisy conditions, wide open, the very conditions when pixel peeping micro-contrast is usually the least concern I'd guess.

Hmm but now I see a few primes where they rank them best at other than wide open so maybe they don't only go by wide open??

I don't get it in any way shape or form. They say the zeiss 50mm 1.4 performs best at f/2.8 (I'm dubious) and that the canon 50 1.8 performs best at f/1.8 (i'm very dubious) and then they compare the two lenses in their overall score table with the zeiss set to f/2.8 and the canon set to f/1.8.

A "sensor outresolving" a lens or vice versa is not a black and white concept. Both have a decrease in "resolving power" (MTF) as the target details get smaller and smaller (with increasing spatial frequency). While the shape of that curve is fairly predictable from sampling theory for a sensor, different lens designs will show quite distinct curves which vary quite a lot. Check out around page 24 of this excellent pdf from Zeiss Very good contrast at low spatial frequency (the overall image) may or may not be accompanied by good contrast at medium or very high spatial frequencies. Similarly, there may be very high frequency detail captured, but lower than average contrast overall (the 400DO is a good example - a little PP shows the fine detail is captured but the lens design has a somewhat lower contrast at lower frequencies) TThis article has some good illustrations

Those who hate (or can't interpret) charts will say things like "good or bad microcontrast" or "fine detail rendering" or "three dimensional look" etc, which is hard for others to use for comparative purposes, but which reflect these variations - check out the concept of SQF - subjective quality factor.

For those who do grok charts, the issue is illustrated on one of Norm Koren's tutorials where the "system resolution" drops as the product of the sensor and lens. (The Zeiss pdf has some good illustrations too.

Bottom line is that their "let's reduce it to one number" approach is kind of pointless, while some of the intermediate data can be useful to those who can decode it. I suspect they could make MTF vs frequency charts for each lens if they chose to. Then people could learn how to interpret them just as they do the Canon (theoretical but consistently calculated) lens MTF charts (or even off-axis frequency response for speakers!) They could even use the existing metric of SQF. I have a science background and could interpret this stuff but DxO makes it too hard for it to be worth the effort...

Scott Stoness wrote:
It has been said before but I will repeat it:
...
4) Currently my view is that Canon has best tilt shift lens. Nikon does not even have a 17ts. And I think the 5diii is the best all round camera. And Canon has the best (newest design) long lens [600v2, 500v2, 200-400]. The d800 is not great for wildlife. So so might say Canon is best (Ts17, TS24, 600v2, 500v2, 200-400) and some might say Nikon is the best (d800 with 14-24) and some might say sony is best with zeiss lens and a7r. So which is best depends on what you are shooting. For me Canon is best for wildlife. And best for me for landscape (TS17)....Show more →

It's often risky to have such a definitive judgement. I could provide you links to wildlife photos shot with a D800 that are amazing (in latin sense !).

BTW I'm not interested in quibble or local quarrel. I started photo with Minolta, entered digital world with Nikon and now I'm mainly a Canon shooter, still having some Sony and Nikon hardware.

I agree that the desire to quantify lens performance with a single, scalar number is often misguided, but the "old" photodo site used their (subjective) mean weighted MTF index to good effect. Virtually any lens with a high wMTF has very high resolution. I used wMTF .GE. 4.0 as a personal guide in my "search for the perfect lens, at a bargain price". Of course, a high wMTF score was only part of the picture. Many other considerations are important, but it's a good start. OTOH, a poor score at photodo did not necessarily mean that the lens was a dog. In some cases, it can be desirable to have a lens with lower resolution (e.g. some portrait and abstract/impressionist applications). Also, since (in most cases, at least) only a single copy of each lens was tested, it stands to reason that some were sub-par. Again, it's just one piece of evidence.

Thanks - a wMTF sounds useful. To your point it could available in two flavours such as A weighted and B weighted with weightings appropriate to, say, landscape (higher frequencies get higher weighting) and, umm, portrait!

skibum5 wrote:
Hmm but now I see a few primes where they rank them best at other than wide open so maybe they don't only go by wide open??

I don't get it in any way shape or form. They say the zeiss 50mm 1.4 performs best at f/2.8 (I'm dubious) and that the canon 50 1.8 performs best at f/1.8 (i'm very dubious) and then they compare the two lenses in their overall score table with the zeiss set to f/2.8 and the canon set to f/1.8.

I mean what the what the what are they doing?

I understand your feelings and I felt similar for tests of other lenses. Now maybe the bias was a defective product and thus it's better to not stop to a single source. But if bias was in the DxO's procedure, it should be eliminated with comparisons, shouldn't it ? And finally, couldn't we simply discard their "not so usefull" overall score and zero in on their raw data to make our own overall score ? Let's be smarter 'cause it's time to learn how to fish (no offense intended)!

Larate wrote:
It's often risky to have such a definitive judgement. I could provide you links to wildlife photos shot with a D800 that are amazing (in latin sense !).

BTW I'm not interested in quibble or local quarrel. I started photo with Minolta, entered digital world with Nikon and now I'm mainly a Canon shooter, still having some Sony and Nikon hardware.

I could likely use my a7r with an adapter on my 600 v2 and get a good picture of a wolf. But I would not do it because its rare to get a good pose and I would not want to miss it with a slow focus or slow fps. So you are right that the d800 can make great wildlife pictures but it would not be more first (5diii), 2nd (1dx or D4) or 3rd choice (1div). Just like I would not choose a 1dx to focus on landscape when the 5diii or 6d or d800 or d600 have better resolution.

Scott Stoness wrote:
I could likely use my a7r with an adapter on my 600 v2 and get a good picture of a wolf. But I would not do it because its rare to get a good pose and I would not want to miss it with a slow focus or slow fps. So you are right that the d800 can make great wildlife pictures but it would not be more first (5diii), 2nd (1dx or D4) or 3rd choice (1div). Just like I would not choose a 1dx to focus on landscape when the 5diii or 6d or d800 or d600 have better resolution. ...Show more →

Once again, you've made a definitive judgement*. Even if I don't agree with it**, I really prefer your opinion given in your last sentence.

* Although I understand it's your opinion, it's only to tell you that some "not knowledgeable" readers might take it as a set in stone fact. That was the sole purpose of my messages.

** Incidentally, I tend to prefer my 1Dx instead of my newly 5D3 for landscape photos. Photos shot with my 1Dx have less noise, require less sharpening than the ones shot with my 5D3 (talking about Raws). And the difference in resolution doesn't yield so much difference in the end (meaning when photos are printed). Of course, it's only my point of view.

jcolwell wrote:
I prefer my 6D for landscapes, but the 1DX will do, in a pinch.

Not enough expensive my son !*

* clin d'oeil (reference) to an old French TV ad made for Renault, a car manufacturer. This advertisement was for the Clio, a small car that had everything but a high price. And the rich man in the ad didn't want to purchase it for its son only because of that : "pas assez cher mon fils" ! Could we say the 6D is the "Clio" of Canon ?!!! :')

Skibum nailed it right on the head.
And people place faith in DXO ratings is the scary thing as many dont know any better.
Roger C's blog has a good post about why basing lens ratings on sensir res is basically as flawed a testing method as you can have. Yet the self proclaimed leaders in testing tech(in there own minds anyway) continue to plow on with pathetic testing proving time and again how poor the Canon super tele range is easily outperformed by the consumer lenses.
Yet people continually place DXO as some kind of bench mark...a company which rates cameras without an AF test or build deserves zero credibility and thats before you analyze there flawed methodology.
DXO=Waste of time...

skibum5 wrote:
or photozone.de for something the most accurate (or best that plus a wide variety of forum and blog comments and lens rentals the few times they test lenses)

I think Lenstip is good too. In my opinion :

+ The plus of photozone is that they test the lenses on the same body (e.g all Canon-mount FF lenses are tested on a 5D2 (*)). They also have tested the Canon's TS-E lenses (although the older ones were tested on 350D only).
And they give an overview of the lenses' bokeh and LoCA / purple fringing.

+ The plus of lenstip is that they're often the first to test new lenses (e.g EF 35/2 IS) or even the only ones (e.g Samyang TS-E 24). Unlike photozone, they have tested the EF 24/2.8 IS and EF 300/2.8 L IS II (I'm interested in these lenses) but not the Zeiss Distagon 21/2.8 (I'm interested in too !)
And their tests are among the most complete (coma & astigmatism, ghosting & flare, focusing, build quality / handling & IS).

* Lenstip now test the Canon-mount FF lenses on a 1DS3, but older tests were made only on APS-C, like the EF 70-200/4 L IS.

dehowie wrote:
Skibum nailed it right on the head.
And people place faith in DXO ratings is the scary thing as many dont know any better.
Roger C's blog has a good post about why basing lens ratings on sensir res is basically as flawed a testing method as you can have. Yet the self proclaimed leaders in testing tech(in there own minds anyway) continue to plow on with pathetic testing proving time and again how poor the Canon super tele range is easily outperformed by the consumer lenses.
Yet people continually place DXO as some kind of bench mark...a company which rates cameras without an AF test or build deserves zero credibility and thats before you analyze there flawed methodology.
DXO=Waste of time......Show more →

Don't throw the baby with the bathwater ! :')
Between the blind pros and the fundamentalist antis, there's enough room for the wise men !