Author
Topic: Dxo tests canon/nikon/sony 500mm's (Read 35692 times)

If transmission really was the most important factor then the Sony should be out scoring the Nikon. The Sony equals or beats the Nikon in all the listed categories and has the lowest transmission of the three lenses.

Transmission on macro scale, not micro. In fact, probably not even measured transmission, but rather the specified max aperture. That's why the 50/1.8 lenses outscore the 500/4 lenses.

If you look at the test results they are measuring the combination of lens and body. Hard to do otherwise. Makes comparison between manufactures difficult and less than meaningful.

Yes, and sorry, but I think you are missing the point. For every measurement, all generated with a body attached, the Canon lens comes out on top, in some cases by a significant margin. Yet, the Score is a tie. So...the score is fabricated, pulled from their nether orifices, etc.

If you go over to photozone.de's page, they'll quite clearly state that the measurements (or scores) from one camera system (make + sensor size) cannot be compared with another. Given this I can't see why the same wouldn't be true for DxO. What does that mean? That you cannot compare a score of 25 for the Nikon lens with a score of 25 for the Canon or 22 for the Sony lens.

To be able to compare each lens properly would require each lens being mounted on the same camera.

How do they calculate their lens score? Would love to know but I'm pretty sure that it is corporate secret. For the Nikon one to be so high must mean that the readings are somehow weight on the sensor (e.g pixel size.)

The point is that they measure several parameters of optical image quality from the lens, such as sharpness, transmission, distortion, vignetting, and CA. They could generate a Lens Score based on those parameters, but they don't. Had they done so, the Canon 500/4 II would have soundly trounced the Nikon 500/4.

For those who argue that it's reasonable that DxO consider the camera in the 'Lens Score', note that the sensor is already factored into the measurements themselves. P-Mpix measures sharpness of camera + lens, pixel size affects CA, etc. Even their transmission measurement changes with different cameras. So by factoring in the camera directly in the measurements (reasonable) then factoring it in again in the overall score where it's given an undisclosed (but evidently very significant) weighting, means their 'Lens Score' is as much if not more a camera score than a lens score.

One more point - considering just their P-Mpix measure of sharpness, by their definition the Nikon 500/4 results in a loss of more that 55% of the resolution of which the D800 sensor is capable, whereas the Canon 500/4 II only decreases the 5DIII's potential resolution by less than 14%. Of course, the Canon lens also outresolves the Nikon by an absolute assessment, even taking the higher resolution D800 sensor into account. But they get the same 'Lens Score'. Right.

They should rename their Overall Scores to a Camera Basic Score and a Lens Basic Score, so we could abbreviate them for what they really are: BS. Actually, that's probably giving them too much credit, because real Bovine Scat makes good fertilizer, whereas DxOMark's BS has no real-world utility.

It does precisely what it is supposed to do, generate a lot of views for the DXO site where they advertise and sell their software. This is evident from this and other threads. It costs them very little since they need to test the gear to make their software anyways.

For those who argue that it's reasonable that DxO consider the camera in the 'Lens Score', note that the sensor is already factored into the measurements themselves. P-Mpix measures sharpness of camera + lens, pixel size affects CA, etc. Even their transmission measurement changes with different cameras...

...They should rename their Overall Scores to a Camera Basic Score and a Lens Basic Score, so we could abbreviate them for what they really are: BS. Actually, that's probably giving them too much credit, because real Bovine Scat makes good fertilizer, whereas DxOMark's BS has no real-world utility.

Your assertions of DxO publishing BS are just as BS as their results because you don't know how they calculate their "lens score" and thus you have to make up reasons as to why it is so. Yes, you're compounding the BS by stating that they're publishing BS. Just don't buy into it.

Except I'm not scoring lenses without disclosing my methods. In my scientific publishing, I must fully disclose my methods such that another scientist can duplicate my experiment. DxOMark is clearly not held to the standards of peer-reviewed scientific publications. (Which is reasonable, as they're not publishing in scientific journals, but only to their own website - so, they can make up 'data' or 'scores' if they choose, change those data post hoc without explanation, whatever they want.)

We don't know what's in their black box? They don't say. Why not? If someone is hiding something, it's usually for a good reason. Could be as simple as maintaining a competitive advantage (against who, I have no idea).

why would i care what the test results of a lens would be mounted to a camera i would never shoot. i would rather see test results from a lens/body combo that i could actually use.

Because lenses, especially one like this, will probably be used on new bodies for the next 10 years or so. It will far outlast the current "best" body of a brand, possibly by several generations. [...]

So my answer is yes: to the greatest degree feasible, a "lens test" should isolate the lens, even mounting the competitors on the same body if possible. (of course that's difficult, but that's what I'd like to see)

But there is still a problem: if you have a pure lens test, are you sure you know how to compute how it will perform on a future, say, 50mp body?

why would i care what the test results of a lens would be mounted to a camera i would never shoot. i would rather see test results from a lens/body combo that i could actually use.

Because lenses, especially one like this, will probably be used on new bodies for the next 10 years or so. It will far outlast the current "best" body of a brand, possibly by several generations. [...]

So my answer is yes: to the greatest degree feasible, a "lens test" should isolate the lens, even mounting the competitors on the same body if possible. (of course that's difficult, but that's what I'd like to see)

But there is still a problem: if you have a pure lens test, are you sure you know how to compute how it will perform on a future, say, 50mp body?

MTF charts are, for all intents and purposes, "pure lens tests". They already give us a way to compare lenses across the board, brands be damned. Simple fact of the matter is a better lens will perform better on ALL sensors, 20mp, 30mp, or 50mp. The problem with DXO's tests is they quite simply don't give you a reasonable camera-agnostic basis from which to compare lenses. The Nikon 500/4 performs "on par" (toung in cheek) with the Canon 500/4 solely because of the higher resolution sensor. That sort of tells you that the Canon lens is particularly good, because it is performing so well on a worse sensor...but you don't really have any exact way of comparing. You only get a "feeling" that it performs so well.

But there is still a problem: if you have a pure lens test, are you sure you know how to compute how it will perform on a future, say, 50mp body?

You don't; however, you do know that a better lens will perform no worse, and likely better, on any future camera than would a lens that performed worse in a "pure" lens test. Though not perfect, it's superior to the tainted lens+body test.

MTF charts are, for all intents and purposes, "pure lens tests". They already give us a way to compare lenses across the board, brands be damned.

If memory serves me well (no guarantee here) most manufacturers use computed (theoretical) MTF charts that are derived from the design of the optics. A few (?Zeiss?) actually test production copies. I hope someone will correct me if I'm mistaken. Yes, it would be great to see MTF charts using data from real, production copies.

This comparison of DXO's results is what makes me EXTREMELY SUSPICIOUS of them. The Nikkor 500mm scores 25, when every single trait that factors into that score is worse than the Canon 500mm...which also scores 25. That is just plain wrong. The Canon has zero distortion, higher sharpness, less vignetting, less CA, and the same transmission...on a LOWER RESOLUTION BODY! It should have a higher score than the Nikon.

But there is still a problem: if you have a pure lens test, are you sure you know how to compute how it will perform on a future, say, 50mp body?

You don't; however, you do know that a better lens will perform no worse, and likely better, on any future camera than would a lens that performed worse in a "pure" lens test. Though not perfect, it's superior to the tainted lens+body test.

But a better lens on, say, the 5D3, will preform better on the 5D4, you know that for sure. Again, where is the problem exactly?

Simple fact of the matter is a better lens will perform better on ALL sensors, 20mp, 30mp, or 50mp. The problem with DXO's tests is they quite simply don't give you a reasonable camera-agnostic basis from which to compare lenses.

Actually, they do. There is a way to extract the pure lens resolution from the data they used to publish (full MTF curves, not the nonsense they publish now).

Quote

The Nikon 500/4 performs "on par" (toung in cheek) with the Canon 500/4 solely because of the higher resolution sensor. That sort of tells you that the Canon lens is particularly good, because it is performing so well on a worse sensor...but you don't really have any exact way of comparing. You only get a "feeling" that it performs so well.

Why in the world would you want to know how a Canon compares to a Nikon without a body? For bragging rights? They tell you what is achievable with the current bodies on which the lens works, the way it is deigned to work. A better lens on one body will be better on future bodies as well.