BTW, one major university with a substantial collection of contemporary photography told me about a year ago that they were not accepting raw files from photographers work that they were collecting unless they were in DNG. Reason? They feared the inability for future generations to be able read the data.

They have a program in place to migrate over time to new storage formats (that's a given) but they believed (rightly or wrongly – and I think rightly) that they didn't believe and trust in proprietary formats for archival and historical storage purposes. They also told me that a national consortium of museum conservators were planning on making DNG their defacto standard across institutions.

So, my personal observation is... where's the beaf?

No beef. Good factual information. I understand their reasons for it. Good thing, that for their uses, that DNG is available.

However, that is not the crux of the disagreement. The major disagreement, in my mind, is that we are all expected to form a consensus that DNG needs to be integrated into all cameras. This is without DNG, yet, being an industry standard and without any understanding, or willingness to discuss, the possible ramifications to the camera manufacturers.

If one does not agree or is not silent, they are summarily attacked, without any regard for reasoned arguments by the person not agreeing. The "by-words" are that by not agreeing we are giving the manufacturers a "pass" and that "if not a part of the solution, you are a part of the problem".

I am willing to agree to disagree. However, would request that, if the issue is promoted again, that we can retain the right to again present our views.

We also should realize that much (most?) of the turmoil is caused by companies like Adobe who, for commercial reasons, force people to update the image editing software, because newer camera models are deliberately locked out. It is therefore no surprise that Adobe also offers a 'DNG solution' (for an issue they partially create themselves), which would benefit them by not having to interpret multiple Raw formats.

Your statement "for commercial reasons, force people to update the image editing software" is not correct...while there may indeed be a component of "commercial reasons" in Adobe's policy, there are substantial technical reasons as well. Aside from the Adobe policy of not updating software they no longer sell, the way software code is developed forces that sort of line in the sand. Both Camera Raw and Lightroom are software platforms that have substantial system level dependancies...

Camera Raw is a plug-in that lives inside the Photoshop plug-in SDK. Each version of Photoshop has it's own SDK that is dependent on the OS level APIs and services. Trying to maintain backwards compatibility of current and future ACR versions inside of past versions of Photoshop would require branching the code into current OS system support and previous OS system support. For Windows, that might not be an onerous burden, but for Mac it would be almost impossible because of fundamental and massive changes Apple has forced on developers over the years. Camera Raw first showed up in Photoshop 7. It would be impossible to port the current ACR to be able to work in Photoshop 7. In Photoshop CS3, Adobe had to migrate from Codewarrior to Xcode to support Universal Binaries that provided PPC/Intel dual support. In System 10.7, support for Rosetta was removed so nothing on OS 10.7+ can run any PPC code.

Just how would you suggest Adobe update previous versions of Camera Raw for current and future as yet, unreleased cameras? Reverse engineer them and go back to that original code and rewrite them to include these new cameras? How many versions of Photoshop and ACR should Adobe offer backwards compatibility for? One version? Three? How many code branches of Camera Raw should Adobe spawn off?

Looking at Lightroom, if Adobe wanted to offer backwards compatibility for previous versions of LR to open new and future cameras, they would have to maintain code branches for each and every version of LR that needed backwards compatibilities...

So, Thomas developed DNG as a method of providing backwards compatibility of new and future raw files for past versions of Camera Raw and Photoshop. With the proper DNG version preferences, users of previous version of ACR and LR can access and convert new raw files for backwards compatibilities. A user that has a camera only supported by ACR 7.4 can use DNG Converter 7.4 to make a compatible DNG that will open using ACR 2.4 in Photoshop CS or LR 1. Oh, and DNG Converter is free...

So, there are indeed real technical issues and yes, Adobe adopted the policy of updating only currently shipping application versions. Yes, there were financial reasons for that policy, but also valid and substantial technical reasons. If Adobe were to adopt a policy of backwards compatibility updates to previous versions of ACR/LR, it would take developmental resources away from current and new development. So, in effect, Adobe would deprive current/future customers for the benefit of previous customers. As a current customer, I would have a problem with that.

Interestingly, when you access a native raw in ACR or LR, the initial thing the ACR pipeline does is convert the native raw data to DNG data on the fly. I don't think this is a secret as I think Eric or Thomas has mentioned this before (if it wasn't, I'm sorry Eric and Thomas :~). The conversion of native data to DNG data is fundamental to ACR/LR's ability to parse the raw image data and associated metadata such as white point, ISO, camera data and lens data. So, it is somewhat ironic that some people who don't want to use DNG actually are using DNG every time they process their natives raws in ACR/LR. It's kinda yet another feather in the cap of Thomas Knoll.

So, no, I don't think it's proper to lay the blame for the current situation regarding undocumented, proprietary raw file formats the feet of Adobe (and other 3rd parties). The blame falls at the feet of the camera makers who eschew the use of any standardized raw file format and continue to propagate new raw file formats each time they release a new camera. And the situation will not improve until the camera companies quit doing that.

Your statement "for commercial reasons, force people to update the image editing software" is not correct...while there may indeed be a component of "commercial reasons" in Adobe's policy, there are substantial technical reasons as well. Aside from the Adobe policy of not updating software they no longer sell, the way software code is developed forces that sort of line in the sand. Both Camera Raw and Lightroom are software platforms that have substantial system level dependancies...

Hi Jeff,

Of course they do, and they hopefully exploit them, and the Camera makers also do not want to maintain deprecated legacy data in new Raw formats, just for the benefit of others.

Quote

If Adobe were to adopt a policy of backwards compatibility updates to previous versions of ACR/LR, it would take developmental resources away from current and new development. So, in effect, Adobe would deprive current/future customers for the benefit of previous customers.

Quote

Interestingly, when you access a native raw in ACR or LR, the initial thing the ACR pipeline does is convert the native raw data to DNG data on the fly.

Quote

I don't think this is a secret as I think Eric or Thomas has mentioned this before (if it wasn't, I'm sorry Eric and Thomas :~). The conversion of native data to DNG data is fundamental to ACR/LR's ability to parse the raw image data and associated metadata such as white point, ISO, camera data and lens data. So, it is somewhat ironic that some people who don't want to use DNG actually are using DNG every time they process their natives raws in ACR/LR.

Which begs the question why a separate DNG conversion does support certain camera models, and a direct attempt to open them (via DNG) doesn't.

Quote

So, no, I don't think it's proper to lay the blame for the current situation regarding undocumented, proprietary raw file formats the feet of Adobe (and other 3rd parties).

Who's blaming them for that? They have no control over what the camera manufacturers need.

Quote

The blame falls at the feet of the camera makers who eschew the use of any standardized raw file format and continue to propagate new raw file formats each time they release a new camera.

You make it sound as if it's a goal, to change the Raw data that is beneficial to recreate the original signal. Could it perhaps be because technology advances and they need to adust the postprocessing possibilities to keep up?

Quote

And the situation will not improve until the camera companies quit doing that.

I hope they keep improving their own technology, in competition with others, despite the wish from some that they reach some sort of an agreement with their competitors about what to store and how. Patents make that a bit difficult.

I agree with Jeff, the primary responsibility for the problems is with camera manufacturers. They want to lock people to their systems. Their primary goal is to make money, not to give you what you want/need.

DNG is useful. I have no issue with it's versions considering it is free. "Never look a gift horse in the mouth."

It would be nice if the manufacturers gave the user the option of output formats. Raw, DNG, TIFF, FITS. I would go out of my way for a FITS capable camera.

Which begs the question why a separate DNG conversion does support certain camera models, and a direct attempt to open them (via DNG) doesn't.

Not sure what you mean...the ACR pipeline (which includes ACR, LR & DNG Converter) all share the same native raw support given the same processing pipeline version. Where have you ever seen ACR/LR fail to open a raw file that DNG could, or visa versa at a given version?

Quote

Who's blaming them for that? They have no control over what the camera manufacturers need.

You were when you said: "We also should realize that much (most?) of the turmoil is caused by companies like Adobe who, for commercial reasons, force people to update the image editing software, because newer camera models are deliberately locked out."

I sure read that as you saying the turmoil was the fault of Adobe (and other 3rd parties). And "newer camera models are deliberately locked out" is also incorrect...nobody is being directly locked out of anything except users of new cameras not being able to open new cameras until software has been updated...they've been locked out by the camera makers not using standardized, compatible raw file formats. Unless you are saying that it's the camera makers deliberately locking out their users–which I actually agree with :~)

I have no fact...only suspicions....if someone really knows, we would all be made wiser by it.....Forgetting, for the moment, the internals to the camera. Canon provides RAW conversion software, which they make no money on. However, even though they know the internals of RAW changes way before the rest of the world, they need to incorporate those changes into their code...which drives an expense, which will effect the overall cost, therefore the price, of the camera. If this cost could be saved without causing increased cost or other issues you can be sure the Canon financial people would on this like flies on stink.

Well it makes no sense to me (in the west).

Jeff Schewe's posts in the previous LR thread have given us some insight into how the Japanese camera manufacturers approach this issue and how Adobe have dealt with the situation.

Ultimately a single, good RAW file format standard would benefit everyone. I've not yet read any credible argument that disagrees with that position.

Not sure what you mean...the ACR pipeline (which includes ACR, LR & DNG Converter) all share the same native raw support given the same processing pipeline version. Where have you ever seen ACR/LR fail to open a raw file that DNG could, or visa versa at a given version?

Then what's the problem, Jeff? Is there a problem opening the (partly proprietary) file data, or isn't there? If there isn't, then why the need for conversion to DNG, and the call for Camera manufacturers to join in a unified file format?

Quote

You were when you said: "We also should realize that much (most?) of the turmoil is caused by companies like Adobe who, for commercial reasons, force people to update the image editing software, because newer camera models are deliberately locked out."

Yes, there is a group of people who blame (=turmoil) the Camera manufacturers, because they themselves would like to have an easier job, which is understandable. The camera manufacturers get no benefit in that game, it would only drive up their cost, slow down development, and create issues with patents. Let's stay realistic, it won't happen that way.

Why wouldn't it ?Did Canon* get any advantage from making different file formats from the 5d, 5DmkII and 5DmkIII ?

The point here is why change the file format for every camera model ? Does it benefit anyone ?

Hi,

First of all, it's not an entirely new file format. It's still basically a Bayer CFA sensel arrangement, with a CR2 filename extension.

There were technological advances from 12.8 Megapixels with 12-bit precision, at 3 frames per second, to 21.1 MP and 22.3MP with 14-bit precision, at a 3.9 and 6 frame rate per second readout speed which required more/multiple (parallel) readout channels which also require calibration data to be recorded. Video capture was added, which probably required more thermal calibration info to be recorded as well. It only seems logical that some changes were required in the Raw data that was recorded.

It seems to me that many users did benefit from those changes (directly in image quality, and later in competitive offerings), and it also urged the Raw converter producers to optimize their software (by utilizing the capacity of the graphic-card's GPUs).

Also, the addition of more lens and dust related metadata in the EXIF maker notes datasection, potentially allows for more targeted distortion and vignetting correction (possibly in the masked Raw data fields) and dust removal, e.g. in JPEGs and video directly from the camera.

Does 'everyone' include the camera makers? Otherwise it will not happen, even if it were possible to avoid patent issues?

Hi Bart

What patent issues?There are already use-cases where camera-makers have adopted DNG as their RAW format.Also, what patent issues are there with every camera been able to generate a JPEG?The idea is to have an ISO-standardized open RAW format.So, whether DNG is adopted as a universal RAW format or something else patent issues are the least of the problems.

And yes it does include the camera-makers.

With regard to your comment about technological advancements somehow being stymied by a standardized RAW format that doesn't wash.A RAW format is a recording medium not a generating medium.With regard to the use-cases above using DNG in several different proprietary brands of cameras and models clearly they did not feel fettered.

The point here is why change the file format for every camera model ? Does it benefit anyone ?

It's completly meaninless, but is there a parameter that escapes to our understanding?

If that non-sense is happening for now too long and put users on bondage, there must be a reason.Who it benefits? I don't know, but it should benefit to some interests.

Now, the fashion here is to archive footage in JPEG2000. I thought that in digital preservation it was "safe" and it apparentlyisn't really that much. There are concerns on JPEG2000 viability in the future. New debates are open, on air currently.

When this Raw video is going to be democratized soon and every camera maker will provide combo cams with video+still capabilities in RawI'm just about sure that very few will adopt cinema DNG and will see the same non-sense that what's happening in still for decades.The surreal part, will be how will they deal with their raw dev like RedCineX? and how those developpers will be integrated into pipelines?It's gona be a huge mess. Nothing will read nothing and DITs will have to invent oscur roundtrippings to make the machinery work.

Suppose that Canon would want to actually exercise some of their patents for tri-chromatic capture per photosite. Do you think that they would like to tip off their competitors ahead of the event? Or, if Dr. Eric Fossum (the inventor of the CMOS image sensor) produces a new (sub-micron sensel) sensor generation (which he is working on), he would want to allow competitors to jeopardize his investments by speeding up their efforts or counter-measures.

Quote

There are already use-cases where camera-makers have adopted DNG as their RAW format.

Yes, and they serve niche markets.

Quote

Also, what patent issues are there with every camera been able to generate a JPEG?

I'm not sure what you are referring to.

Quote

The idea is to have an ISO-standardized open RAW format.

Yes, I wouldn't mind. But what issue does it solve?

Quote

So, whether DNG is adopted as a universal RAW format or something else patent issues are the least of the problems.

I'll leave it to IP and patent lawyers to pass the ball between them, but I do recall issues around GIF file patents threatening everybody using the internet, and claims around IPIX panorama file formats and panorama photography in general that at the time drove Prof. Helmut Dersch out of providing freely accessible panorama solutions to everybody. That certainly stifled progress in that area of industry.

Quote

With regard to your comment about technological advancements somehow being stymied by a standardized RAW format that doesn't wash.

Why not?

Quote

A RAW format is a recording medium not a generating medium.

Doesn't the data in the Raw file determine the potential output quality?

Quote

With regard to the use-cases above using DNG in several different proprietary brands of cameras and models clearly they did not feel fettered.

To give just one example, I've corresponded with some Pentax 645D users who report significant (quality) differences/limitations between the DNG and PEF files and the various available Raw conversion alternatives/qualities.

Lets put the issue in perspective.All the patent issues that you are referring to involve analog to digital signal conversion.Once that, and all the subsequent number crunching is done, then the finished product is recorded as a RAW file of some description.There is no need for Canon, or anyone else for that matter, to show their cards.The recording medium per se does NOT have to be unique for Canon files to still show unique image characteristics, such as they are, or will be.

If these apparently unique RAW file formats can now be converted to DNG right now in Lightroom with no loss of quality what would be the problem with this happening in camera?

I am aware of the interesting differences in how Canon RAW converter handles .CR2 files compared to Lightroom or ACR, and while the differences may be partially due to secret metadata instructions in the RAW files that Adobe is not privy to, subjectively I question the benefits from an aesthetic perspective.

As for you querying the benefits of a a standardized open RAW format, if by now you don't see it perhaps you never will.Why do we bother having JPEGS and TIFFS?According to your reasoning we should get along quite well with everyone producing their own unique output formats.

Actually, Bart, I do detect an understandable scepticism about whether an open RAW format will ever come to pass. I understand that because I share that concern. However, that does not detract from the importance of tackling the issue.

The camera manufacturers get no benefit in that game, it would only drive up their cost, slow down development, and create issues with patents. Let's stay realistic, it won't happen that way.

Let's just be clear here...there is no know issue with patents as it relates to files written to media. Exactly what patents are you referring to or are you just imagining there are patent problems?

As for camera makers getting no benefit, actually, that's not really true...each and every new camera requires additional work for the camera makers to update their own software. That could be mitigated of they adopted a standard because new cameras would enjoy auto support. Yes, they still need to do the work specifying the metadata and the sensor responses–which they have to do anyway...you are overstating the amount of work that would be required...why are you doing that? Simply to be devil's advocate? Or do you know something I don't know?

There were technological advances from 12.8 Megapixels with 12-bit precision, at 3 frames per second, to 21.1 MP and 22.3MP with 14-bit precision, at a 3.9 and 6 frame rate per second readout speed which required more/multiple (parallel) readout channels which also require calibration data to be recorded. Video capture was added, which probably required more thermal calibration info to be recorded as well. It only seems logical that some changes were required in the Raw data that was recorded.

Uh, ok...so some changes are recorded as metadata in the raw file...so? What's to say that data can't be recored in a meaningful instead of undocumented, proprietary method? Do you think there is intellectual property in proprietary raw files? I don't...(and neither do the guys I know who decode them). The big secret is there are no secrets...

now, if you are talking about analog>digital converters and the onboard DSP chips, yes, that is big time, top secret tell nobody secret. But once that data is written to media, no so much.

Come on Bart, I'm expecting better of you...you should know this stuff :~)

I've been following this thread with a sense, as Yogi Berra famously put it, of “déjà vu all over again.”

Long ago, when the Internet (then called by other names) was in its infancy, there were a bunch of competing electronic mail storage formats and retrieval protocols.

Each manufacturer claimed its approach was superior to the others. Each had customers who were attached, or at least accustomed to, a particular manufacturer’s product.

You could exchange messages between one system and another through the manufacturers’ proprietary electronic mail gateways, assuming the manufacturer provided the appropriate conversion software and the current version of it worked properly, but if you changed messaging service products, it was essentially impossible to maintain access to your message archives unless you had squirreled away a copy of the old software and still had the appropriate platform to run it on.

This fragmentation of file formats and protocols seemed to work for the manufacturers for a while. It created a significant barrier to customer migration from one product to another. On the other hand, it also meant that a manufacturer which developed a superior product faced a significant barrier to increasing its market share: users of inferior products were reluctant to make the switch and orphan their message archives.

Both the manufacturers and the customers were mostly stuck in a rut until a software developer with no financial stake in the status quo (the late Mark Crispin) published a messaging service standard that incorporated essentially all the features of the various proprietary products, the Internet Message Access Protocol (IMAP), and the university which employed him (the University of Washington) made available to the public, at no cost, a software suite to implement it.

It still took quite a while for IMAP to be universally adopted. Not so much because the commercial purveyors of proprietary products felt disadvantaged by it—they remained and still are free to offer competing user agents with different looks and feels in an attempt to keep existing customers and appeal to new ones—but because ... well ... it's bureaucratically difficult for many companies to change the assumptions that underlie their products, even when the technical obstacles to doing so are modest.

What finally did the trick was market demand. Mostly from large corporate and governmental customers, which didn’t want to be locked into a particular messaging product and began migrating to ones that conformed to the superset international standard.

This is where the historical analogy begins to break down, I'm afraid, because there isn't the kind of corporate customer presence in photography that there is in enterprise IT. Although the government archivists really ought to be making more of a stink about the lack of standardization than they seem to be. However, I suspect a sufficient grass-roots groundswell (I don’t think that’s a mixed metaphor) from many individual photographers eventually will bring the camera manufacturers around.

Funny thing is, it actually would be to their benefit to coalesce their raw formats around a public standard. It must cost a lot of money to maintain the different formats and I never heard of anyone who claimed to have purchased a particular camera because he preferred NEF to CR2, or vice-versa.

Yes there are plenty of stories in the world of electronics that have a similar ring.

In our community or industry - photographic - there is a lot of misinformation and confusion about what things mean and what they don't.Currently as any reading of the threads dealing with this issue show it is clear that there is no consensus that there is a problem, much less what should be done about it.If there are any stakeholders out there who have a vested interest in maintaining the status quo they must be well satisfied.Even on a forum such as this, apparently populated by many well informed individuals, we appear unable to even agree that the sky is blue!

Perhaps, despite ourselves, useful change may occur, but I doubt that it will.We, the consumers are generating a lot of heat, but only by random Brownian motion right now, not a hint of cohesive movement.Maybe Michael (Reichman) will be able to achieve something with a word in a strategic ear but if he is expecting the backing of the forum members as a whole it seems he must be disappointed right now.