I'm starting this to ask:If we, as a photographic community,believe it is our best interests to support the adoption of a non-proprietary raw format, by the camera manufactures or not. If so why and if not why.

While reading a recent thread it was pointed out how large the LULA community is and since the camera companies just might (hopefully) be listening let's tell them what we think.Personally I think that a standard format is necessary for archival reasons, for the long term preservation of our images, as well as being able to re-develop the image with newer processes, which, for me, is one of the main advantages that digital photography offers. Since the camera companies create the formats for their products, the ball seems to be clearly in their court.

I'm starting this to ask:If we, as a photographic community,believe it is our best interests to support the adoption of a non-proprietary raw format..

Gerry, the answer is a resounding yes!

Currently DNG would fit the bill.Far from being a proprietary format it has been developed, yes by Adobe, with the express purpose and hope of being adopted for exactly that role.Adobe has ensured that the DNG format is open and has offered the format lock, stock, and barrel to the ISO so that the format could become an industry standard.Some people are seeing a conspiracy theory in this but I invite anyone to research the development of the TIFF format (who did that) and its eventual adoption as a standard format by the ISO.

However, does this mean that DNG would have to be the format eventually adopted? Simply, no.

Many mainline camera manufacturers are currently ignoring the issue however there are no technical reasons mitigating against a DNG being generated in-camera.Even if they offered the option of DNG (or another mutually agreed on open RAW format) alongside their own proprietary format this would be a great step forward.In the last few days I have read some wild arguments about firmware bloat as a consequence but again some knowledge about the DNG format and its historical timeline demonstrate an intriguing fact: before the release of DNG as an open format most camera manufacturers had appalling RAW formats that were a real mess. Subsequent to the release of DNG the next generation of proprietary RAW formats across a lot of brands suddenly improved out of sight. Coincidence?

A commercial argument has also been put forward to defend the camera-makers current stance. However, Jeff Schewe has been making the point, ad nauseum, and he is right, that the real 'secret sauce' for the camera-maker lies in the sensor/processor, and what that combination does, not in the subsequent recording of 0's and 1's in the RAW file.Even if the makers want to embed secret information in the RAW file for their own RAW processing software the DNG standard already allows for that.As already pointed out it is likely that DNG genetics are to be found in many proprietary RAW formats right now. So, for those who know how this works there would not be a financial cost for the technical adoption of DNG, or perhaps another RAW format.

Rather, the prime stumbling block appears to be a stubborn selfish pride by most camera-makers to even acknowledge that there may be a problem.Actually, my last statement is pure speculation since the SILENCE from these camera-makers on this issue has been ABSOLUTELY DEAFENING.So, we have no real way of knowing what their motivations are for ignoring the issue just that it would seem to have nothing to with a technical barrier nor, in my opinion, is it a financial/marketing issue.

What is going to be needed is sustained, reasoned (and reasonable) pressure from us - their customers - to address the issue.Once the bulk of camera-makers acknowledge the desire for a standardized open RAW format, my guess is that the ISO, for them anyway, will move fairly quickly on this. Everyone knows what needs to be addressed for ISO standardization to be achieved.

Tony is absolutely right: "The answer is a resounding yes!"In that other thread, Jeff Schewe and Andrew Rodney explained the issues (over and over again) very clearly.

As a Canon owner all my raws come as either CR or CR2 files, and since LightRoom and PhotoShop handle those so well, I've not bothered to save them as DNGs. But I am about to start saving all of my raws as DNGs on a separate hard drive. Once we have a standard, non-proprietary raw file format, I'll switch to that. But I fear it may not happen in my lifetime.

Many mainline camera manufacturers are currently ignoring the issue however there are no technical reasons mitigating against a DNG being generated in-camera.

Actually, with DNG v1.4 it could be said that DNG has actually been advanced to the point where DNG is now a different story. DNG 1.4 allows two additional flavors of DNG, Lossy DNG–which takes the linear raw image data, maintains the linear scene referred image data and allows JPEG compression which means that Lossy DNG maintains a scene referred image (as apposed to a gamma encoded and thus baked color space) which allows far more flexibility in editing that JPEG offers. The other option is downsampled Lossy DNG.

The potential file size savings could mean that camera makers could offer both standard DNG and Lossy DNG instead of native raw and JPEG.

While Lossy DNG is not truly raw–it requires demosiacing while maintaining a linear gamma, the same as the previous Linear DNG offered, the ability to add JPEG (currently about JPEG level 12 I think, but that could change) mean you can keep an image in scene referred color while maintaining raw type editing.

So, Lossy DNG is not quite as small as JPEG, but retains the flexibility to retain raw style editing from a linear file vs. trying to edit a baked color space/fixed gamma JPEG file.

If you wanted to save even more and only need a lowered amount of resolution, you could use both lossy DNG plus downsampling–with far better potential than Canon's SRAW. So, if you took a 24MP raw capture and wanted a lossy DNG that was 10MP in size, the resulting file size would be down to 4.3MP.

So, what you THINK you might know about DNG, has recently changed. There's an except from my Digital Negative book detailing some of the recent changes in DNG in a DPP article: DNG File Format & DNG Converter.

BTW, this article also contains info about the National Digital Information Infrastructure and Preservation Program (NDIIPP; www.digitalpreservation.gov) about some of the aspects and risks inherent in the long term conservation and preservation of all types of original digital files (it's not just a digital photography issue). Another organization that is working on these issues is the National Archives in England.

To my way of thinking, this is all a no brainer...photographers who understand the issues (and it would behoove photographers to know this stuff) should lead the industry to the position that the current proliferation and spread of undocumented, proprietary raw file formats is putting our digital original raw images at risk for long term access for posterity.

There are different views about what to do about this situation...no, I don't suggest boycotts of camera companies who don't document their raw files. Photographers are too independent (and hooked on new cameras) for that to work. But, I do think that if the photographic industry at large all shared the view that undocumented, proprietary raw files are less than optimal and that the camera companies should be pressured by public opinion to reassess their positions and work toward improving the current poor situation and also come forward and offer to work with various agencies that are working towards long term conservation and preservation practices and standards would be a good result.

And no, I'm also not suggesting that photographers rush to convert all their proprietary raw files to DNG and discard their original raws...I've never said that nor has Adobe. I use DNG for specific reason in certain cases...

The main thing I see as being the best short term result is that photographers of all the various types of photography all agree that the current situation must change and to quit arguing on the behalf of the camera companies' continued poor policies regarding undocumented, proprietary raw files.

At this point, digital photography has been like the wild west with unprecedented advances and changes in the industry. As we've seen recently, the revolution is changing to an evolution. Cameras and sensors are making mild advances not revolutionary changes of late. This is the perfect time to start the serious discussion about the issues because it's now that various aspects of a standardized raw file format might actually become a competitive advantage for some camera companies. The current best example is Leica whose recent cameras all offer DNG raw file output which enjoys de-facto support in any application that full support DNG...Leica no longer needs to create and maintain raw file formats, all they need to do is follow the DNG specs (which is really not that hard–just look at the DNG SDK if you are geeky.

So...I've said my piece...as you might expect, Michael has similar thoughts. So, if you have technical questions, post them here. I'm pretty sure we have an excellent LuLa member here that can speak directly to any technical DNG issues, Eric Chan (MadManChan). Just know that it would be more useful to have a civil discussion without name calling and other bullshyte taking place. On the previous thread about this, I tried very hard at restraint and encourage the Mods to be flexible and not close the thread...

I'll admit up front that I don't know a lot about this subject but it seems to me the camera makers would be more likely to include a DNG option once it's accepted by the ISO. Until then it's someone else's format so the camera makers feel they might as well stick with their own.

I fully support having dng as an option in camera. I convert all of my Canon raw files to dng upon import. I do so mainly because of the smaller file size. I can see absolutely no difference in the file content on my screen when pixel peeping and certainly not in a resulting print. I think that dng is at least just as future proof concerning the ability for people in the future to be able to read the file format when compared to ANY raw format. The fact that the Diginerds/geniuses at Adobe can take the manufacturer's proprietary raw files that contain "secret" information and reverse engineer it to create an application that does a better job in both output as well as user interface than the people that create the source file tells me all I need to know.

I have really enjoyed reading Jeff's and Andrew's comments in other threads. I just wish the two of them would open up and really tell us what they think on the subject.

I have often thought about sending my Canon software cd that contains DPP back to Canon with a note attached stating that there software does not support an open documented raw file format, therefore I can not use it. Please give it to someone that can use it. I do not use it even for CR2 and CRW files that my camera generates, by the way. Adobe's PV2012 and ACR/Lightroom produces the best overall image quality with my files in my opinion. YMMV.

In the last few days I have read some wild arguments about firmware bloat as a consequence but again some knowledge about the DNG format and its historical timeline demonstrate an intriguing fact: before the release of DNG as an open format most camera manufacturers had appalling RAW formats that were a real mess. Subsequent to the release of DNG the next generation of proprietary RAW formats across a lot of brands suddenly improved out of sight. Coincidence?

Hi Tony,

I suppose you have some objective evidence for that theory?

Quote

A commercial argument has also been put forward to defend the camera-makers current stance. However, Jeff Schewe has been making the point, ad nauseum, and he is right, that the real 'secret sauce' for the camera-maker lies in the sensor/processor, and what that combination does, not in the subsequent recording of 0's and 1's in the RAW file.

Is that so?

Raw files do more than record the exposed and converted data as processed by the camera. Outside the exposed image area there is a number of proprietary masked and other sensel data (to do with black level, vignetting, banding, amongst others). Is all that copied into the DNG file?

Quote

Even if the makers want to embed secret information in the RAW file for their own RAW processing software the DNG standard already allows for that.

If so (and it certainly wasn't in early DNG versions, so who guarantees it is now?) what does a(n) (error free?) conversion add for high quality future conversions? DNG is just a conversion to another format. My Raw files can be read just fine, bit by bit, and can be converted into an image by the manufacturer's software (with generally reported better color than from other converters), and by a number of independent software solutions, and even with something as universal as DCRaw, although they won't be able to utilize the proprietary data in the Raw file (that is, assuming it is still there).

Quote

As already pointed out it is likely that DNG genetics are to be found in many proprietary RAW formats right now.

Is that so? Do you have any credible and independent source for that theory?

Quote

Rather, the prime stumbling block appears to be a stubborn selfish pride by most camera-makers to even acknowledge that there may be a problem.

It's just another theory. Do you really think that pride is the reason, or could it be that they see no benefit in putting Adobe in a spotlight. What commercial benefit would having to spend resources in keeping up with a changing file-format bring them? Besides, it would be more of a benefit for Adobe to only having to deal with a single Raw file format than the manufacturers of cameras. Is Adobe willing to share some of those savings with the camera manfacturers?

The camera manufacturers now have the liberty to produce hardware, which produces a Raw file, with exactly the data in it that the camera generates, and can utilize for a potentially high quality output. Then they move to the next technological step and that will produce a somewhat different Raw file, without the need to think about incorporating deprecated legacy parameters or file structures. They can also reserve space for features that didn't make it in time for the initial release, but which will be needed for a later firmware upgrade (and they won't have to tip off Adobe about it in advance).

I think people have a too romantic view of commercial manufacturing companies. They are not in this world to secure the future of our Kodak moments, they are here to make money. I'm not saying that's a good thing, but it is what it is. They will only adopt something universal, if it brings a lot of clear (competitive) benefits, and no downsides.

I'll admit up front that I don't know a lot about this subject but it seems to me the camera makers would be more likely to include a DNG option once it's accepted by the ISO. Until then it's someone else's format so the camera makers feel they might as well stick with their own.

Unfortunately the problem is likely to be a bit circular with not a lot of action likely to be taken on standardization of DNG, or some other format, in this context unless the big camera boys make the right noises.So to suggest that camera makes should wait for the standardization of a format before adopting it won't work.

A lot of effort is going to be needed to bring a very unwilling bride and groom together for their nuptials.However since what is wanted is a good and constructive thing a shot gun would not be appropriate for the wedding.It is only going to work if the camera makers see, appreciate, and accept the benefits of what is being proposed.

With regard to DNG itself is a rapidly maturing format with considerable flexibility and utility and could be an excellent choice for a standardized RAW format.In reference to Jeff's comment on how DNG is changing - so is every other useful format.JPEG and TIFF are regularly being updated - it is just that the ISO controls the process. One of the key issues here is ensuring continued compatibility between newer versions and older versions.So, in contradistinction to several posts decrying DNG because it is "changing", we want useful formats to continue to be updated and upgraded because doing this maks them even more useful with time.

The bigger Digital Asset Management problem is growing every second with formats being orphaned all the time including sound, video, graphics, and of course RAW photographic formats. Trillions of documents and files of various nature are already, to all practical purposes, orphaned and not accessible. Perhaps, if you have the budget and time of a large government agency and acces to both the software and hardware that made that particular format and you are able to get it working it is ,perhaps, possible to port information from one format to another.For most of us though software that ran on Windows95, remember that?, so recently and is not currently supported on newer hardware and operating systems is gone for us.I don't myself have access to a system that could run Windows95.Offhand, I don't know anyone who does.

With regard to cameras dozens of first-generation RAW formats are already orphaned not least because their manufacturers are out of business and no-one else is prepared to support their formats now.On another note, the third-party RAW converters, generally try to maintain backward compatibility of as many formats as possible but currently I doubt many will add dead formats now because very few individuals will be using those cameras.The problem is that millions (?trillions) of images may be residing on hard drives completely inaccessible because no software can currently open them.

The point of the current exercise is to prevent this problem proliferating to truly catastrophic levels.Having a open format RAW would be the first step in safeguarding our precious images (nothing necessarily to do with financial worth - but for many it would be) but bullet-proof back-ups and archives are also required if we are not lose our images.Basically we are in the unenviable position of negatives of my grandmother shot in 1910 surviving for another hundred years and my super-dooper digital RAW images dying a death on an expired hard drive or having nothing to read them with in a few years.Even companies like Canon, Adobe, Nikon, and Phase One are not immune from the extraordinarily brutal Darwinian process governing business in the digital age. Remember Kodak and IBM - they WERE truly iconic multinational companies that at one time almost defined business.Kodak is dead and buried, and IBM: so invisible currently that I am not sure whether they exist anymore!

So remember that nothing is forever and if we, both as individuals, and as a group, do not protect our interests no one else will.

Raw files do more than record the exposed and converted data as processed by the camera. Outside the exposed image area there is a number of proprietary masked and other sensel data (to do with black level, vignetting, banding, amongst others). Is all that copied into the DNG file?

If so (and it certainly wasn't in early DNG versions, so who guarantees it is now?) what does a(n) (error free?) conversion add for high quality future conversions? DNG is just a conversion to another format. My Raw files can be read just fine, bit by bit, and can be converted into an image by the manufacturer's software (with generally reported better color than from other converters), and by a number of independent software solutions, and even with something as universal as DCRaw, although they won't be able to utilize the proprietary data in the Raw file (that is, assuming it is still there).

Is that so? Do you have any credible and independent source for that theory?

It's just another theory. Do you really think that pride is the reason, or could it be that they see no benefit in putting Adobe in a spotlight. What commercial benefit would having to spend resources in keeping up with a changing file-format bring them? Besides, it would be more of a benefit for Adobe to only having to deal with a single Raw file format than the manufacturers of cameras. Is Adobe willing to share some of those savings with the camera manfacturers?

The camera manufacturers now have the liberty to produce hardware, which produces a Raw file, with exactly the data in it that the camera generates, and can utilize for a potentially high quality output. Then they move to the next technological step and that will produce a somewhat different Raw file, without the need to think about incorporating deprecated legacy parameters or file structures. They can also reserve space for features that didn't make it in time for the initial release, but which will be needed for a later firmware upgrade (and they won't have to tip off Adobe about it in advance).

I think people have a too romantic view of commercial manufacturing companies. They are not in this world to secure the future of our Kodak moments, they are here to make money. I'm not saying that's a good thing, but it is what it is. They will only adopt something universal, if it brings a lot of clear (competitive) benefits, and no downsides.

Cheers,Bart

Bart, if you are asking whether I have evidence on a level that a Grand Jury might find useful, then no!However if you are prepared to do some digging around you will find references to the interesting coincidence regarding the state of many proprietary RAW formats pre and post the release of DNG as an open format.I don't have a specific list of references to give you: my statement (theory) was a summary composite of sources read over the years.Do your own research and draw your own conclusions.

As for the structure of DNG I seem to recall even Wikipedia mentions the options for companies to place proprietary information into DNG files. Perhaps you could look at more technical information such as the specification - I have it downloaded somewhere but it is not the most user-friendly document in existance today.BTW I am well aware of metadata additions to the file that govern the issues that you mention. DNG is able to interface with that information as well as any other RAW format. Certain manufacturers are using DNG as their RAW format in current models that are in your local camera store. Also bear in mind that none of this is lost if one converts proprietary RAW to DNG in Lightroom or with Adobe's standalone DNG converter.Also, if that information is really hidden unable to be accessed by DNG then no third-party RAW converter will have access to that information either.

Considering that even blind Freddy can see that there is a growing, and alarming, problem with the proliferation of RAW formats do you not think it perplexing the absolute silence that, particularly but not exclusively, Nikon and Canon, maintain on this issue. Your quote of my text was somewhat one-sided since I go on to stress that none of us know what they are thinking - PRECISELY BECAUSE THEY WON'T TALK ABOUT IT: NOT ONE WORD, NIKS, NADA, NOTHING!

There is nothing romantic in anyone's view who believes that a universal standardized RAW format is the way forward.That is not the end of the problem should it be solved since we also have to have safe, archival-level storage of our images that is cost-effective.If you are the CIA you probably have a solution - for the rest of use it is still costly, but improving.

Unfortunately, this discussion is much like any discussion with someone from the far right side of religion, abortion, gun rights, etc.

I respect their position. However, they do not respect mine.

I, at least for now, have decided that any response or attempt at logical discussion with them only gives them a soapbox to preach from. I am hoping that lack of response will allow any thread they open to have a quick dead.

Frankly, it really does not matter. Any thought that these discussions by a few will influence the actions of Canon or Nikon are, in my opinion, delusional.

Raw files do more than record the exposed and converted data as processed by the camera. Outside the exposed image area there is a number of proprietary masked and other sensel data (to do with black level, vignetting, banding, amongst others). Is all that copied into the DNG file?

Yep...outside of some early raw file formats (Canon's CRW is one I know of) and some oddball raw file formats like Sigma cameras, all the raw image data and all metadata–even if DNG doesn't understand or use them–is stored in the DNG. Both Thomas and Eric have stated this and there are provisions for this metadata migration in the DNG SDK.

I also know (cause I've seen it) that certain individuals can actually take a DNG without an embedded raw and reconstitute a valid proprietary raw file format from it. Currently, there is no way of doing that which has been released...so it's sort of a home brew operation :~)

DNG is simply a well formed and documented raw file format that is consistent with TIFF-EP which is the basis of most the current crop of proprietary raw file formats out there (with a couple of exceptions such as Sigma).

Yep...outside of some early raw file formats (Canon's CRW is one I know of) and some oddball raw file formats like Sigma cameras, all the raw image data and all metadata–even if DNG doesn't understand or use them–is stored in the DNG. Both Thomas and Eric have stated this and there are provisions for this metadata migration in the DNG SDK.

I also know (cause I've seen it) that certain individuals can actually take a DNG without an embedded raw and reconstitute a valid proprietary raw file format from it. Currently, there is no way of doing that which has been released...so it's sort of a home brew operation :~)

DNG is simply a well formed and documented raw file format that is consistent with TIFF-EP which is the basis of most the current crop of proprietary raw file formats out there (with a couple of exceptions such as Sigma).

Yeah, but as an open format, it has a weired downside as well. In the motion version, the fact that it is open actually blocks the developpement of support into proprietary NLEs. That also happens with open EXR too. Nobody is willing to develop the support and put engineers at work as it will also be usable by the competition. So we got this surreal situation that DNG cinema can be supported by a camera that features it,in the case of Black Magic, in their software Resolve; but it's not a standart yet. Only if and when they'll have no other choice that commercialy support it (because the demand will be such) they will do it.For example, Avid has an environement where proprietary formats can write their codes for native support, say Red for example. It's a convention between the brands, But who they will ask to for DNG support? (exactly like what the US president asked to the europeans: "if I want to call Europe, who am I talking to?").

To be honest, this strikes me as a solution in search of a problem. If you are worried about future lack of support for proprietary RAW files in the future, then convert to DNG on import. Other than possibly making your imports a bit slower, this would seem to solve all the potential problems that people are fretting about. No?

Logged

Peter"Photographic technique is a means to an end, never the end itself."View my web pageView my Facebook page

If you are worried about future lack of support for proprietary RAW files in the future, then convert to DNG on import.

Yes, but that puts the burden on the user and again gives the camera companies a pass to continue the proliferation of undocumented, proprietary raw files. So, this doesn't address the underlying problem, just shifts the burden to millions of users instead of a relatively few camera makers.

Perhaps LightRoom's default could be to convert imports to DNG? No burden on the user, no need to whine to the camera manufacturers, and we can get back to making beautiful photos.

Again, this is less about DNG (it's a proposed standardized raw file format submitted to the ISO for inclusion in the next TIFF-EP spec) and more about dealing with the root problem of the proliferation of undocumented, proprietary raw file formats. If you don't want to deal with the issue personally, that's fine. However, there are people who see the issue as a problem so it would be useful if those who don't care at least do nothing to interfere with those who do care.

Yes, but that puts the burden on the user and again gives the camera companies a pass to continue the proliferation of undocumented, proprietary raw files. So, this doesn't address the underlying problem, just shifts the burden to millions of users instead of a relatively few camera makers.

Exactly. Those are the key words describing the situation, perfectly resumed in this sentence.

Yeah, but as an open format, it has a weired downside as well. In the motion version, the fact that it is open actually blocks the developpement of support into proprietary NLEs. That also happens with open EXR too. Nobody is willing to develop the support and put engineers at work as it will also be usable by the competition.

Well, I'm not at all familiar with the issues of digital video file formats...I'm aware of CinemaDNG but I really have knowledge of the issues and the various venders involved. So, I don't know if CinemaDNG has any technical benefits or weakness. I gather it's gotten some support but I don't know where it stands relative to the DNG raw file format.

Well, I'm not at all familiar with the issues of digital video file formats...I'm aware of CinemaDNG but I really have knowledge of the issues and the various venders involved. So, I don't know if CinemaDNG has any technical benefits or weakness. I gather it's gotten some support but I don't know where it stands relative to the DNG raw file format.

Apparently, Adobe removed it to Premiere support because officialy they were not happy with the delivered performances within the app. It's true that it is not "suitable" as an editing format, but then, so is "AVCHD". The prob is that nobody wants to write for free. The main issue would be on the debayer aspect. But I'm afraid that now that Raw video is becomming more democratized, we will see the same mess very soon as still and an endless list of proprietary Raw formats wich support is going to put everybody crazy. Will the motion industry fully embrassed cinema DNG as a standart? I think there is a pressure among users but...It happened to be similar with open EXR when Industry Light and Magic intended to built a standart. Here all FX artists are using it but as it's not supported everywhere we have roundtripping obligations and in the end the users are on bondage.