Lightroom: Prevent loss of Edit Histories when Reimporting Photos

When importing DNGs with stored edits (included XMP data) then the history of the photo just shows "Imported..." instead of the list of edits.

I have a corrupt catalogue. (I did nothing to cause the correction :()
The catalogue contains photos which are not associated to folders in the library module. When I choose "Got to folder in Library module" from the context menu for such photos, nothing happens. I imported them just like any other photos, but somehow the corresponding library folder wasn't created or lost.

I tried synchroning the parent folder but the missing subfolders are not created again.

That's why I decided the only way forward is to create a new catalogue. However, the new catalogue doesn't have any of the edit history. The rendering is OK and I can reset it to see the original version of the photos but I cannot see the edit history anymore.

Why is the edit history not recreated? The essence of it must be available because otherwise the correct final rendering could not be created.

I believe edit histories should be available for JPGs, RAW and DNG files. When I decided to use DNG files vs RAW files with sidecar (XMP) files, I didn't know that I'd lose the history with a fresh import of a DNG file. I suppose that if I had XMP files, I could copy these and still had my edit histories.

Edit history isn't in the XMP because plenty of people - me included - simply wouldn't want it there.

If catalogue corruption is the real problem, it's that problem that needs to be resolved. eg by the user using a backup (which contains edit history), or by Lightroom's backup being smaller (ie a transaction log allowing rollback/forward)

I see your point and concede that if the catalogue corruption would not have occurred, I would not have reported the lack of edit histories. However, now with a corrupt catalogue, I'm left between a rock and a hard place. Either I use the corrupt catalogue and cannot access some of the images in it properly, or lose all edit histories.

Note that some people may prefer to have all information that is needed to recreate a LR catalogue in DNG or RAW+XMP files. That would alleviate the problem of having to backup catalogues separately.

Also note that I didn't notice the corruption a long time. I went back to multiple catalogue backups and they didn't help. I guess LR never added the library folders properly (just added the photos to the catalogue as orphans) and hence there is no "good" version. The only option would be to go back to an intact but incomplete catalogue and try to build it up again. I'd rather prefer a synchronise or repair operation to succeed.

TK said: "Also note that I didn't notice the corruption a long time. I went back to multiple catalogue backups and they didn't help."

Whoa. Yeah, people act as if backups are a panacea, but that assumes your backups aren't wonky.

I've known that edit history is only in the catalog for a long time, so it wouldn't have been a surprise, but it was a surprise the first time... - I feel for ya.

So, I take it the catalog integrity check that I assume you were smart enough to perform before the backups (if not, I've heard confession is good for the soul...) was passing, despite the problem? I hope you've sent the funky catalog to Adobe for inspection(?)

Anyway, if the catalog is mostly OK, it may be possible to reconstruct the missing pieces using an SQL client. Or at least someone at Adobe maybe could, I'm not sure I know enough, or you...

Unfortunately, the corrupt catalogue passed and passes the integrity test. If it is just an SQLite integrity check then it wouldn't mind about orphan photos (photos who lost their direct folder association), would it?

I'd be happy to send the catalogue to Adobe but won't do it unsolicited. I seem to remember some bug like this being discussed in the U2U forums.
IIRC, such corruption can occur when you move folders within LR. Seems that moving them with the OS and then finding them again in LR is safer. But the latter issue could be a separate one to what I was experiencing because some of the missing folders were never moved; just imported "into a subfolder" as I always do.

One idea is to create a new catalogue, and then use File > Import as Catalog to import the corrupt catalogue. As someone who has almost deliberately corrupted catalogues (eg by running SQL) I've found that importing into a clean catalogue can purge awkward problems. A nice bloke at Adobe sometimes helps fix serious cases of catalogue corruption too. Have you posted elsewhere about the corruption?

Overall, I am not blindly against saving history steps to XMP - though XMP does have the adjustment settings so you're only losing the "route". I do think there should be an option (haven't we had this conversation before?) to save history to XMP, but it should include other Lightroom work that isn't currently saved out. So I'd like it to be my choice whether to save stacking info, assignment to collections, virtual copies etc.

On the other hand, I wouldn't want Adobe to see this as anything more than a very secondary level of backup.

"One idea is to create a new catalogue, and then use File > Import as Catalog to import the corrupt catalogue.": Thanks a lot for this suggestion. I thought about doing that but wasn't sure what end result to expect from it. I'll try it. Thanks again.

"File > Import as Catalog" isn't panacea either. I've done this before to try to fix a catalog and only realized much later that I had lost many of my collections and web galleries. That one still stings...

Good news:
I discovered that the corrupted catalogue had only one corrupt entry. It was an image whose "root folder" was "nil".

I exported all photos in the corrupted catalogue -- except for the one problematic one -- to a new catalogue and, surprise, surprise -- the new one shows all the missing subfolders again!

So LR stumbled across one corrupt database row, causing it to drop a number of library folders from the display. I also noticed that LR became very slow with the corrupted catalogue in some operations (displaying "All Photographs").

Maybe (wild speculation) corrupted catalogues can sometimes be the source of some "performance" problems?

Question (to the staff): Should I post this as a feature request? It seems it is not really a bug because the loss of the edit histories is "as designed". But it seems that there might be cases when saving editing histories in or with photo files would be desirable.

It seems that I cannot convert this bug report into a feature request anymore, so should I create a new one?

Yes, it does. Thanks Chad! It doesn't really read like a feature request, though. If it is clear what I'm after then that's OK with me. I'd be happy to rephrase the original bug report as a feature request, if this would be useful.

Personally, I'd like anything that can be stored in XMP to be stored there. I don't trust the catalogs or backups of the catalogs as I've been burned by them going bad on me several times (Dan saved me once). I'm not willing to lose the work I do between catalog backups which means I really want a reliable backup after each image edit, which is totally impractical using the catalog backup technique. As of now, I've given up entirely on backing up the catalogs and only backup the XMP data, which is image-by-image, basically instant and has saved me a couple of times when a catalog went bad inexplicably (I would have lost perhaps ten hours of work each time going back to a catalog backup). Basically, the catalogs seems fragile to me and the XMP data is not so I rely on the far more robust XMP backup strategy instead of the fragile catalog backup strategy. The bad news is that I can't use many of LR's features including collections, stacks, VCs and flags because they aren't stored in the XMP data. Personally, losing the Develop history is the least important to me as I rarely use it for anything anyway.

Lee, I perhaps wouldn't call the catalogues "fragile" but I agree with your overall sentiment. I think you should press the "like" button for the feature request. :)
Still not sure whether the former bug reports works as a feature request. Shall I create a new one?

I think I waited a bit too long to create a new thread. There are so many contributions in here that it would be suboptimal to start afresh. However, maybe my first post could be edited to feature the rewritten feature request I posted here as a comment.

John, I had quite a number of backups of the catalogue but they didn't help one bit in this instance because the error was a silent one and I didn't discover it before several generations of corrupt backups. Going back multiple generations to one that is still clean means that a lot of work has to be redone. This is not optimal.

Two thoughts on this: I've generally been of the opinion that significant metadata should (eventually) find its way into XMP since XMP can be used as a kind of distributed catalog backup. The main reason I could see not to have history in the XMP is that it'd be too big, but if it were stored smartly as a series of deltas against the final settings, it's should be fairly small in most cases.

Which reminds me of another mini-feature I've wanted for the history panel: a "minimize" option that elides multiple (even non-contiguous) adjustments of the same slider into a single step. It would turn a messy history list into a simplified summary of all the actions taken to get from original to final.

The reason that's possibly relevant here is that minimize could probably be implemented in such a way that it could compare the default settings with the current settings which would mean an image imported without history would get a concise summary of the changes. Not as complete as the original, but it provides at least some of the utility.

If one's to have an *option* to write the edit history in the XMP, Dan, I'd rather have it undiluted. History's about how one got to the final result, and restoring only the differences would mean people wouldn't be able to go back to a certain stage of work or use it for Before/After comparison. How could one get that value from the differences from default?

And you don't even need to store the differences in the XMP - LR could already compute them from the adjustment values already in the XMP. So again, where's the value?

Application of presets? You mean upon import? I'd say they form part of the difference. But that's arguable. Some will want them in, some out, and as there's not a lot of value from having simplified history....

Dan, I like your idea of an "essential history". It could be just a view on an underlying full history but maybe the user could also be given the option to "compress" the history to retain only the essential steps.

History size in XMP files or DNG metadata could be best controlled by using data compression (c.f. .zip files). I think in compressed form it would be feasible to store complete histories. Those who prioritise storage space could opt to store essential histories only.

Current XMP contents must be close to "essential histories", otherwise LR wouldn't be able to produce the final rendering from the original.

Lee, you wrote: "Remember, due to the Camera Raw defaults and the application of presets, the starting point isn't always fixed." I don't understand what you mean. When LR only got DNG or RAW+XMP files to work with (i.e., not catalogue data) it must be able to recreate the final rendering as if the catalogue data were available. Hence an entire essential set of instructions (which is close to but not the same as a essential history) must be contained in the DNG or XMP. I don't understand your comment about already having a simplified history.

See my remark lower in the thread. Zip compression works remarkably poorly on extremely large develop histories. They do better on large numbers of nearly identical develop settings, though even that is better accomplished by not storing identical settings many times. Most catalogs contain large numbers of photos with the same settings applied (either defaults or preset applied settings).

The minimize feature would definitely lose the complete path from start to finish as that isn't the intention. For me, I never (ever) care about the whole path. I care about a summary of the set of adjustments I made (that is, what path would I have taken if I'd known where I was going and made no mistakes). That's a distinct way to view history.

On the other hand for storing complete history in XMP, even if you store the deltas, you can reconstitute any step along the way (source control systems use this trick for efficient storage). It is undiluted, just very smartly compressed.

As an example of the reason for using deltas, I once was sent a corrupted catalog that was 7 GB. The size was dominated by the data for a small handful of photos. The history for the largest of them was was 450 MB (it had been heavily retouched with localized corrections). As an experiment, I blasted that data (history step by history step) into a Mercurial (source control) repository and committed each step. The repo at completion compressed to just under 700KB (and was only a couple of MB total, only about half again or maybe double the size of the XMP file containing only the final settings which were about 1.6MB).

All that is to say, while there are reasons not to store history in XMP, size is not one of them unless the implementation is naive.

"I care about a summary of the set of adjustments I made": Yes, I do too. I'd call that "essential history". It would be nice if one could compress/minimise one's histories into essential histories.

Regarding your Mercurial experiment: Certainly version control systems can store the latest version of a document efficiently as a delta against a base. However, given that you need the base as well, I reckon that most of the compression you saw (the couple of MB total vs the original 450MB) came from a compressed (à la .zip files) storage.

FWIW, I've first logged that "collapse history" function as a feature request in the 1.x days. So count me in! I don't think it needs to be anything fancy UI-wise, a menu item would do the trick. Needs to work on all selected photos.

As you know this started as a bug report. If I could edit my original post, I'd update the text to read as a feature request. Maybe a staff member could replace the text of the first post with the following?

=== Feature request ===
LR catalogues store full edit histories. However, when changes are saved to image files (DNG, RAW+XMP, JPG) only a set of instructions to recreate the final rendering is stored, all of the edit history is lost.

When importing such image files one only gets a "Imported..." entry, instead of the full list of edits.

I suggest that users have the following options regarding data stored in image files:
a) no history stored (current situation)
b) essential history stored
c) full history stored

An "essential history" only contains those steps that are necessary to create the final rendering. For instance, all "back and forth" settings to a single parameter are replaced by a single parameter change. It would actually be nice if users were given the option to reduce the in-catalogue histories to "essential histories" as well.

The motivation for having histories outside catalogues is twofold:
1. It would simplify backup strategies in that one would only need to back up image folders to retain final renderings with their edit histories.

2. It would create a safety net against catalogue corruption. If a catalogue develops a problem one may not immediately notice and thus create multiple backups with the same problem. Instead of being required to rebuild a clean catalogue from (potentially very old) clean backups, one should have the option to just create a new catalogue from the image files.

The size of histories stored outside of catalogues could be addressed by efficient storage schemes (e.g,. binary compression).

Surprisingly, no. Zip compression of the 450 MB develop history only reduced it to a third of its size or so (IIRC), which is beaten by a factor of nearly 100 by deltas. Local corrections develop history compresses really badly with the zip algorithm. A RAR based compressor did much better (making the whole 7 GB catalog only 130 MB), but was MUCH slower to perform.

The 10:1 compression ratio comes (I think) from the metadata table's XMP content, which has its own efficiency issues.

Thanks a lot for your comments, Dan. It is very good to see that you have looked into compression options already at some point.

It seems that there is little contention about compressing (older) backups. I'll do that manually for now, but it would be nice if some automation would be available in the future. The latter could include moving backups from a (quick working) drive to another (archival) drive once backups have reached a certain age.

If the purpose is just for backup/transfer/security, perhaps I could throw another alternative into the net - how would it work for people if there was a LR-specific sidecar option, which could contain absolutely everything including History, flags, etc. Something like XMP sidecars, but without interfering with the XMP spec and without having to worry about it breaking other programs that use XMP.

TBH storing extra fields would not "interfere with the XMP spec" or trouble other programs. It's the X in XMP - extensible by you, me or Adobe. Adding extra fields is what it's all about, and other programs simply pass over fields which they aren't set up to read. OTOH a new format or sidecar would make it more complicated to update other programs as they'd have to have logic for a new file type and data format, rather than just being told to read a new XMP field.

I've said I don't see much value in Dan's summarised history, but I did say "I do think there should be an option... to save history to XMP, but it [the option] should include other Lightroom work that isn't currently saved out. So I'd like it to be my choice whether to save stacking info, assignment to collections, virtual copies etc.". I'd probably switch off the history option though, and think off should be the default.

If we could ensure the storage design was efficient enough to not be an undue burden (bloated size) and XMP was extended to be more complete with respect to the catalog, I would say the default behavior should be to preserve all data and let people optionally choose to exclude the data they don't mind losing rather than making presumptions about what is important and what is not.

The default only matters if you don't know you have options, or where to go to change. Consider a warning dialog - first time save: warn about the fact that not everything is in, or it includes the kitchen sink..., then the user knows what to expect, can decide to do it differently, knows where to find the setting to change mind in future...

Ian, considering most people assume that all of the data IS stored in XMP and are shocked to find their history etc. missing, I would have said that the default should be ON in this situation. More knowledgeable users who know they don't want it could then turn it off.

This is a fundamental change in behaviour - folk have been using Lr since 2006 and haven't had history in their files. Adobe make it the default and everything from then on has history. User unaware of the change sends image to third party who now knows exactly what was done. If you think that won't cause a riot then you've a lot to learn.

Ian, "h.ll to pay" (earlier comment) "you've got a lot to learn"? You're making a valid point here, but there's no need to take that condescending tone when making it.

If you're sending XMP + raw content to a third party you're implying a substantial level of trust. Sending full history is admittedly tipping your hand a bit further, but if you really don't trust the third party not to be reverse engineering your technique, you should be sending baked in rendered formats with most metadata stripped anyway.

What's clear from the thread is that careful thought will have to go into the defaults and options, but there are multiple equally valid choices here depending on the needs.

Ian, "h.ll to pay" (earlier comment) "you've got a lot to learn"? You're making a valid point here, but there's no need to take that condescending tone when making it.

If you're sending XMP + raw content to a third party you're implying a substantial level of trust. Sending full history is admittedly tipping your hand a bit further, but if you really don't trust the third party not to be reverse engineering your technique, you should be sending baked in rendered formats with most metadata stripped anyway.

What's clear from the thread is that careful thought will have to go into the defaults and options, but there are multiple equally valid choices here depending on the customer.

Perhaps a better scheme than a no-vote/pro-vote would be a priority ranking. I'm often in a quandry - do I vote for something that I think is a good idea, but isn't one of my top priorities? - Some have said "absolutely not" - since it dilutes the priorities. Others have said "yes, of course" - a good idea is a good idea...

If one was able to vote:
0 - this would be a step backwards... - I wouldn't want this, period.
1 - If everything else were already done, then this good too...
2 - good idea but not high priority
3 - good idea but not top priority
4 - top priority.

Perhaps voters that choose "-1" (and perhaps "2") should be required to leave a comment to justify their choice. Staff could then include or exclude such votes from the (internal) tally depending on the merit of the comment.

I'd be against allowing "-1" without a required comment because there is a chance that people will unduly dismiss ideas just because they want to help other ideas by voting down ideas that don't seem to have any value for them.