Recommended Posts

Scenario:
- Existing drive, good backups
- For !@#$% reasons, a chunk of an NTFS partition was literally zero'd out at a low level
- Result: files are all in place but I guarantee their contents have changed. Consider it bit rot on a massive scale

What I need:
- Restore essentially all backed-up files, ignoring the fact that the metadata matches perfectly
- In other words, I want to do a restore-with-overwrite.
- Important note: I do NOT want to simply restore the volume if at all possible, because there are many new/changed files not in the backup.

I can't find a way to do this. What am I doing wrong?
Mostly, it auto-deselects all the files that appear to already be there (matching in size and date stamps).

Any ideas much appreciated.

I'm about ready to do the obvious/pain-in-the-neck workaround...
- Find all new files and copy elsewhere and/or
- Delete all files that aren't new

THANKS!

Pete

Share this post

Link to post

Share on other sites

Yes you can do this. Setup your restore job under Manage Scripts. On the Destination Selection window, there's a list box at the top of the window "Retrieve Files & Folders". Click the down arrow and choose the option you want. If you can, you should restore all files to a new location.

My advice, if this is at all possible, is to replace the drive and then restore. What you're doing is super risky as I'm sure you're aware. Make a Disaster Recovery disk to do this, and run it on the problem system.

My further advice is to go to http://grc.com and purchase Spinrite and run it on your system first. If your drive isn't healthy and is repairable, Spinrite will do it. Your system must be able to run 'chkdsk /f' first before you use Spinrite. (I can't type the Cee-colon in that command, the stupid forum software keeps inserting a smiley face for some reason.) You should also have SMART turned on so you're warned if the drive is close to failure.

Share this post

Link to post

Share on other sites

Sigh. Mark, thanks for confirming what I sensed. This really is a bug in Retrospect. I'm about to report it.
Let's say you choose "Replace Corresponding Files" from the dropdown.

My expectation: it will replace the corresponding files. The documentation is clear about this. Unfortunately, it doesn't work. Retrospect ASSUMES that the data content matches, if the metadata matches. Boo.
I guarantee that the "already copied" files aren't the only ones... and in fact, if I prepare a *backup* of the bad data, it's going to copy way more than a few hundred files.

Bit Rot and Drive Replacement: In reality, bit rot can happen on ANY drive. In no way does bit rot imply a drive that needs to be replaced!

Just for example: Consumer SSD drives retain data "perfectly" for approximately one year (if you've not overwritten a sector) before bit rot begins to show. It's worse for enterprise SSD's (because they are focused on performance over data retention.) This is not exactly advertised by the industry. Dig into the latest research and you'll discover the reality..) NOTE: some mfg's MAY be compensating for this and taking care of it in firmware... but I have yet to see such workarounds documented. And YES I have been harmed by this. Consider: what is the one sector on a drive or partition that NEVER is overwritten?...

(How to avoid this: regularly rewrite every sector of an SSD. I do it about 3-4 times a year.

I could point you at the (very!) technical research work that underlies what I wrote above. I agree that it's more than a little surprising.

Share this post

Link to post

Share on other sites

....Bit Rot and Drive Replacement: In reality, bit rot can happen on ANY drive. In no way does bit rot imply a drive that needs to be replaced!

Just for example: Consumer SSD drives retain data "perfectly" for approximately one year (if you've not overwritten a sector) before bit rot begins to show. It's worse for enterprise SSD's (because they are focused on performance over data retention.) This is not exactly advertised by the industry. Dig into the latest research and you'll discover the reality..) NOTE: some mfg's MAY be compensating for this and taking care of it in firmware... but I have yet to see such workarounds documented. And YES I have been harmed by this. Consider: what is the one sector on a drive or partition that NEVER is overwritten?...

(How to avoid this: regularly rewrite every sector of an SSD. I do it about 3-4 times a year.

I could point you at the (very!) technical research work that underlies what I wrote above. I agree that it's more than a little surprising.

Thanks for the info, MrPete,

For hysterical reasons I have been alternating between the use of two drives for Retrospect versions on my "cheesegrater" Mac Pro "backup server", one HDD and one SSD. ATM I have Retrospect Mac 16 on my SSD, with Retrospect Mac 15 on my HDD. However I've elected to keep my Catalog Files on the HDD, just in case I have to switch back or upgrade forward. Retrospect Inc. (should I still call them that?) is kind enough to provide new point-releases of Retrospect Mac 16 about every 3 months, which I'm sure they do just so I can refresh my SSD.

I was vaguely aware that SSDs had a shorter lifetime than HDDs, but I thought progress had been made on that problem. Now you're saying it hasn't, so what was good enough in the 1970s is still better over the long term than its replacement. Spinning rust forever!

Share this post

Link to post

Share on other sites

Remember, it's not that the SSD fails. It's simply not an archival storage medium.

Not so sure we can say HDD's are *that* much better today. My work on early drives (yes I've been doing HDD's since the 1970's around that long) ) was simpler, because the stored bits were monstrously big compared to today. (Anyone here remember the technique of literally using a pencil eraser tip to shove an HDD into spinning? They were truly not sealed!!!

Meanwhile, today we no longer store bits in the HDD magnetic wiggles. We don't even store encoded bits.

Today, it's more like a Douglas Adams sci fi joke! Adams talked about the "improbability drive"... well, how about a "probability curve"?! To compress the data, we calculate an exponential function that represents the bits in a sector, and store the parameters of the function. If one bit is wrong, you just lost an entire sector. That's "PRML" (Partial Response, Maximum Likelihood") ... such fun 😄

Share this post

Link to post

Share on other sites

My expectation: it will replace the corresponding files. The documentation is clear about this. Unfortunately, it doesn't work. Retrospect ASSUMES that the data content matches, if the metadata matches. Boo.

The documentation could be clearer -- "Retrospect leaves files untouched if their metadata is identical...".

That's good enough for the vast majority of use-cases. What you want would require checksum comparisons and a whole lot of extra processing. Could they offer that as an option? Maybe.

But "Replace corresponding", even if it worked as hoped, wouldn't work for you anyway since you said there are changed files which you *don't* want replacing -- in fact, you want "files whose metadata matches but checksums don't". In this particular case I'd guess your best bet is to full-restore to another disk then walk the original's tree, copying across files that match your criteria -- a nice little scripting project 🙂

Since you appear to have a time-point, you might also be able to build a copy with a backup restore followed by a time-filtered Duplicate from original disk to the copy. You could then replace the original with the copy. No easier than your proposed workrounds, but you could let Retrospect get on with it while you enjoyed your weekend...

Share this post

Link to post

Share on other sites

Make a new backup, to include everything created/modified since the last of the old backups

Restore Volume using the old backups

Replace Corresponding using the new backup

Assuming the matrix is correct, that should mean that "corrupted" files are returned to their uncorrupted form then new/changed files are overlayed, replacing previous versions where appropriate (assuming you weren't doing something silly like editing file data without updating the metadata to suit). All you should lose is data created/modified *after* the last old backup but *before* the borkage *if* it was affected by said event -- that would require data recovery of some sort.

Share this post

Link to post

Share on other sites

Remember, it's not that the SSD fails. It's simply not an archival storage medium.

Not so sure we can say HDD's are *that* much better today. My work on early drives (yes I've been doing HDD's since the 1970's around that long) ) was simpler, because the stored bits were monstrously big compared to today. (Anyone here remember the technique of literally using a pencil eraser tip to shove an HDD into spinning? They were truly not sealed.

IBM 2311 or was it 2314? I go back to the days of 026 card punches myself.

HDDs have improved since the days I worked for Shugart Associates.

Share this post

Link to post

Share on other sites

Scenario:
- Existing drive, good backups
- For !@#$% reasons, a chunk of an NTFS partition was literally zero'd out at a low level
- Result: files are all in place but I guarantee their contents have changed. Consider it bit rot on a massive scale

What I need:
- Restore essentially all backed-up files, ignoring the fact that the metadata matches perfectly
- In other words, I want to do a restore-with-overwrite.
- Important note: I do NOT want to simply restore the volume if at all possible, because there are many new/changed files not in the backup.

I can't find a way to do this. What am I doing wrong?
Mostly, it auto-deselects all the files that appear to already be there (matching in size and date stamps).

Any ideas much appreciated.

I'm about ready to do the obvious/pain-in-the-neck workaround...
- Find all new files and copy elsewhere and/or
- Delete all files that aren't new

THANKS!

Pete

Pete,

Frustrating situation I agree. Can you detect which files are new/changed on the source drive?

If so, may I suggest that you get a new drive that will serve as the source. Copy over all files from the current source in a way that preserves file dates, to a new location of course. Then use this utility http://scootersoftware.com/ to compare your current files with the restored ones. Beyond Compare will automatically highlight files that are newer/changed, so you could ignore those. I use this utility on a daily basis.

My first job as a programmer (initially a trainee after two weeks of reading)—starting in June 1964—was with C-E-I-R Inc., the pioneer of non-manufacturer service bureaus. We had professional operators in the air-conditioned computer room, who would have broken my fingers if I dared to touch hardware—with a pencil eraser tip or even pushing a switch.

IBM 026 card keypunches were used for input to 2nd-generation computers such as the IBM 1401, beloved because it was so easy to program in assembly language (as in this example whose statement labels and field names are in programmer-chosen abbreviated German but the operation codes are IBM-standardized abbreviated English). IBM 029 keypunches were a later model used for input to IBM System/360 computers, to which 2311s and 2314s were attached.