A few users have reported either files missing when they view Live Mesh folders on their computer (but appearing as expected on their Live Desktop) or their computer showing all files but reporting many more conflicts than expected (and their Live Desktop showing duplicate copies of any file with a conflict.) We’ve spent the weekend investigating this, and we have a few details below. But the key news is this:

Live Mesh is not deleting or losing files. Even if they don’t all appear in Live Mesh folders on your computer, they are all present in a hidden state, as well as on your Live Desktop if the folder is synchronized there.

Uninstalling the Live Mesh software from a PC will delete the files that are in a hidden state on that PC. If the folder is not synchronized on any other computer or your Live Desktop, then uninstalling will result in the hidden files no longer being available.

If you stopped synchronizing a folder on your PC, then re-synchronized it, that folder is likely to generate some number of false conflicts.

We're testing a fix for this now and will deploy it as soon as we're confident it’s ready. In the meantime, if you have a Live Mesh folder that appears to be missing files (that is, synchronization appears to be complete but expected files are still not present) or one that has an unexpectedly large number of conflicts, in order to prevent data loss, do not uninstall the Live Mesh software. In addition, do not re-synchronize any folders you had previously stopped synchronizing.

Why is this happening? A technical explanation

In the interest of being open with you, our beloved and valued technology preview testers, we’re happy to provide more detail on what’s causing this behavior. The key to a good synchronization algorithm is understanding the history of all the files in a folder. You want to make sure that any client, even one that hasn’t synchronized in a good long while, can get an accurate list of what changed and what needs to be brought up to date. Part of a complete folder history is keeping track of files or folders that have been deleted, so that if a client comes online and tries to sync a file that doesn’t exist in the up-to-date folder list, we can tell whether it’s a new file or a file someone deleted a while back but is still on this particular client because the file deletion was never synchronized to it. We use the industry jargon tombstone for these deleted files.

What we discovered recently is an interesting edge case around tombstones. Live Mesh performs its synchronization in batches of roughly 50 files at a time. The client asks for the first 50 changes, processes those, then asks for the next 50, and so on. If, through sheer chance, the next batch of 50 items is made up entirely of tombstones—that is, it’s a list of 50 files that have all been deleted since this particular client last synchronized—the storage server gets a little confused. It returns that list of tombstones and then concludes that synchronization has ended. If there are additional file changes, they never get sent to the client. So this can lead to two patterns:

If you’re synchronizing a folder for the first time, and that folder has this pattern of 50 tombstones in a row, then you get only a partial sync. Any files the server returned before this block of 50 tombstones show up just fine, but any files after that won’t be synchronized to the client.

If you have first stopped and then restarted synchronization for a particular folder, it gets a little more complicated. In this scenario, when synchronization resumes, the client ought to just quickly see which of the files have changed and get everything up to date. But if a client hits this 50-tombstone-in-a-row bug, the Live Mesh service ends up confused. The client might have tens or even hundreds of files that you and I know are up to date and part of the folder, but the server failed to tell the client about them. So the client has no choice but to conclude that you have a bunch of new files, and that’s what it tells the server. The server, of course, recognizes that files with those names already exist, and so it mistakenly thinks the client has just created a bunch of new files that unfortunately have the exact same names as files already in the folder. Result: sync conflicts, and potentially lots of them.

Now, conflicts are an inevitable part of any synchronization system, and so both the server and client are built to understand what a conflict is and to store any conflicting files in a separateholding area, where they remain until the user decides how to resolve the conflicts. While they are in the holding area, the client software might not display the files as being part of the folder, while the Live Desktop instead represents the conflict by showing both files (yes, this is something we are already working on improving and making consistent ;). So the files are all in your mesh; it’s just not obvious where they are or how to access them.

The one catch is that the client holding area is stored in the Live Mesh folder for temporary application files. This directory is something we remove when the Live Mesh software is uninstalled (also something we are already working on improving—either warning you that uninstalling will remove any files in a conflicted state or just copying all conflicts somewhere else where they won’t get removed on uninstall). So if you somehow end up in a state where the folder is present only on one client, uninstalling the Live Mesh software from that client may remove all files in a conflicted state. That’s bad, we know it’s bad, and that’s why we suggest you don’t uninstall until we fix this issue.

Thanks as always for your continued participation and feedback. Ferreting out issues like this is exactly what our technical preview is designed to do, and so while it’s sometimes a bit painful, we appreciate your help and your patience as we work to get the issue fixed.

Last week, we posted about an issue that might cause your Live Mesh Folders to either appear to have missing files, or have unexpectedly large numbers of conflicts. We mentioned that we were testing a fix and we’d update you as soon as possible…and then it got quiet here for a bit. The good news is that we’ve made a lot of progress since then, so here’s an update.

The root cause of the issue (clients not synching properly after encountering too many tombstones in a row) is fixed. We actually put this fix into production early last week, and then watched to make sure tombstones were no longer causing any issues. It’s been a week now, and we’re feeling good that we have this particular tombstone issue fixed. We grudgingly admit that we are not perfect, and there might be other bugs left in our Tech Preview that cause incorrect conflicts or synch behavior – keep reporting issues that you see, and we’ll keep fixing them.

Of course we also want to help users who previously hit this bug and have folders that are in a confused state. If you were hitting the issue where folders were only partially synchronizing, you should have already seen this fixed – the first time your client software contacted the server with the fix in production, the server should have returned the complete file list.

If you hit the issue where a folder had a large number of conflicts, we have good news for you here as well. This particular tombstoning issue affects folders in a very distinct way, since it ends up creating conflicting items whose names and file contents are identical. Conveniently, this also means that it’s fairly easy to automate clean up of such a folder. Since we can tell that there are no real conflicts (that is, the files are in fact identical as determined by hashing file contents), we can remove any conflicts and just keep one copy of each file. The net result is that we’ve written a tool which can identify folders affected by this particular bug, and then automatically clean them up. The tool runs entirely in our datacenter, it never actually accesses file contents (we create/store the hash on upload), and it never exposes any of the filename data outside the datacenter. We can ask the tool to give us back a list of LiveIDs of users who have affected folders, and then we can tell the tool to go fix all the folders that a given LiveID owns (if you create a folder, Live Mesh treats you as the owner.)

This tool should fix many of the conflicts created by this tombstone issue, but not all of them. If your folders have been out of synch, and you’ve been making changes to files on multiple PCs, well then you have actually generated a real conflict (two different copies of the file exist), and it will take your attention to decide which copy to keep.

Here’s the announcement about the tool that we’re posting to the forums with details on how to get your affected folders cleaned up:

Last week we posted a sticky in our forums containing the details of a recently discovered issue which has resulted in some Live Mesh folders being only partially synchronized. To better serve our customers, we have analyzed our storage service to identify folders that might have been impacted. If our analysis shows that you have a folder that might have been affected by this issue, resulting in either a large number of conflicts associated with your folders or fewer than expected files, you will be receiving an e-mail from us within the next three days asking whether you would like us to perform an automated clean up tool on our servers that will resolve spurious conflicts and retrieve missing files. Further details about this process will be included in that e-mail. Note that while this tool can greatly assist those with large numbers of affected folders, we cannot guarantee that all files and folders will be restored to their original state.

We will also soon be posting steps for performing a manual recovery to help those of you who have a small number of affected folders.

Finally, please remember that Live Mesh is currently a Tech Preview and that you should always backup any data stored in Live Mesh.

UPDATE: We have posted instructions for how to manually clean up affected folders here.

We are pleased to announce that our next update is now available. The new build contains a crucial Live Mesh folder synchronization fix – details below. By default your Live Mesh software will automatically update itself within 24 hours of the new build being posted, but you can always right-click the Notifier icon in the system tray and choose to force an immediate update.

Fix:

·Fixed issue where deleting a Live Mesh folder on one device, or stopping the folder from synchronizing on that device, would cause the folder to stop synchronizing on all devices.

If any of your Live Mesh folders have been affected by this issue, please use the following steps to re-establish stable sync relationships for those folders:

1.Update the Live Mesh software on all of your device to the newest version.

2.If any affected folders are currently set to synchronize with your mesh, please stop those folders from synchronizing on each of your devices.

a.Right-click on the folder and select Live Mesh Options, then click Change Sync Settings

b.For each device, under Synchronize files, select Never with this device.

3.Once you have removed the affected folders from your mesh, you can then add them back in again by right-clicking the folder and selecting Add folder to your Live Mesh.

We've had several posts here talking about Live Mesh as a platform, and we've said that we'll tell more of that story at our Professional Developer's Conference in October. The session list for PDC is now mostly complete, so you can start to get a sense of what we'll have to say about our platform:

We're working on a couple more sessions as well that we hope to unveil once the conference starts. And our VP, David Treadwell, is confirmed as a keynote speaker.

They'll be a bunch of folks from the Live Platform team spending time in the community areas at PDC, and we're looking forward to your feedback and suggestions, and hearing what you plan to build on top of our platform.