Apply retention policy on Tentacles failing after upgrade to 3.12.6

After we upgraded to Octopus version 3.12.6 today deployments started failing all over the place. They all fail in the "Apply retention policy on Tentacles" step with the same error. Reading the release notes it looks like this is related to the changes around long directory paths. The relevant paths don't can't be considered long, but they fail anyway. This is a blocking issue for us so I hope you can provide a quick fix

Thanks for reporting this issue. As a temporary workaround could you disable retention policies on your Lifecycle? It will not require a new release to be created.
Is it happening for all environments or only a few?

You're right, that looks like an issue to do with the changes around long paths, but not because that's specifically a long path. It looks like a library we're using to deal with long file paths is failing on a recursive delete.

To help diagnose, can you have a look at the folder it's trying to delete and let us know what's in it? The one in the log you pasted is: C:\Applications\master.Build.Testlab\Confirmit.TaskSystem\22.0.374.

There appears to be a folder tree under there that's causing the exception when Calamari tries to delete it: Tasks\customcode.

Could you try deleting the whole 22.0.374 folder manually (ideally using the same account that Calamari is using) and see if that succeeds?

This has been deployed on hundreds of machines and hundreds of releases prior to 3.12.6 so I don't believe it is an issue with the directory itself

I think I've found what is triggering the problem: All the directories that fail on delete are actually junctions. The junctions should not be traversed, but just deleted. This has worked fine in the past.

We typically use junctions to add some shared libraries/folders to the applications. Has worked fine for years.

I've had a look and this will require some work to resolve. Bear with me, I'm trying to work out how to downgrade your Calamari to the previous version - that's likely to be the safest option until we can create a fix.

It turns out to be easy to reproduce. Just create a junction in one of the deplyoment directories that will be cleaned up. Retention policy will always fail.

In the fix, make sure that the retention policy don't delete the content of the junction target directory. This is the behavior prior to version 3.12.6. Deleting the content of the junction target directory will have VERY bad effects

A downgrade of Calamari sounds like the best short term option. I've blocked all attempts for deployment to production now until this is resolved or Calamari is downgraded on all hundreds of machines

It looks like you may just be able to deploy the previous version of Octopus Server over the top, and it will downgrade the Calamari version for you when the next deployment occurs. The version is checked as part of the deployment, so it should have immediate effect.

Are you able to try this and let me know if it resolves the issue for the time being?

I'm really sorry for the problems here - it's an edge case we didn't consider. And yes, definitely noted that we don't want to delete the contents of the junction.

I'm not an expert on Chocolately, but I think you can downgrade an installation using these instructions. I'd recommend not doing an uninstall and reinstall (although that should still work), but doing an install and specifying the previous version instead.

The tentacles should be fine running a version higher. Because it's a patch release, the versions should be compatible.