The pre-release forum

Topics

R81 pre-release / Parallel execution

The central new feature in Release 81 is its support for parallel backup execution - an ability to execute multiple backup steps at once.

Parallel execution

Prior to R81, the file system scan and file ID retrieval were already done
in a parallel fashion, using several threads.

Release 81 extends this behavior to all phases of backup, allowing the program to execute all suitable operations in parallel with each other.

If you are familiar with the /mt switch of robocopy, it's a similar idea, but with automatic thread count management and several other important improvements.

Use cases

In practical terms, parallel execution results in a *very* noticeable speed increase in the following cases:

⦁ Backups with a lot of small- to medium-sized files
⦁ Backups that require creating a large number of folder
⦁ Backups that require deleting a large number of folders or files
⦁ Backups that "touch" a lot of folders

⦁ Fixed an issue with handling failures when _retrying_ a backup of a
_new_ file.

More specifically, if a new file appears at source, the backup is run and
it fails with a retryable error mid-copying. For example, a network
connection goes away. This will cause the program to retry the copying
after a pause. If this retry would also fail with an error, then bvckup2
would pop up the "something is wrong" message and shut down.

⦁ Fixed an issue with the retrying mechanism panicking when the last
attempt fails. There was an invalid self-consistency check (an assert),
which is as ironic as it gets. A bug in a bug checking code.

⦁ Reworked retrying logic that is used to re-execute failed steps in a
presence of transient errors (e.g. network disconnects). This also
fixed an issue that caused self-inflicted crashes in cases when
copying of a _new_ file succeeded after several attempts.

⦁ Reworked logging module to report step info as it's being executed
IF this step happens to be the only one active and no other steps are
being scheduled until it completes.

This happens when bvckup2 is backing up a very large file, at which
point it will finish all other active steps, but won't queue any new ones.
This is done to allow the bulk copying to proceed in peace if you will.

Prior to 99.5 all step-specific info was recorded into a backup log only
after the step finished. However if there's exactly one step in progress
we can dump its progress into log as we go - which is exactly what 99.5
implements.

⦁ Another pass over the retrying-on-errors logic. Apparently, a network
hiccup may not necessarily kill _all_ active operations. Sometimes it can
cause just some of them to error out, while the rest will keep on going.

⦁ Reworked all UI animations to use FPS-based sequencing. Previously,
animation used the "run for X ms using Y steps". X and Y were hand-
coded and because of that the frames-per-second value (FPS) was all
over the place. With this release there's now a single global FPS value
for all animations and another for all fade-ins and fade-outs.

⦁ Resolved (hopefully the last) issue with the retrying logic - there was a
race condition in the code that preemptively cancelled queued, but not
yet executed steps when the program ran into a retryable error.

⦁ Fixed an issue with program closing down when needing to trim the
$Archive. Trimming is also done in multi-threaded (parallel) fashion
now, but it was done _before_ the parallel execution was configured.
No config => no way to trim => whoops.

Kudos to Greg for the traces.

⦁ Fixed an issue with overly strict self-consistency check in the retrying
code. Had to do with retrying the backup of _new_ files. If this sort of
operation was queued and then cancelled due to other steps failing
(e.g. because the network connection went down), then in some cases
the program would panic by seeing a new-copy step in a retry state.