This is a big change, which was long due. The central piece of code that shuffles data from a file to its backup copy was completely redone. The previous version is still perfectly fine, but it is now several years old and we've learned a thing a two during that time.

⦁ Copying is now resumable after cancellations and IO errors. In fact,
you can now yank an USB drive from its port in the middle of a write
and Bvckup 2 will automatically correct the file corruption that this
may cause, while preserving as much of existing data as possible.

Starting with this release the backup engine will retry backup steps when they fail in a certain way. By default, the retrying is enabled for network-related failures as this is the number ones source of intermittent failures that tend to resolve themselves in a short amount of time.

The retry schedule is 20 times 15 seconds apart, configurable as per usual.

The engine already had a list of so-called "fatal errors" which cause a backup run to be aborted when encountered. This list includes errors like "Device not found", "Volume dirty" and everyone's all time favorite "Disk corrupted".

UI's ini file *was* saved in response to Windows "we are rebooting" notification, but it was initiated _after_ the UI responded to the latter. So in some cases Windows ended up nuking the program before the save was completed.

⦁ Resolved two issues with handling of file symlinks

1. The engine erroneously skipped symlinks that were pointing back into the source tree. This is a _folder_ processing logic (because we'd end up in a loop if we were to allow that). File symlinks we should be following regardless of where they point.

2. When copying a file via symlink, the engine used wrong API to query its size, so it ended up always thinking the size was 0. This would've not been a problem if the engine didn't have an optimized copying routine for zero-sized file. This routine works by creating a backup file and then closing it write away, without attempting an IO.

Previously the engine would force-flush file buffers after copying each file if the backup device was removable. In retrospect this is not really needed, because it's a duplicate of Windows' per-device write cache policy, which is what needs to be changed IF a device is indeed prone to sudden removals.

⦁ Delta copying thresholds are now 16 MB and 64 MB (see below)

That is, when enabled, delta copying is now used for all files over 64 MB and for recently-modified files over 16 MB, whereby "recently" is "within last 30 days".

As it turns out it's NOT a default behavior of native Windows controls.

⦁ "More" links are now shown in blue

In some configuration windows there's a plaintext "More" at the bottom right corner. It used to be dark gray, so some people assumed it was a disabled option and never bothered to try it... even though it changed color invitingly on hover.

As of this release, these are blue. Looks a bit more busy, but hopefully a bit more actionable too.

⦁ Resolved an issue with resuming delta copying after a write failure
due to an out-of-disk-space condition
⦁ Resolved an issue with resuming delta copying after a cancellation
that was preceded by a write failure
⦁ Resolved an issue with sending alerts over TLS
As per - https://bvckup2.com/support/forum/topic/1069

⦁ Resolved an issue with scanning getting stuck in case of network
errors under certain circumstances.

⦁ Resolved an issue with delta copying - the copying module would
erroneously deem a delta state invalid if the only change on the
previous run was to the last block of the file *and* the file length
was not a multiple of 64KB.

⦁ Resolved an issue with shadow copying not working on XP
⦁ Resolved an issue with archiving trimming reporting too many
completed steps in a presence of network errors
⦁ Added a safeguard to catch sporadic crashes in the guts of Lavasoft
Adaware when issuing SetFileCompletionNotificationModes requests

⦁ Resolved an exotic issue with ultra copier in a presence of network
errors - in some edge cases when a network connection drops some
of file system requests still may complete successfully, while most
won't. The ultra copier assumed that all will fail and that led to some
misplaced panicking on its part.

⦁ Fixed scheduler's behavior when an over-the-network real-time job
ran into a non-network related error. It's a mouthful, but basically it
means that when a share, for example, ran out of space, the engine
kept running the job in a loop, because it assumed it was a retryable
error.

⦁ Revised how the installation fingerprint is calculated. As in, Microsoft
is at it again, breaking things with recent W10 updates.

⦁ Added support for "final" qualifier to default exclude/include rules.
This is something that can't be explained briefly. Stand by for a link
to a separate detailed post...

⦁ Reworked the retrieval of file IDs during the planning phase (these
are used to track moved and renamed files) - the process is now fully
parallelized, with IDs being retrieved by a pool of threads. In short,
this step now should be noticeably faster... unless you are running
on a single-core CPU, in which case you should see no difference.

⦁ Patched up archiving code to strip leading space when archiving
files with no name but just an extension. Previously, if we were
archving ".abc" it ended up being saved as " (2018-08-01 ...).abc".
That is with a space in front of the opening brace. No more now.

⦁ Fixed an issue with delta copying when updating a file that has
shrunk since the last time and running into an error trimming
the backup copy (e.g. because of the network error).

⦁ Assorted cosmetic fixes to the backup logs, mostly to make them
read more concisely in collapsed form.

Release 79.6.1 - 79.6.4

⦁ Fixed several issues with the program complaining that "Something
went wrong" under certain conditions.

All issues were related to the logging module, or more specifically
to an incorrect self-consistency checks that were added in 79.6 as
a part of the logging module cleanup.

That is, these weren't the bugs in the actual code. These were the
bugs in statements that verified correctness of the code, in real-time.
So they caused the program erroneously panic over an otherwise
perfectly valid code.

⦁ Fixed two issues with the ultra copier. One was related to updating a
backup copy of a actively shrinking file and another - to handling of a
backup location going away precisely after all the copying was done,
but the backup file wasn't yet closed.

⦁ Fixed an issue with copying NTFS alternate streams on Windows XP.
It is basically unsupported, but the program tried to do it anyway
and panicked when it saw no required API support.

⦁ Fixed an issue with editboxes no longer vertically centering the input
text on preview builds of Windows 10. In some cases the input was
so off center that it was clipped.

This was caused by Microsoft outright breaking the handling of
EM_SETRECT message in "Edit" controls. Completely destroying
it for no apparent reason or explanation.

⦁ Fixed an issue with the Task Scheduler task that Bvckup 2 creates
to launch itself at user's logon, when set to run as Administrator.

As of this release the task is called "Bvckup 2, for %SID%", whereby
the last bit is the user account's SID. Previously, the name ended
with "... for user@domain" and that caused some issues if the user
account or the computer itself was renamed.

⦁ Fixed an at-launch issue when the engine cannot open a log file for
a backup job. In certain cases the program would panic and exit.
Now it doesn't.

⦁ Reworked the on-launch check of whether the program's window is
visible on any monitor.

There is a built-in Windows function for that, but apparently it will
sometimes fail (and show that the window is completely off screen)
*unless* the program first enumerates all present monitors. This is
most likely a quirk with a 3rd party display driver... though in the
only reported case it was NVIDIA.

⦁ Resolved an issue with destination scanning, which manifested in
logging entries being recorded at the wrong log level. Internally,
the issue was due to not marking destination scanning phase as
such, so the engine ended up re-using source scanning rules for
the destination scan. In the vast majority of cases this made no
difference, but in some cases it did, potentially causing more files
to be retained in the backup than required.

The issue was introduced in 79.8 release, so an update is strongly
recommended if you are running 79.8.

⦁ Changed "archive_modified" override to NOT require deleting option
to be set to "archive" as well. That is, it's now possible to set a backup
job to archive just modified files and not deleted ones.

⦁ Resolved an issue with file system information not being saved for
newly created jobs. This caused this information to be re-queried on
every run, which in turn caused a real-time job to run back-to-back
even when there were no real changes.

⦁ Reverted default window size to its original values. An earlier change
was to let Windows size it, but in some cases it resulted in a window
being set too wide and tall.

⦁ Fixed an issue with querying destination file system and device
information.

In some cases the program would try and do it before backup folder
is created, logging one or two errors as a result of that. Operationally
this made little difference in the vast majority of cases, but it was still
a bug, so fixed.

⦁ Fixed an issue with erroneous "Login required" failures after a
successful authenticated connection to a publicly inaccessible share.

That is, if you were backing up to/from a share that _required_ an
authentication and that wasn't accessible otherwise, then on the
first backup run the program would successfully connect to the
share (using credentials supplied), but then it would bail out
erroneously thinking that it still didn't have a connection.

⦁ Fixed an issue with redrawing pushpin icons in the Backup To/From
fields in the Backup Settings dialog when collapsing "More section"
of the window. In some cases there would be some junk drawn in
place of the icons, because the latter weren't re-drawn after the
window was transformed.

It's a mouthful for this - when copying larger files, the UI will show
the progress bar and the %-age done so far. Depending on the
copying speed and size of file, the percentage may show one or
more decimal places (up to 3) to make sure there _is_ some non-
trivial activity shown even for slow-going copies. The issue was that
this bit was broken in recent releases and the UI always showed no
decimals.

The context is this - Shadow Copying service may refuse creating a
volume snapshot if another snapshot is being created at the very
same moment. This is an inherently retryable situation, so this
release adds just that - the engine will now retry up to 10 times 30
seconds apart when running into this condition.

Retry parameters are configurable through the INI.

Retrying is enabled only when "More Options" > "Shadow Copying"
is set to "Require", and not for the "As needed" setting.

⦁ Speeded up saving of destination snapshots at the end of a backup.

⦁ Suppressed logging of deduplicated files during the scanning phase.

These were logged because dedup'd files are reparse points and RPs
were unconditionally logged before this change.

The issue surfaced when running without Administrator privileges.
About 20 minutes after launching the program the UI would stop
receiving updates from the engine, so while you could start a job
with Go and it would execute, the UI wouldn't actually show any job
progress.

Another symptom was that when an activation code was used to
license an installation, the UI would confirm the activation going
through, but then keep on saying that the program was still in trial
mode. That too was due to the UI not receiving respective updates
from the engine.

# Kudos to Andy, Lanka, Michael and John for helping with
reporting and tracking down this issue.

For those on their coffee break, here's some light reading as to Why
this was happening.

The post-mortem

The cause of the issue was a trivial mistake in the code that tried reading some VSS-related information without first checking that VSS was available. This led to an "Access denied" fault, which should, under normal circumstances, trigger a crash. So it should've been a really visible issue that would've been caught in testing on the first pass.

However in this case the code in question was in a callback set with SetTimer() function. When the top-level UI code called DispatchMessage(), the control went to user32.dll, which issued the callback, the callback ran that code and the code crashed.

You'd think what difference does it make, a crash is a crash... Yes, exactly. Welcome to the club.

Apparently, callbacks issued by SetTimer() are framed by a pair of RtlActivateActivationContextUnsafeFast() and RtlDeactivateActivationContextUnsafeFast() calls, which set up and tear down their own SEH handler. So the timer callback code can crash all it wants and you'll be none the wiser.

So what happened in Bvckup's case is that it would receive a timer callback, which will run its internal event loop. The loop will dispatch event handlers and one of them will throw a SEH exception, which will be promptly eaten by user32.dll instead of being passed up to the UI for reporting.

This left the UI thinking that its event loop was still active, so on the next timer callback the UI would see that the loop is still busy and it would skip running it - hence completely stalling the event processing, including the reception of the updates from the engine.

Long story short - set your exception handlers for all your callbacks from Windows API or end up seeing your code behave in a way that makes no sense whatsoever.

⦁ Changed "Create Backup" flow to allow destination folder to be a
subfolder of the source, with a warning. Previously this was only
allowed if destination was the OneDrive folder, and it was flat out
disallowed in earlier releases.

The root cause was that the service and UI processes have their own
microsecond-resolution clocks that are used for high-prevision things
such as IO timers and some such. These clocks are started when the
process starts, so in service mode they end up being offset relative to
each other (because the UI typically starts some time after the service).

So what happened is that the start time of current backup step was
recorded by the engine using this clock rather than the system clock.
It then passed it to the UI for display, but on UI's scale it was offset
and appeared in the future => hence the delay.

⦁ Fixed an issue handling out-of-disk-space condition when saving
destination snapshot at the end of a backup run.

⦁ Resolved an issue with bvckup2.exe starting to eat ~10% of CPU once
machine's uptime goes over 49 days.

This was a good one.

For internal housekeeping tasks the engine uses a millisecond
resolution clock that is stored as 64 bits. This is plenty enough
for storing milliseconds, but there was one place in the code that
treated it as 32-bit.

The problem was with the code that scheduled timers. You tell it
that in X ms you want a callback, and it gives you a callback around
that time. Timers.

This code sorts all scheduled timers into logarithmicaly-spaced
buckets similar to "timer wheels" as used in the Linux kernel [2].

The problem was with a bit of code that determined how far out the
next timer was from _now_. It erroneously treated milliseconds as
a 32-bit value, so after (2^32/1000/60/60/24) days of uptime this
code invariably decided that the next time was already past due.

This caused the main loop in the engine to spin without sleeping
and that led to elevated idle CPU usage.

BUT. You are rightfully thinking - how come this only affects 79.15?

Excellent question. Prior to this release RDTSC value was offset so
that it would read 0 at the moment of _program_ launch. This meant
that the 10% CPU issue was still there, but it didn't manifest itself
until the _program_ was up for 50 days. This is not a common case,
so the whole issue remained unnoticed since... well... pretty much
version 0.1.

Then, in 79.15 this RDTSC offset was removed to sync internal clocks
between the engine and the UI. This resolved the first "Fixed" entry
on 79.15 change log, but it also pushed the CPU issue into the
spotlight as there *is* a fair number of boxes with good uptime.

⦁ Resolved an issue with backup jobs auto-disabling themselves. In
particular this happened when the program was running in service
mode *and* if the UI was not launched within first 30 minutes from
the service launch.

Again, this was specific to 79.15 because it included a rework of the
UI-to-engine communication protocol. One part of the change had to
do with verifying program's self-consistency and one of the checks
failed if the UI wasn't present. This led to the program entering a
"safe mode" of sorts when it disabled a backup job after a run as a
precaution.

⦁ Resolved an issue with post-backup delay for real-time jobs not being
observed consistently. That would be the setting behind "Edit details"
next to "Backup Settings" > "When to backup" > "In real-time" option.

More specifically, when set to copy files in full (and when copying files
smaller than the delta copying threshold) the engine defaults to
using so-called "staged copying". That is, it would copy the file into a
temp file first and then rename it in place, overwriting an older copy
if one exists.

The fix is in response to the report that described "created" time on
a backup copy somehow getting out of sync with the source, even
though the timestamps sync completes without errors. The fix is
_tentative_ because file tunneling is the only plausible explanation
for this lovely behavior, and because the report came in late in the
release cycle.

⦁ Resolved an issue with erroneously suppressing copying of "created"
timestamps when a quirky Synology volume is detected at _source_
location.

⦁ Added "etc_name" and "etc_path" variables to the pre/post-backup
script env variable set. Etc_name is set to the name of job's settings
folder (e.g. backup-0003) and etc_path is set to the folder's full path.

⦁ Added stricter validation of responses for storage device queries as
some of these may come from 3rd party drivers and be not exactly
up to Microsoft's specs.

⦁ Tidied up the log produced by the storage tracking module.

⦁ Resolved an issue with the engine not "seeing" removable drives that
are first safely ejected and then plugged back in.

⦁ Resolved an issue with "Copy line" and "Copy block" options being
mixed up in the log viewer's context menu.

First, drives that appear offline after program's start, but can be
"resuscitated" with Windows' help are now correctly recognized as
online after connection is re-established. This bit got broken in
79.18 with introduction of the new storage tracking module.

Kudos to Greg for reporting the issue and helping with isolating it.

Second, the engine now also detects network drives going offline,
which is something it didn't do before at all.

⦁ Reworked reporting of ini/override warnings.

In particular, this relates to reporting bad override entries on job's
reload. As of this release, the engine tracks which configuration
setting was picked up from which INI file and it will report this file
when logging an INI warning.

SD card readers of certain vendor will for reasons unknown sometimes
report that "the volume is not ready" when the volume is in fact A-OK.
In other times they would refuse to report drive's geometry or partition
information. The program can now tolerate this eccentric behavior.

The workaround itself is to treat a reparse point as a regular file IF
its reparse tag is less than 0x00100000 (hex) and if it has a Sparse
attribute set. This can be disable in job's settings if needed.

⦁ Assorted UI tweaks and changes, including faster double-buffered
custom control painting, revised and unified color palette (for reds,
blues and grays), patched up tabbing order is several dialogs and
fixed hints in "Add new filter" window.

⦁ Resolved an issue with handling of a swap file when running out of
disk space during the copying phase of backup.

⦁ 79.21.1 fixes a small color issue with the Welcome Guide.
⦁ 79.21.2 fixes an issue with progress bars not showing progress
during scanning and planning phases.

Keep in mind that the latest revision won't show up via in-app checks for updates if the installation already runs a 79.21 release. Instead it can be downloaded from https://bvckup2.com/update for a manual update.

⦁ Resolved an issue with internal delta state clean up. This is referring to
the removal of block-hash state files after respective source file is either
deleted or shrunk down to no longer qualify for delta copying.

⦁ Resolved an issue with updating empty files when a System attribute is
set on their backup copy.

Previously this would fail with "Access denied" error due to a certain
(rather obscure) provision in the file system API. With this release
Bvckup2 will clear this attribute off the backup copy prior to updating
it. This is an extension to an existing mechanism that clears ReadOnly
and Hidden attributes, so the program now basically checks if any of
these three attributes are set on the backup copy and clears them if
they are.

⦁ Reworked storage monitoring to be asynchronous. The initial scan of
the storage stack is still done synchronously (that is, the program
waits for it to complete before continuing with its initialization), but
all subsequent updates in response to device/volume activity is now
done asynchronously, in the background.

The main reason for the change is that despite all the efforts to avoid
making any blocking or slow requests, there are still cases when even
the most trivial request may take several seconds to complete. This
includes cases of not fully spun up HDDs, dying drives that are still
unsure if they want to die completely or not, etc.

⦁ Added support for requerying faulty storage volumes. This is related
to the previous change. Apparently there exist device/drive combos
that announce arrivals of new storage volumes before these volumes
are actually operational. So if the program attempts to query basic
volume details _immediately_ after its alleged arrival, there are cases
when this request will fail.

Previously, a volume like this was assumed to be faulty and ignored.
So any jobs bound to this volume remained in "Waiting for device"
state and didn't get to run.

With this change, the program will try and re-query the volume once,
in 5 seconds after the original attempt. If this too fails, only then the
volume will be marked as broken and ignored.

⦁ Added support for verifying file size after copying. With this release
the copying module will check if the resulting size of a backup copy
as reported by the file system matches up with the expectation.

This is added for one very specific case - for FAT volumes (not FAT32
or ExFAT, just plain FAT) that accessed remotely, through Samba.
Apparently in this case it's possible to create a file larger than 4 GB
in size, with all writes completing successfully, but the resulting file
size will be 0. This is some issue with Samba/FAT combination, so
this check is basically meant to detect this combo and warn about it.

⦁ Resolved an issue with the storage monitoring module. The size of
the storage stack information was growing by 20 bytes per device,
per rescan due to cross-references between devices and volumes
not being cleared correctly.

This is obviously not a big problem unless there's a lot rescanning
going on... which is exactly what happened in one reported case,
where a dying flash drive was repeatedly generating "physical
layout changed" events. This caused the stack information size
swell in size, eventually growing over the maximum size of the
UI-to-engine protocol message... which triggered an internal sanity
check in the code and shut down the program.

In good news, that sanity check worked as designed.

⦁ Resolved an issue with scanning errors of the top folder - that is, if
the program ran into a _retryable_ error when just starting to scan
either the source or destination folder, the retrying didn't work as
expected. Fixed now.

⦁ Resolved an issue with the About window - in some cases it would
show an "action" button, i.e. "Reactivate", partly off the screen and
have a problem with showing the right license status.

Revisions

⦁ 79.23.1 automatically expands "How to send" window if advanced
options are configured.
⦁ 79.23.2 fixes an issue with not being able to open Backup Settings
window if DACL or SACL copying is enabled *and* email alerts are
enabled. This was due to an invalid self-consistency check for the
state of the window when it was put into respective state.

Keep in mind that the latest revision won't show up via in-app checks for updates if the installation already runs a 79.23 release. Instead it can be downloaded from https://bvckup2.com/update for a manual update.

The storage monitoring module now queries SMART/NVMe health
data from local storage device, if possible.

When the program starts up, it first does a quick device scan just to
collate basic device information. This scan is done synchronously -
the backup engine waits for it to be completed before entering
the "running" state.

Then, once started, the engine then polls devices for their SMART
data and this is done asynchronously, in the background, because
retrieving this data can be slow and it may also require spinning up
the drives (in theory it shouldn't, in practice - sometimes it does).

After that it will also query SMART data from new devices that get
plugged into the machine. This too is done asynchronously.

SMART data is not (yet) refreshed periodically, so this version does
not _monitor_ SMART status per se. Nor does it actually report it
through the UI. Health status of the devices is logged in the storage
log and the summary of it is also logged in the central program log.

The main goal behind this change is to complete gradual roll-out of
the new storage module and evaluate it against larger installation
base:

79.18 introduced the new storage monitoring module, sans SMART.
79.23 switched the module to use asynchronous scanning.
79.24 enabled the gathering of the SMART data.

If all goes well (it should), the next step will be the tracking of SMART
data over time and integration into the UI as a first-class feature.

⦁ Reworked the handling of faulty storage volumes - this has to do with
how the engine deals with volumes that can't be queried for their
basic information.

79.23 introduced a single re-query attempt under certain conditions,
but these conditions proved to be incorrect and sometimes caused
the volumes to be re-queried perpetually, in 5 second intervals.

⦁ Cleaned up storage module logging - it's just marginally better
looking and a bit more pleasant to read... if that's your thing.

⦁ Resolved an issue with the storage module - in certain cases the
volume extent information (offset/size) were recorded incorrectly,
which caused the tracking ID of respective device to become ill-
formed, throwing off device tracking. This should be fixed now.

⦁ Resolved an issue with UI transitions - in certain cases the UI would
erroneously trip an internal self-consistency check when validating
the layout of a window after it was transformed from one state to
another. This affected More options" page of Backup Settings,
"Device Tracking" and "How To Send" dialogs.

⦁ Support for indicating rudely interrupted backups - that is showing if a
backup run was aborted part-way through because Bvckup 2 process
terminated unexpectedly. Details - https://bvckup2.com/wip/30042019

Previously, if a backup job was set to "Keep backup copies of items
deleted at source", then it would still emit a log entry for each delete
operation that was planned, but ultimately suppressed.

With this release these delete operations are logged only if there's
a non-delete operation that depends on them.

For example, if a file called "abc" was replaced with a folder called
"abc", then backing up this change requires first deleting the backup
copy of abc-the-file. Since this is disallowed, the folder cannot be
created and so the file-delete operation *is* logged to establish the
context for folder-create failure. Edge cases, gotta love them.

⦁ Resolved an issue with the file move detection logic. In particular,
if a symbolic link was replaced by a file with the same name, this
caused the planning module to first emit delete-link, create-file
operations, which were then erroneously coalesced into a move-file
that was, ultimately, tried to rename file into itself and that triggered
a self-consistency check in the backup engine when it was trying to
execute this nonsense.

⦁ Added support for "vertically maximized" window state - that's the case
when you double-click on the top or bottom of the window frame and it
gets stretched out vertically to the max. With this release opening and
closing Log Panel now works just like it does if the window was properly
maximized.

⦁ Cleaned up error reporting a bit - sometimes the log needs to record
a failed condition rather than a query (that may fail in a variety of ways).
Prior to this release these failures were recorded with a fake error code
of 12345, i.e. it would say "VssIsOkToContinue() failed with 12345".

This nonsense is no more. It will now simply say "Xyz() failed".

⦁ Upgraded the toolchain used for assembling the program. Moved
from v120_xp platform toolset to v140_xp, which largely means
using a newer (and possibly better) compiler and a new version of
the C run-time library.

Incidentally, the latter also caused program's EXE to swell by almost
40KB just because. On the plus side, this helped fishing out a couple
of smaller typos with log formatting... Still, at 20K in binary code per
typo it's a bit steep.

Revision

⦁ 79.26.1 resolves an issue with (sometimes) not properly handling
read requests completing out of order when using a network source.

⦁ Resolved an issue with the program failing to initialize its window at
launch (and complaining that something went wrong). This was due
to support for "vertically maximized" window state that was added
in the previous release.

This state is not officially documented, so detecting it was based on
heuristics, which proved to be insufficiently strict and prone to false
positives on certain Windows 10 builds.

Fixed these to be far more conservative.

⦁ Resolved an issue with the diagnostic console caused by the switch
to the new platform toolchain, again in the previous build.