There are several ways in which a program can query security
attributes (Owner, Group and ACLs) of one file and copy
them to another file.

In a true Microsoft's way, some are well-documented, but they
don't work as advertised, and some work as needed, but they
are not documented.

—

First hit on Google is Get/SetFileSecurity. The name
checks out, it accepts a HANDLE, so it should be quick and
simple way to do what we want.

Nope, it's not.

It's an older API that doesn't correctly handle inheritable
ACLs and its use is not recommended. Not by the API documentation,
mind you, but by a random blog post from one of many Microsoft
development teams, which you still need to find first. What
they recommend instead is Get/SetSecurityInfo.

And you'd still naively think it's the one.

—

Nope, it's not.

The problem is that SetSecurityInfo will fail with "access
denied" if you are trying to set DACL and the file is not
opened with READ_CONTROL.

You may wonder why the heck it needs read control...
excellent question. That's because it wants to look up
the name of the file and then *feed that name into SetNamedSecurityInfo
function*. The latter will then re-open the file and start
spraying kernel with requests, which is the last thing you
need when you are already quite busy with copying a
million of files from A to B.

Looking at the trace of SetSecurityInfo with Procmon shows
that it generates over a dozen of kernel calls...
whereas only one is really needed.

—

The name of that one call that we want is
NtSetSecurityObject, but it's a part of
so-called Native API and as such it is officially
off-limits for any application use.

However, if we are to comb several pagefuls of
relevant APIs,
we may spot something called SetKernelObjectSecurity,
which looks suspiciously like NtSetSecurityObject.

—

Long story short, you want to copy security
attributes - use Get / SetKernelObjectSecurity.
Now you know. Congratulations :)

The answer is that alt. streams is as a
generic and more flexible replacement for an older "extended attributes"
mechanism. It is meant for the cases when you may want to pin an extra
bit of info to a file (or a folder), the info that is application-specific,
but that doesn't belong in a file itself.

The book example is a small tag that browsers add to executable files
downloaded from the Internet. This tag is used by Windows Explorer to
show its "Ugh-oh, something from the Internets" warning when you are
trying to run such files. The tag is stored as an alternate stream called
Zone.Identifier. You delete this stream - you won't get the
warning.

Long story short, Bvckup 2 now can replicate these secondary data streams.
This option is in Backup Settings > More Options >
Copy Also section and it is Off by default.

New version of
Backup Settings > More Options > Copy Also
is coming up in Release 78.6.

—

• File attributes are out.
• Owner and Group are now a
single multi-state option.
• DACL is now called Security info.
• SACL is now in, as Auditing info.
• Alternate NTFS streams are in as well.

The rationale behind File attributes option removal
is that it was the only one from the list that was
familiar to non-professional users. That instilled a false
sense of confidence when looking at other options, enticing
to tick them all on.

From now on the list comprises exclusively... erm...
alien-looking stuff, so hopefully this will deter people
from randomly enabling options they should not really be
enabling. This also lets those in need of a complete
replication configure it now directly in the UI.

Here's a way to present two closely-related options
with a single checkbox with 4 distinct states.

It is very rarely (if ever) that anyone would need to copy Owner
information without also copying file's Group info. However,
these are separate bits, so bunching them together
doesn't seem like a right thing to do. But giving each of
them their own piece of UI real-estate doesn't feel right
either.

So how can we pack them up?

—

First, let's look at how this can be done conventionally.
We'll just be verbose about it and list all the options as
is:
This is OK in terms of usability, but when taken as a part
of the whole it makes window look cluttered:
And *unnecessary* cluttered at that, because in a vast
majority of cases Owner and Group will be either both checked
or unchecked. Ditto for Security and Audit options.

So let's try something else.

—

Windows natively supports something called "3-state"
checkboxes that can be in checked, unchecked
or indeterminate states.
"Indeterminate" is a rather strange term as if
the program is not sure in which state the option is.
However we can think of it as an in-between or
partial state and do something like this:
This is not bad, however we lose the overall meaning
of the option when it is in a partial state:
So let's put the context back:
Better, but now we are back to things being too
verbose and looking cluttered. Let's simplify
it for two most common cases:
Better, but now it's off visually:
How about with chop the specifics and move them to
the side:
Almost there, just need to patch up visual hierarchy
a bit to help telling principal parts from the
secondary ones:
And that's it. The result is an UI that looks clean
and simple for two most common cases, fully supports
marginal ones and packs 4 options in a space of two.

One is an option
to hide revoked licenses from the list and another is
a support for automatically adjusting all displayed
dates/times to browser's native time zone, with an
option to override if needed. Previously all times
were displayed in CET time zone (because that's
where our backend servers are).

Release 78 will introduce changes to file/folder filtering
system to make it a bit more flexible and logical, while
also keeping it as point-and-clicky as possible.

—

In pre-78 releases if you excluded a folder, you weren't
able to then go and include some of its subfolders or files.
Similarly, it wasn't possible to exclude some of the items
in an explicitly included folder.

In fact, it wasn't possible to exclude anything
at all if using the Start with an empty list mode
or include any item when using the
Include everything option.

—

While this approach mapped on a simpler code and sort of
made sense when it was first coded, in retrospect it wasn't
that good. So, R78 will come with a fundamental overhaul of
the filtering module... and whatever new and exciting bugs
that may come with it.

—

What the middle pane now does is it allows quickly setting
the action (include or exclude) for folder and files by
simply clicking on them. For files the action is final -
if you exclude it, then it is excluded. End of story.

For folders however it also sets a default action
for all its files and subfolders. This action may
be overriden for any item by clicking on it, which simply
reverses the default.
So when bvckup2 looks at what to do with any given file
or folder, it will first pass it through this set of
filters, yielding a preliminary include or
exclude action.

Next, the item will be passed through generic filters,
from the bottom pane.
First hit on any of these
determines the outcome. If there are no hits, then
the default action it is.

—

There are also new, finer controls over the filtering
flow. In short - it's now possible to mark any generic
filter as final, which will suppress generic
filter matching for all folder contents.

— Bonus material —

If you are to think about all this for as looong as
I have, you might realize that the Include everything
/ Start with an empty list selector at the top,
this one -
is actually no longer needed!

All it does is merely sets the default action for
the top directory - and we may just allow doing that
by simply clicking on it in the middle pane.

However I think having it a separate option helps
making the UI a bit more approachable and easier to
use, so it will stay.

As you might've heard the log viewer happens
to be one of the most complex parts of Bvckup 2. It looks
simple, but on the back it is anything but.

—

1. Log files are created by the backup engine,
but displayed by the UI module. In desktop mode this
is easy enough to handle, but in service mode the engine
runs in a process of its own and under a different user
account, meaning the logs are simply inaccessible to the
UI.

So what ends up happening is that the engine opens logs
on behalf of the UI and passes open file handles back
to it.

PS. Ever wondered what all those "process" handles
are doing on the
DuplicateHandle() argument list? Exactly to
allow one process to do something privileged on
behalf another process.

—

2. Log files are
rotated,
so what you see in the UI is actually a seamlessly
glued together list of all log entries across all
copies of a backup log.

This would've not been a big deal if it weren't for...

—

Finally, the UI needs to expand/collapse log
parts and do
it nearly instantaneously. This is a hard problem,
especially considering that logs can grow very
big.

As the engine writes out the logs, the UI scans
them and creates an auxiliary index, which it then
uses to quickly understand what should and should
not be shown in any given state.

These ndecies cannot be kept in memory, not in full.
Instead the UI stores them in a specialized on-disk
database, which too needs to be very quick.

—

However, all this complexity had a nice little side-effect.
It made adding the Error view very easy.

All the UI had to do is to build another index and include
just the error entries and their immediate context. This way
switching to/from the Error view is a simple matter of
swapping one index file for another. That's it.

This might not be the most exciting thing ever, but still
has some elegance to it, doesn't it?

One of several tweaks coming up in Release 77 - the
on-hover extension of long log lines.

Windows comes with native support for this sort of
functionality and it's called "in-place tooltips".
For example, they are used in Windows Explorer
when displaying file and folder names that don't
quite fit the allotted space.

But in a true Windows fashion, while it works,
it also... flickers!

Blinkety-blink. Here's a slo-mo in all its glory:
So after killing several hours on trying to make it
behave, it became clear that it's faster and simpler
to just redo it from scratch.

It is internally known as Perks for Tweets and
it allows receiving a bit of a discount in exchange
for mentioning Bvckup 2 on Twitter.

The main reason we are doing this is to try and reach
out to new people, so the fine print is as follows.
Your Twitter account should be at least a month old
and you should have some number of followers and updates
(tweets) on record. A tweet should include a link to
bvckup2.com and/or an @bvckup
mention (but without being directed at @bvckup).

Keep in mind that this is one of them wild marketing
experiments, so things are subject to change in the
next few weeks.

Starting with 76.9 when the app is switched to run as a
service, the engine will now run under a dedicated
user account.

The account is called "Bvckup 2 Service". It is
automatically added when the app is switched to
the service mode and removed when the app is switched
back to the desktop mode or uninstalled.

The account is made a member of Administrators and
Backup Operators groups, which gives it the same set
of privileges as provided by the LocalSystem account.

Ideally, it should've only needed Backup Operators
membership, but in some cases this group lacks network
access privileges, so the default is to add it to the
Administrators group as well.

—

It is still possible to install the service under the
LocalSystem account - click the More button at
the bottom left of the Switch Mode dialog and
flip the Run service under option.

—

This change has been long coming as it's an altogether
better and tidier setup. It helps isolating Bvckup 2
from the rest of the services, eliminating effects of
any custom LocalSystem policies that might be in place
and it also allows for a fine-grained control over its
permission set. So if another Windows update comes out
and (again) breaks things, it should be much easier to
hammer it all back in place.

Reworking the welcome popup that is shown on
the first launch. it will now briefly go over
the differences between file and
system backups, explain how the trial
works and what licensing options are available.

This set of sketches shows various styling
options for the index on the left hand side
of the window.

First is a cosmetic change to how an inline editing
is done - click on a Pen icon, start typing, press
Enter to save, Escape to cancel (or click on respective
button if that's your preference).

—

Second change is a bit more interesting - it adds support
for deactivating licenses via the dashboard:
Please note that this feature is currently
reserved for site license and larger resellers
only, and it is enabled on a case-by-case basis.

Another change in upcoming 76.5 is a new
warning that is shown when copying of
owner, group or DACL attributes is enabled
for a backup job that goes across the
computer boundary.

As you might guess the warning is due to
what we are seeing on the support end,
when some people enable some backup options
merely because they look reasonable, but
not because they actually need them.

Added support for pre-seeding backups and
adjusting source or backup locations when
the on-disk data is being moved around.

Previously, if either source or destination location
was changed in backup settings, the app would purge all delta
copying data and the snapshot of destination if present.
This caused destination to be re-scanned on the next run
and all files to be re-copied in full next time they
needed an update.

With the next relase (76.5) the app will now ask if
the backup state should be purged or not. This way if
you are setting up a remote backup job, you may first
pre-seed the backup by attaching the drive locally and
making a backup on it. They you'd take the device to
the remote machine, change Destination path in the
backup job, say No to the state reset and - voila -
you are back in business as if the drive has always
been remote.

Device ejection has to do with "safe removal" of USB
drives, the same thing you can do by clicking on a
standard Windows "safe remove" icon in the system
tray before yanking drive's USB cable out.

However, in a true Windows fashion ejecting a media
sometimes also ejects a device and sometimes you can
also eject media from an external USB HDD (and, yes,
this makes absolutely no sense).

—

In any case, the next maintenance release (76.5) will
get ejection support, but without the UI part. It will
be possible to set it up with a good old INI file edit
only.

Then, next full release (R77) will get the UI support
and above sketches are for that bit. The tricky part
is that ejection settings are ultimately per-location
preference, so they are needed for both source and
destination drives and this takes 2x of screen estate
to accommodate.

Secondly, ejection is available only for local drives
and not network shares, so the Option 2 series aren't
really good. They all allows enabling ejection for
non-local drives, meaning that we have to do extra
"sanity checks" and error messaging when validating
backup settings. That's just... messy.

In comparison, Option 1 simply doesn't show ejection
option for non-local drives, just like it doesn't
show device tracking (the pushpin icon) in that case.
So it's tidy and logical.

This is a development test of new parallel copying
feature that may be making its way into the next release.

If you are familiar with robocopy,
it's an equivalent of its /mt option, which causes several
files to be copied, well, in parallel.

As it's probably obvious from the above screenshot it works
really well, especially for smaller files when an overhead of
opening and closing file is comparable to the bulk I/O time.
Here, for example, C:\Windows\Fonts
folder with about 1500+ items is getting copied to a Synology
NAS over 1Gbps link.

This feature is going to be available under a Pro license only.
More on this to follow.

A list of storage device vendors
as observed by Diskovery to date.
I must admit that I haven't even heard of at least 3 of these.

—

Here's few more notable findings from the aggregated analysis
of 200+ logs submitted so far:

Top three vendors - Seagate (19%), WD (18%) and Samsung (12%)

219 HDDs and 58 SSDs - 79% and 21% respectively.

The only SMART attribute present in all SMART reading was #12 -
the power cycle count.

All but one device also reported power on hours and reallocated
sector count.

This is good news, because the "reallocated" attribute (#5)
is the principal predictor of an imminent drive failure. Once you see it
starting to climb up, it's time to change the drive.

The temperature attribute (#194) was reported by 71% of HDDs and
51% of SSDs. However some devices report temperature readings in
different attributes, so these percentages may be off.

Attributes in the 17 to 159 are unused (with an exception of one SSD that
reports something #64).

Lots and lots undocumented attributes. Some vendors have incomplete specs
and others (e.g. Seagate) just don't release SMART specs at all. This means
that we can't even label some of the attributes.

Lots of inaccuracies in the specs too, meaning that it's hard to reliably
interpret the attribute's raw values.

Raw value is a 6-byte sequence
of vendor-specific data, so you *must* have a spec to understand them. And
you do want to understand them, because that's where the actually interesting
data is.

Regrettably some specs would say that it's just a 48-bit counter, whereby in
reality it'll be a 16-bit something followed by a 32-bit counter. Furthermore,
certain vendors (e.g. Kingston) tend to change the format not just between
their product lines, but between sibling products that are just a model away
from each other. It's really quite nuts.

Finalized the last change that will go into R75 -
full UI support for advanced filtering rules that
have been supported internally
since last August.

More specifically, the UI can now be used to set up
rules for matching on the file size and
created/last-modified times. Attribute matching
has also been extended to support
more exotic values such as Offline and Don't-Index.

Added an option to copy parts of the log to the
clipboard - either just a single line or the line
and all its children (the block).

If the block is large (over 64K), then the app will
also display its size on the right-hand side on the
menu. The neat thing is that this information is
readily available in the
log index,
so this requires
no log scanning of any kind or no interim memory
allocations.
Basically, it's as
O(1)
as it gets.

With the next release the UI will start adopting
a new "progressive refinement" approach for
managing advanced and less-frequently used
features.

After trying this and that, the design ended up
converging
to a simple More button at the bottom left
corner of the dialog that, when pressed, injects
additional settings into the dialog.

This is certainly not something new and revolutionary
and it must've been done before, but it does feel a
bit more natural compared to a more conventional
approach of popping up a separate window with all
the advanced extras.

Also, the initial iterations had the More button
toggle
between "More..." and "Less..." when clicked, but
it looked a tad too busy, hence the current revision
of a simple "sticky" background highlight when in
the expanded state.

Added support for displaying both effective and
all supported SATA signaling rates for physical
drives. These are also interchangeably knowns
as "transfer speeds" and "link rates".

Detecting supported signaling rates is
trivial. After all, it is a part of the drive's
ATA IDENTIFY block,
word 76.

In comparison, drive's effective signaling
rate is a bit trickier to determine. In theory,
it is also a part of IDENTIFY block, word 77. In
practice, this is only available from drives that
support ACS-3, which is a more recent revision of
the ATA command set.

So for drives that don't indicate their effective
signaling rate Diskovery looks at "phy link rate"
as reported by CSMI-compliant
drive controllers.

But, wait!

What if both rate readings are available and
they are different? Not possible, you
say? "Ha-ha" says an outdated Intel RST driver
that ships with Windows 8.1. In this case
Diskovery will default to the drive's version
of the value and also issue a warning of the
discrepancy.

Diskovery can now also talk to the
Virtual Disk Service
and extract all sort of interesting information from it,
including exact volume type (spanned, striped, mirrored,
etc.),
its online/offline status and its health.

With Diskovery coming together,
I wanted to take a moment and show what a UI sketching
process for an app looks like.

—

In a vast majority of cases native UI controls (widgets)
work just fine. There's really no point in re-skinning,
say, a button unless, of course, your goal is to make it
stick out like a sore thumb.

There are however cases when the app needs to display
information in a way that is not natively supported
by the OS. In other cases, while it might be possible
to render the data with a native control, it won't do
it well or the result won't look good.

In these cases you have no option but to go with a
custom UI, but, luckily, if you start with a need
(rather than a quest for bells and whistles) the
process is fairly predictable and logical.

—

Diskovery deals with displaying the bits and pieces
of the computer storage stack - from physical drives,
to parititions, to volumes, to logical drives and
mount points.

These fall naturally into
a hierarchy,
with each level comprising a list of key-value(s) pairs.

So taking the first stab at the UI, we would quickly arrive
at something like this:
Primary list of drives and volumes on the left,
with details of a selected item on the right, organized
into a tree of "key:value" entries.

This is functional, but the right pane is not exactly easy
to follow. So the first change will be to tabulate the key-value
pairs:
Next, note that items in bold act like section
headers, so they don't really need to be collapsible.
So we do away with the buttons and connecting lines
between top-level items:
This gets us a bit of extra horizontal space and trims
a useless, but still actionable UI element.

Next, a couple of tweaks to how we render selected items.

There's no need for any items on the right to be selectable
at all, so we won't render a selection highlight in the
right pane.

The left pane can use a less jarring selection style,
consistent with how Windows itself
does it.
Next, we try and improve readability of the data on the
right by helping to guide the eye from a key to its
value.

The simplest option is the grid lines:
At this point, we take a sip of our cold coffee, squint
an eye and assess if we like how it looks.

It's not bad, but we can try something else:
Now, that's not bad either and it has a nice
visual consistency with the selection bar in the
left pane.

However a minor nit here is that we now have bold items
(section headers) paired randomly with row
highlights. See how Storage space is on white
and Partition table is on light gray?

Moreover, adding or expanding items will cause
this pairing to flip-flop, e.g. we add an extra
row about Partition table and it now
sits on the white background:
That's not a disaster, but we can do better.

The idea is to recognize section headers as such
and style them independently:
With this change in place we can just restart the row
highlight sequence at the top of each section and
all's good.

Next, we apply the same styling to the left pane
as it happens to also comprise of sections:
Another sip of coffee, re-squint the eye and
the whole thing now looks a bit cluttered and
busy.

Easy enough to rectify by adding a bit of
vertical padding:
Next, we remove vertical connecting lines leading
from section headers to the top-level items.

However, we remove them on the right, but keep
them on the left. This is because on the left we
have a hierarchy of devices, so
connection lines make sense. It also seems
to look better this way.
Next, we move the expand/collapse buttons to
the right.

In this particular app expandable items contain
secondary details that 99% of users will not ever
need. These items will also be collapsed by default
(even though they are expanded here, but that's just
because we are sketching through the details).

This change also helps decreasing the leading pad
of top-level items and providing for a more compact
horizontal packing of the data.
Alright, so that's about it as far as functional
tweaks go. So now it's where it gets subjective. We are
going to spruce it up a bit.

It's all about colors, padding and little nuances.
If we were designing for the web, it would also
involve texturing, but for a desktop app texturing
is a sure way to ruin native look and feel.

In any case, moving on - invert the headers from
dark-on-light to light-on-dark:
Add a bit of hue to the row highlights, tint their
edges a bit and also flatten the look of
expand/collapse buttons:
Splice in a scroll bar and make sure our precious design
still looks reasonable:
Re-style the left pane to look more like an actual connected
hierarchy of disks and volumes. The rationale here is to make
it more obvious that items on the left are clickable.
Almost there. Take a final look and adjust vertical padding
and spacing to de-clump items a bit more:
And here it is. Compare to the
starting point and you'd
probably agree that there's at least some improvement.

—

Needless to say,
one man's extra vertical padding is another man's complete
waste of space, but any customization is always a combination
of rational and subjective choices. The former is what gives
an app the ease of use and the latter is what give it the
personality. Can't have one without the other.

Diskovery is a little companion app that throws
a simple UI on top of the new S.M.A.R.T. diagnostics
module of Bvckup 2.

The app will go live shortly and its main purpose is
to let us estimate how well the diagnostics module fares
against the real world hardware. Because there are specs
and there's life.

In particular, certain USB drives and older RAID
controllers don't always cooperate fully (or at all),
and it would help to know what- and how common they are.

Once released, Diskovery will remain publicly available
and we'll update and support it. It's a simple tool for
quickly inspecting one's storage hardware and I can think
of a number of cases where it can come handy.

When connecting to a mail server over SSL/TLS, it's not
uncommon for the server to present a certificate that
doesn't match its hostname.

In particular, this happens when a custom domain is used
with a managed (shared) mail service. You'd grab a domain,
set its MX to mx.domain, add CNAME for mx.domain
to point at mx.mail-service and - voilà! -
you are served with mx.mail-service cert when
connecting to mx.domain.

Ultimately, this is a mail server mis-configuration issue -
your MX should really be pointing at mx.mail-service.
In theory this shouldn't be happening, but in practice it does.

—

To address this, it's now possible to specify the exact
domain the app should be looking for in an SSL
certificate:

In the above example, you'd simply set this field to
mx.mail-service and that's it.

—

To make things a bit easier that app now also helps
you to pre-populate this field with the actual domain
from the server's certificate.

It does that when you send out a test email and the
field is left blank. As you can see from an opening
screenshot, the app will recognize the certificate
mismatch and re-run the test, accepting any and all
certificates. It will then make note of the domain
in the cert and copy it into the "SSL domain" field
for you.

—

Finally, setting "SSL domain" to * will effectively
disable all certificate checks and force the app to
accept any certificate that the server sends, even
if it's an expired or a self-signed one.

Earlier versions had general and alert-specific settings
mixed up. For example, "From" doesn't
really belong to the alert configuration,
while "Include" from
General Settings does.

This has been sorted out.

A larger picture here is that the app will be getting
additional alert types - e.g. for S.M.A.R.T. diagnostics
and backup verification - and this requires a proper
structure to support these cleanly.

—

Also, while at it, the UI got a new framework for
handling window transitions between different states.

For example, in the above screencap there's are 5
primary states and 2 intermediate ones. Prior to R75
moving window between these states required quite a
bit of hand coding. This is no more, so we can
now animate the hell out of any window. Consider
yourselves warned :)

The age of cars trying to drive themselves and rockets
trying to land on sea barges. Graphics cards are effortlessly
churning photo-realistic images in real-time, mining crypto
money and cracking passwords.

And here we have Windows still defiantly refusing to use any
of that double-buffering nonsense when re-painting one of its
basic common controls. Because who wouldn't want a bit of
flickering?

Tab control for multi-page configuration windows in general,
and for the email notification configuration in particular.

Prior to R75 there was one primary email alert, which was
sent on backup completion. Now the app is getting other
types of alerts, so the UI needs to be adapted to support
that.

—

The idea is that the config window for specific alert type
will give access to both the alert details and general
email settings that are recycled between all alert types.

So here are several sketches for organizing page index
of a multi-page configuration window. Nothing terribly
novel here, it's more just a matter of getting styling
details right - make sure it's obvious that it's an
index and that its items are selectable.

—

Option Z (right-most) looks OK, but it's not obvious
that index items are actionable as they look like
static section headers.

Option Y is a bit better, but the arrow actually
points at "From" field and you can bet some people
will end up in a stupor because of that.

There's a good guiding principle coming from Paul Graham that says
make something people want.
It's very simple and it's very effective.
It also makes real-world user opinions a critical part of any
good product design.

Unicode,
as you may know, is a standard that assigns pretty pictograms to
numerical codes from the 0 to 0x0010FFFF range.

In simpler cases
pictograms represent letters, numbers and symbols. In other cases
they are decorative glyph elements, like umlauts, that are meant
to be combined with the letters.

—

There are also Unicode encodings - UTF-8, UTF-16, etc. -
these deal with how specific Unicode numerical IDs are translated
into sequences of *bytes* for storage, transmission, etc.

The simplest encoding is UTF-32 - it simply stores Unicode ID as
a 32-bit number. This is however rather wasteful as at most 21
bit is needed to represent any Unicode ID.

Prior to Windows 2000, Microsoft used a simple 16-bit encoding
that mapped every Unicode ID from the 0 to 0x00FFFF range to the
exact same 16-bit number. This is called UCS-2 encoding. The
Unicode range was smaller back then, so this approach worked.

Then, Unicode range grew and IDs no longer fit into 16 bits.
So to address this, UCS-2 was modified to encode symbols
from the 0x010000 - 0x10FFFF range using two 16-bit numbers,
from the 0xD800 - 0xDFFF range. Exact details of this encoding
are not important, but what it meant was that some 16-bit numbers,
e.g. 0xD812, were no longer valid Unicode symbols just on their own.

This is called UTF-16 encoding, and this is what Windows uses
at the moment.

But wait !

As Apple likes to say after a screw-up - "it turns out" that
despite routinely talking about UTF-16 in their API documentation,
Windows file system still uses plain 16-bit encoding for the file
names!

For example, "\xD812Haha" is a valid file name even though it is
*not* a valid UTF-16 string. This in turn means that if the app
is converting file names into other encoding, it should go easy
on full UTF-16 compliance checks and be ready to digest ill-formed
UTF-16 string.

—

Bvckup 2 keeps all text in UTF-8 format internally, so it does
quite a bit of conversion from/to UTF-16 and this leads to some
interesting things when the app runs into an malformed UTF-16
name. Like those created by *ahem* Adobe *ahem* After Effects
*ahem*, for example.

The solution is simple - try and convert as if we have the UTF-16
encoding, but fallback to UCS-2 when running into invalid UTF-16
sequences. Also, apply the same approach when converting back to
UTF-16 and all will be good.

Release 74 also implements new delta copying defaults
and the results are ... interesting :)

—

Delta copying, when enabled, has always been used conditionally
and the criteria was the size of the file. For smaller
files the overhead of setting up the delta copier is
larger than any potential savings, so these files
are simply copied in full.

The file size threshold was initially set to 128KB and the
value was inherited from the original Bvckup version,
where it made sense because the I/O was generally slower
back then.

Fast forward few years and 128KB is no longer a good default,
so as of Release 71 it was increased to 2MB.

—

With Release 74 the criteria changes again.

First, the threshold goes up to 32MB.

Second, the delta copying will also be used for files
between 2MB and 32MB, but only IF
they were modified in the last
30 days OR if they were previously delta-copied.

The rationale is pretty simple - we try and not
bother with delta copying unless files are large
or they change frequently.

—

This change has two main effects.

1. Smaller backup state.

Delta copying stores per-block hashes in a backup's
configuration directory. This is done for every file
that is delta-copied, so the fewer files are, the
less data ends up being in the \deltas folder.

The above screenshot shows the \deltas folder stats
after backing up \Windows\System32 directory. And
here are the same stats after backing up with an
older version -

That is, twice the disk usage and nearly 10x more files
and folders in the \deltas folder.

2. Faster overall backups.

For smaller files
the overhead of setting up the delta state is comparable
to the time needed for the actual copying. We skip
the delta setup -> we complete the copying faster.
Magic.

Before:
After:
That's nearly a 10% speed up. This is obviously specific
to the contents of \Windows\System32, but it's not hard
to extrapolate from here.

Prior to Release 74 app's memory usage was
proportional to the number of items in the
backup. This was because both the source and
destination file trees were kept in memory
in their entirety, as was the backup plan.
While not a big deal for smaller backups,
it does translate into quite a bit of RAM
when the file count starts to climb into
a million range.

—

Release 74 implements the off-RAM storage for
large data sets. In previous release the run-time
arrangement was like this -

With the new release the objects are kept in
the app's memory only if they are in active use.
The rest is trickled down into a swap file for
storage, through a couple of intermediate caches.

The object cache holds fully assembled instances
of application data objects and its size is kept
at or below preset item count.

When the object cache grows too big, least recently
used objects are written down to the swap file. The
writes are passed through a typical page cache to
coalesce the I/O into larger chunks.

—

This lunch is obviously not free.

There's a cost to
even successful cache lookups and there's most
certainly an expense to doing the disk I/O.

The app mitigates this by using aggressive cache
optimization (buckets, MRU lookups, etc), raw
IO caching and also by allowing keeping swap
files in a custom location, and presumably on
a different physical drive. But, again, all this
starts to really matter only when backups grow
into a million item range.

While reasonable defaults are provided, the cache
sizes, the trimming policy and other details are
fully configurable through the .ini files. The
app may also be set up to log cache/swap performance
if required.

—

This change is big, it took nearly two months to
implement and it means that Bvckup 2 can now handle
backups of unlimited size. To say that it's
a very big deal would be an understatement.

Still working on major engine changes to cap its
run-time use (and I will be posting data from the
test runs shortly), but meanwhile here's something
simple that I'm considering adding to the next
release.

It's an option for the app to report your backup
statistics to one of our servers and then being
able to display these stats as a simple PNG served
over HTTP from the same stats server.

It's really meant for trying to motivate other
people to back up their things as well as to
indirectly promote the app as well. And, yes,
I lived through the 90s, that's where I got the
idea from :)

Currently working on the improvements to the
scanning and planning modules of the app. No pretty
UI screenshots to show, just some performance numbers.

—

The ultimate goal of this rework is to put a cap on the
run-time memory usage, which in turn is needed to better
accommodate very large backups (tens of millions files
and up).

—

The app comes with a formal backup planner, which is
a module that accepts two full directory trees (one
for the source location and another for the backup one),
digests them and produces a list of simple steps that,
when executed in order, bring one tree in sync with another.
The app is then diligently goes through the list
and creates, updates, renames and deletes files
and folders as directed.

An alternative to formal planning is to to traverse
the trees and make on-disk changes just as you spot
the differences. There are several drawback to this
approach, including not being able to estimate the
amount of work needed, to understand disk space
requirements and to detect file/folder moves. It's
just an altogether messier and less predictable way
to go about the whole thing.

That's how Bvckup 1 worked and that's how robocopy
works, for example.

—

Long story short, having a backup planner is the right
thing to do. However, it means that the app needs to
have two full file trees readily available for the
planner to digest. If these trees are kept entirely
in memory, it may get expensive with large backups.
At 100 to 300 bytes per item, a million file backup
means 200 to 600 megs of RAM just for the trees.

That's
quite a bit, but more importantly this memory usage
grows linearly with the size of the backup. This is
no good.

—

The solution obviously enough is
to not keep full trees in memory. This is a fairly
complicated endeavour and it maps onto several large
changes to the app inner structure.

One such change is nearly done. Both the scanning
planning modules no longer require for the trees
to be resident in memory at all times. Instead,
they now use them in a piecemeal fashion.

Behind the scenes, trees still reside in memory for
the time being, but the underlying rework has some
nice side effects. Trees now take less memory, they
are faster to populate, faster to save and load
to/from a file, they take substantally less space
on disk and they compress better with NTFS
compression.

This will ship as a part of R74.

After that, R75 will replace in-memory trees with
version that can swap out tree parts to the disk,
significantly reducint the run-time memory footprint
of the scanning/planning process.

As you may or may not know, Bvckup 2 uses a
dual-module architecture of the inside.

One module is the backup engine. It maintains
the list of backup jobs, schedules, executes
backups and generally does other useful
things that don't require direct user involvement.

The second module is the user interface. This
is the part that displays the app's window and allows
users to monitor and control whatever the engine
module is doing.

The principal point is that the engine and the UI
are isolated from each other. They share no
run-time data and communicate with each other using
a messaging protocol.

This isolation is what enables the app to run in
service mode.
In this mode the engine runs in its own process,
as a system service, and the UI runs as a regular
desktop app.

It was all well and beautiful, but there was,
however, a caveat.

—

The caveat was the backup logs.

Logs are inherently shared data - they are written by
the engine and they are read and displayed by the UI.
Logs tend to be bulky, so it is shared app data
of a special kind and it requires special handling.

Prior to R73 the app included a third module (in its
dual-module architecture, no less) - the log manager.
It was a complicated contraption that abstracted log
access for the engine and the UI, and the biggest issue
with it was that it behaved completely differently
depending on which mode the app was in.

In desktop mode, the manager was writing *and*
indexing logs on behalf
of the engine and the UI was merely displaying them.
However, in service mode, the engine did no
indexing, just the writing, while the UI was making
copies of all logs for its own use and then it was
indexing *them*.

This is exactly the kind of non-uniform conditional
processing that the dual-module design was meant to
avoid!

—

Enter R73.

The separate logging module is no more. The engine no
longer does any log indexing, it is now entirely the UI's
reponsibility. Indexes can now be rebuilt on demand at
any time, and finally, the UI no longer creates its own
copies of the log files. Instead, it works with the
engine to gain access to the engine's copy of the logs,
using
DuplicateHandle
to move open file handles across the process boundary
when needed.

The UI now can display logs while it is still catching up
with them.

Think -- running in service mode and launching the UI only
once a week. On each launch
the UI needs to index a week's worth of log entries it hasn't
seen before. Prior to R73, the UI would just sit there and
show "synching ... XX% done". Now the sync is done in
the background (as per the screenshot).

The UI now can also recover gracefully from indexes getting
corrupted (e.g. due to a cold power cycle or blue screen).

Just as important, the underlying code is now much simpler,
lighter and more uniform, the executable is 8KB smaller
and the developer is generally happier. Good news all around.

Running an extended set of tests to fine-tune the
performance of the Bvckup 2 bulk copier.

It's an inherently simple matter of determining
an optimal size and count of read/write buffers
used to ferry data from source to destination.
There are however ... erm ... details.

—

A source can be local or it can be over-the-network.
There are slow local devices (e.g. a USB2 drive) and
there are fast local devices. There is a warm, there
is a cold cache and there are files that are cached
only partially.

The writes can go out in lots of smaller chunks, all
in parallel, or with a smaller number
of large buffers. Again, it can go to a local device
or over the network. The app can also explicitly
flush all buffers after copying or leave flushing
to the system.

—

The above graph is the I/O throughput of a 0.5 GB
file being repeatedly copied from the local HDD to
Synology Diskstation over a Gigabit wire. What
changes between the runs is the buffering strategy
and as you may notice there is a difference.

Still need to run several over-nighters to gather
more statistics, but there is already a couple of
things that can be changed in the copying code to
make it run faster.

Upon even further consideration it looks like
the From field is not of a prime interest either,
so it too can go into the Details.

This cuts down the number of fields and renders
section headers unnecessary, so they too can go.

Basically the idea is to reduce the dialog down
to fields that actually need user's input and
stash everything else away... I don't know if
it's the right thing to do though. Perhaps having
a summary of the whole email setup (including the
subject, From and the message) is not such a bad
thing. Not sure. Need to sleep on it.

Release 71 is getting email notifications, so
here's all you always wanted to know about teaching
your app speak SMTP.

First of all, let's get
MAPI
non-sense out of the way. As per usual, it sounds
good in theory, but an awful mess in practice. So,
no, MAPI is not an option unless your goal is to
burn through your support budget and/or practice
your communication skills with less-than-happy
users.

SMTP is an inherently simple protocol - send a text
command, receive a text response. Repeat. Open a
socket, connect, send(), recv(), done. A couple of
hours of work at most. Ha.

—

It starts to get complicated when you throw TLS in
a mix to secure the communication with the server.
In particular, this is required when you want to
send emails in a way that requires logging into a
server first.

There are two ways in which SMTP can be combined
with TLS.

First is when you take an existing SMTP session
and say STARTTLS. The server says OK and then
you basically stop talking SMTP over your TCP
connection and start talking TLS. Then, once the TLS
handshake completes, you go back to SMTP, but now it
all goes over the TLS session, nice and encrypted.
This is a standard way, that's what every decent
SMTP client should do. This is TCP/25 and TCP/587.

Second option is SMTPS.
You first establish a TLS session and then start
talking SMTP over it. Just like HTTPS. Apparently,
this is non-standard and discouraged way, but
still widely used. This is TCP/465.

—

Then, there's a matter of how to add TLS
support to the app.

The obvious choice is OpenSSL, but that will cost
you 200-300KB in extra binary code, so unless you
already have OpenSSL linked in, it's not really
an option, not for a *backup* app.

Digging through MSDN you may come across something
called SSPI
or Security Support Provider Interface. It is an
attempt to stuff several security protocols like
SSL and Kerberos into a single framework in a faint
hope for a miracle of reuse. One of SSPI providers
is a mysterious entity called
"Secure channel",
which, upon closer inspection, turns out to be ...
yes, TLS!

So, the next step is try and work through all the
abstraction fluff of SSPI to make it actually do
the TLS handshake. The idea is actually quite neat
- you can create a stateful object (context) that
takes in data you get from the peer and spits out
some data for the peer. At some point
it completes the handshake and becomes available
for accepting app's own data, which it
encrypts and hands you a blob to send to the peer.
You would then also get blobs from the peer, pass
it to this context/object and get back plaintext
data for the app. Whether it's TLS or ROT13 on the
back is completely transparent to the app.

That's the idea. In reality, the documentation is
subpar and lacking and there are exactly two examples -
one is incomplete and another one is buggy.

The good news is though is that once this bloody
contraption gets off the ground, you get full TLS
support at the cost of just several kilobytes.

—

With TLS done, there's still a bunch of smaller
things to be taken care of, e.g. lack of asynchronous
DNS query API in pre-W8 windows. Not a big deal,
nothing a spawning of a worker thread would solve,
but still ... a bit here, a bit there and it all
adds up.

In other words, it's all coming together, but as
per usual it all takes longer than anticipated.
Shocking, I know :)

When configuring or reconfiguring a backup,
the app would automatically enable/disable
device tracking when a location is changed
to use a different drive.

Since this change happens behind the scenes,
the app will attract the attention to the
fact with this little animation. When hovered
over, the tooltip for the pushpin button will
explain what's up and suggest reviewing the
changes.

My only concern is that this still might be
a bit too subtle, but it's better than what's
in there now... which is no notification at
all.

This is one of those changes that look almost trivial
when done, but behind the scenes take a disproportionate
amount of time to actually flesh out.

In this case, the main problem was that the feature
needs a detailed explanation and the question,
of course, was how one can fit a bunch of text into a
reasonably small space so not to have this text take
over the actual configuration section.

—

The first iteration was the good old message box. You
click on the "More on this..." and you get :

Nice, but looks like a hackjob, does't it?
So, naturally, the
next step was to spruce it up a bit - put text into
a dedicated window, use RTF to add a bit of in-text
formatting and a bit of animation to
liven things up :

Nice again, looks better, but feels off. Too much
going on. It looks like a conventional help window,
but ... crippled. It's also not clear how to
act when the user wants to resize the window - allow
it, not allow it, restict minimum/maximum size, etc.

In short, what's lacking here is the cohesion.
Luckily, the solution is right on the surface:

... whereby the text panel slides out on the "More
info" click.

This is better, but now we have a
window that sits lopsided on the desktop when the
panel is open. It is also becoming wide, meaning
that the app needs to mind the desktop edges,
re-center the window, etc.

But if we just sleep on it, we might suddenly decide
to make it an "either-or" window and then the UI
magically becomes
(a) compact
(b) cohesive
(c) balanced and
(d) with unlimited text space!

So there you have it - a new mini-help subsystem
coming to Bvckup 2 in the Release 71, deployed
on as-needed basis across the app.

Release 71 is going to reshuffle the contents of
the backup configuration window a bit and location
fields are getting equipped with a "device tracking"
icon as a result.

In other words, for a network location the app will
display the "key" icon (see /wip entry immediately
below) and for local paths it will show "tracking"
icon. Both fronting respective dialog for configuring
the details.

Release 69 also adds an option of simulated backups
or, as they are also known, "dry runs".

The option is accessible via the right-click context
menu and, when invoked, it puts the backup through its
preparation phase, including snapshot creation, scanning
and planning steps, and then dumps compiled backup plan
to the log without executing it.

Simulation also executes the archive pruning without
making any actual changes to the destination.

VSS writers are the components of the shadow copying
framework that are tasked with ensuring consistency
of specific files in VSS snapshots. For example,
when creating a snapshot, Windows would ask the MSSQL
writer to flush any pending database changes in a way
that would make on-disk copy of the database self-consistent.

Some of the writers are required and always participate
in the snapshot creation process, but some are optional
and they need to be explicitly enabled by the initiating
app. Starting with R69 Bvckup will be enabling all optional
writers and it will also provide a mechanism for excluding
specific ones if needed.

Cleaned up overall look and feel a bit, got rid
of baby blue theme and anorexic body font,
reworked the copy, expanded Features section
and so on. Still need to slice and code it.
Also asked a couple of people for the feedback,
so waiting on that too.

The plan is to have a 14 day trial that doesn't require any
explicit licensing. Just install and run. It will pop up a
notification half way through the trial period and it will
also say something on the last day.

Once the trial period is over and until a production (paid)
license is installed, the app will go into an unlicensed
mode and it will activate two restrictions (see screenshot).
It will remain largely functional, but presumably not
convenient enough for regular or unattended use.

—

You would then hit /purchase page on the website, go through
the purchase process and get an activation code in return.
The code will then need to be sticked into the activation
form on the client (see 2nd screenshot) and it will fetch
the license from the server.

If the licensing server is inaccessible for whatever reason,
there's a couple of ways to retrieve the license manually
(see last screenshot).

—

For those with Beta licenses, they will be accepted at the
checkout as a discount code.

For those looking at large deployments / enterprise setups,
there will be an option for automating licensing process.
Get in touch for details if you have questions.

When hopping through errors in a backup log, log viewer
will now briefly highlight next entry if it cannot be
pulled directly under the cursor. This basically adds
a visual terminator to the interaction sequence, which
is a surprsingly useful thing to have.

Having thought about it a bit more, I can even
remove that slight upward slope from the hump
of the blue graph. The usage still climbs just
a little bit during the run, but for practical
intents and purposes it's flat.

Wrapping up changes to the logging system. Here's
a quick graph to demonstrate the effects of the
rework.
For an initial full run of a 100,000 file backup
memory usage is down from about 200 to 50 Megs.
For a 1,000,000 file backup the effect is similar
- peak memory usage dropped
from 2.9GB to 0.58GB
.

—

Previously, the UI would keep logs of last few
backup runs in the memory and for larger backups
this would eat up a good chunk of memory. This
was especially true for the initial backups where
all files needed copying and thus for 100K files
it would generate close to half a million log entries.

For subsequent backups it wasn't a big deal as they
would handle just the changes and generate only a
handful of steps and a bit of log data So in the
end, the memory usage was capped, but the cap was
higher than it could've been.

Beta 60 addresses that.

—

Only a very small part of the
log is now kept in memory, just enough to fill the
log viewer panel in the UI. The rest is kept on the
disk and read from there on as-needed basis.

However this is easier said than done.

Bvckup logs
are hierarchical, meaning that they are
generic trees
with nodes that may have an arbitrary large number of children.
Running a 1 million file backup would yield a
"Processing..." entry with 1 million children,
each with up to a dozen of descendants.

—

Displaying this tree would've not been a big deal
if all log entries were always visible at all times.
When the UI would've wanted to show entries 700,503
through 700,523 entries, it would've simply gone
to a respective location in the log index file and
plucked the data out. But...

The nodes can be collapsed and expanded. This hides
and shows arbitrary chunks of the log tree and then
the UI has this issue of finding 700,503-th *visible*
item. And this is hard.

This implies quick look-ups in some sort of super-imposed
binary search tree of visible items. A tree that changes
every time a node is expanded or collapsed and that sits
in a disk file, which restricts how you can walk and modify
it without getting an I/O penalty.

This is fun. This is what cost me 3 weeks of shuffling
through CS books and several complete rewrites of the
tree indexing code. Believe it or not, but this is
basically an original research and it seems that no one
had to deal with this issue before. At least not publicly.

—

The solution is to use slightly customized version
of an AVL tree
to store the list of node's children (that's using
tree within a tree), use it for the visible item
look-ups and then optimize the hell out of the index
I/O through aggressive write caching.

When all i's are dotted and t's crossed, this yields
a logging system that can push up to 1 mil entries
per second on a resonably equipped box. This might not
be knock-your-socks-off impressive, but it's plenty
enough for the task at
hand.

This is a part of an effort to further reduce and cap app's
run-time memory usage. More specifically, it is meant to help
the app cope gracefully with multi-million file backups that
involve millions of steps and generate lots and lots of log data.

—

At the moment Bvckup's UI keeps backup logs in
memory. It puts an effort to trim them to a
reasonable size, but this still eats quite a
bit of RAM, especially with large backups.

—

It costs about 120 bytes plus the size of an entry to
keep a single log line in memory, whereby these 120 bytes
help weave entries into a hierarchy (a tree) and track
their state.

If we want to reduce the memory usage, then the most obvious
optimization would be to keep the actual entry text in a disk
file and load it on demand, when it's actually visible in the
log viewer.

This certainly helps, but as the log grows, the memory
usage still climbs. A million file backup generates
about 4 million log entries. At 120 bytes per entry
that's 0.5 Gig in just supporting structures. This
is unacceptable.

—

So it means that the entire tree structure of the log
needs to sit in a disk file and to be read from there
as needed.

Disk-based tree structures are routinely used in
databases and file systems to store indecies of
data sets, typically in a form of a
B-Tree and
its variations.
The kicker is that these structures are meant for
storing sorted data and they optimize
for searching. Trying to adapt them
for storing unsorted data would be nothing short of
fitting a square peg in a round hole.

—

Long story short - there are no ready-made code for
storing and manipulating generic data trees on a disk,
leave alone fast code.
This appears to be an esoteric problem that everyone
solves on their own.

The screenshot above shows a part of a small "tree
database" library that I ended up writing, together
with its own caching module, predictive page loader
and a pony.

Took almost two weeks.

—

On a bright side though, this code is perfectly reusable
for storing disk snapshots as well and this paves way
for making the planner module work almost entirely off
the the memory.

Once done, Bvckup memory usage will NOT depend
on the particulars of a backup at all - this a very
big deal and an incredibly improtant feature to have
for any robust backup software.

This one -
the *nix implementation of Windows Networking. If you have
a NAS box the chances are that it's using Samba to support
network access from Windows computers.

—

It is a good software, simpler and much snappier than its
native Windows counterpart, but in case of NAS devices it
typically runs on Linux. This means that the underlying
file system is anything but NTFS.
This in turn translates into no support for "created"
timestamps and no support for certain attributes,
including System and Hidden.

Moreover, the attributes that cannot be set or cleared
depend on whether it's
a dot file
or a regular file.
Who would've thought? :-|

From what I understand it *is* possible to set things up
so that Samba would record these timestamps and attributes
and make things transparent to its Windows clients, but
in reality this doesn't seem to be happening.

—

Long story short - starting with Beta 49
Bvckup will now
be explicitly testing destination file system to check
whether c-times are supported and which attributes can be
set.

Additionally, it will also explicitly test both source
and destination file systems to determine the granularity
of c-times and m-times. This is again thanks to some
NAS devices that are customized to use 1 sec for m-times
and 1 usec (micro) for c-times.

It starts with the above window and directs
users to the website to pick and purchase a
license code, which is given on spot and also
sent in the receipt email.

Then copy-paste the code, click on Verify,
get this -

If this works (the server is accessible, the
license code is valid, etc), then this is it.
Thank you very much, click Ok, you are done -
Otherwise, it explains the failure and offers
either to retry the activation or to use an
alternative -
There are two alternatives, both are simple -
For offline boxes and locked-down setups
there's the Copy To Clipboard option. Take
the URL to a networked machine, open it in
a browser, get a licensing file back and
then drop it into the app's config folder.

Prepare the email option creates a new
email with default email client and puts the
Installation ID, Fingerprint and License Code
in the message along with some pleasantries.
The license file will be attached to the reply
and it will simply need to be dropped into
%LocalAppData%\Bvckup2\engine.

Like ID, the fingerprint changes with hardware upgrades
and OS reinstalls, but, unlike ID, it changes only in
parts. This makes it suitable for detecting incremental
changes to the user's computer.

Both ID and Fingerprint identify user's installation,
the difference is that the ID is used on the app side
and the Fingerprint - on the licensing server side.

—

Next, ID and fingerprint are submitted to the licensing
server, either by the app itself or by a human through
a website. When licensing for production use (as opposed
to beta or evaluation), the request also includes a
license code - a random token that is issued to the user
in exchange for payment.

The server responds with a license, which is a short
text file that binds together few pieces of information
and seals them with an
RSA
signature of the licensing server.

For beta and evaluation licenses the "license code" is
generated automatically by the licensing server. The
large block of gibberish at the bottom of the license
is the RSA signature.

—

When the app starts, it loads the license and verifies
it using public key of the licensing server that is
embedded in the app.

If the license is valid, the app works as normal. If
the license is missing or expired or doesn't match the
installation ID, the app disables certain features and
periodically nags to do the right thing and what not.

—

Now, let say I upgraded my box. This screws up the
Installation ID and invalidates the license. The
app recomputes new ID and Fingerprint and sends
them to the licensing server.

This is where Fingerprint comes into play. The server
takes old and new fingerprints, chops them into
individual hashes and does a "fuzzy" match. If enough
hashes match, we can assume an incremental software or
hardware change on user's end and re-issue the license
for new Installation ID.

If fingerprints are wildly different, then we make fuss
and require contacting human tech support.

—

In conclusion, needless to say that the whole scheme
relies on the validation code in the app being trustworthy.
If someone pries the app open and replaces selected
IFs with NOPs, then obviously no licensing will work.

The answer to that is that the app should implement
self-consistency checks and deploy counter-measures
when these checks fail. This is a separate and very
large and interesting subject ... which I have no plans
on covering here :-) A good starting point here would be
fravia.org archives
and then follow from there.

Basically the idea is to show back/forth arrows only
on hover and make them pull next error to the spot
where the cursor is. So to hop through errors you
would just go click-click-click without moving the
mouse.

This compliments well existing feature of jumping
to the first error of the last run by clicking on
"Xxx errors" in backup's main window entry.

As per previous post, this leaves a question of how
one would get to the *nearest* error if none is
currently visible and the answer is that one would
simply use conventional scroll for that.

All in all, having played with this for a bit I
think it's a good solution. If it doesn't seem
like one to you, give it a try and it will grow
on you.

The above screenshot is the first option for navigating
between the errors in the log pane -- see two red arrows
next to the "x" in the pane header. The second option is
this -

This time the next/previous arrows are in the error entry
itself.

—

I like the second option better, because it's contextual
and I think it's more obvious. However, first option is
superior in functionality because it allows jumping to next
or previous error
when there's currently no error visible in the pane.

First, when the update server is processing an
update check from the program, it now takes into account
program's current version.

It uses this version to compile a consolidated
list of all changes between program's version and
the latest version available. It then uses this list
to understand if there are any critical or important
outstanding changes. This yields an importance
indicator of the update, when one is available.

Secondly, both the list and the indicator are now
sent back to the program.

—

On program's end, the indicator translates into the
"Update is recommended" line. It is one three options
with "optional" and "critical" being the other two.

"Recommended" is a routine update that adds this and
fixes that. It would better be installed, but it's
not the end of the world if it's not.

"Critical" is critical - "you'd be sorry if you don't
install it" kind of update.

An example of "optional" update is a new version that
updates program's French translation when the user is
running an English version, i.e. an update that can be
ignored altogether.

—

There's now also the "View changes" button that
expands the window and shows a neatly formatted
(RTF) version of the change list.

The biggest deal, of course, is that it's a consolidated
list of all changes between the installed and the
latest version, rather than a laundry list of all changes
since the dawn of time. It makes the list infinitely more
useful compared to the raw version.

Adding support for running external command
before and after each backup.

Both commands have a configurable timeout
value. Additionally, the pre-backup command can
be tagged as critical, in which case bvckup checks
the exit code and aborts the backup if the code
doesn't match specified value.

The screenshot shows
touching
a TrueCrypt volume
before the backup. This is needed, because TrueCrypt
goes out of its way to preserve timestamps on its
container files and bvckup sees such files as unmodified.

It has already grown quite tall and it still needs
to take on few more configuration options, so
expanding it in place
as it's done for all other
sections doesn't work. The window gets too big
and simply doesn't fit on a desktop.

Instead, it pushes everything up and out of the
window and floods it with its own content. As I
add more config options to the section, it will
also expand the window down a bit if needed. All
this should help keep all configuration within
single window and also keep window's height in
check.

Took me a while to converge to this solution, but
it is a pretty decent way to structure it I think.

The tree selector widget is a piece of art
even though it doesn't look like one :) It
is basically a front of an asynchronous
parallel disk scanner that dynamically
rearranges its scanning queue to first
scan subdirectories that a user might be
accessing next.

In other words, if you expand a directory,
then the scanner will postpone scanning
anything else and focus instead on all its
subdirectories. So if you are making your
way through a massively populated disk, it
will try to pre-load what you are looking
at and scan the rest on a lower priority.

Also, since the widget starts showing
tree contents right away, it allows quickly
navigating to and excluding a desired item
without needing to wait for the
entire scan to complete. Again, quite a
bit of speed up with huge trees.

Advanced filter configuration. The pattern
matching is all brand new and much improved
over the disaster that it was in Bvckup 1.
It also adds an option of matching on item
attributes, both set and cleared.

One of the outstanding features is the archival
of deleted items and this is the supporting UI
for it.

Now deciding between a compact format of the
config widget or a verbose one. Compact format
has a single descriptive field that is re-used
to show the explanation of a currently selected
option. Verbose format has descriptions for
each option visible at all times - it is likely
easier on first time users, but it is also a
waste of screen space on all subsequent uses.

The new and much improved delta copier is in.
( the original version is
here )

—

Significantly improved performance through
multi-core, parallel hashing, asynchronous I/O and
more aggressive caching of the state data.

—

Strengthened the change detection algorithm
through addition of the whole file hash. When all
individual block hashes come back unchanged,
the algorithm now uses the file hash to confirm
that the file has indeed remained unchanged.

Combined with the original use of two separate
hashes per data block, this eliminates the need
for a precautionary full-file sync every N copies.

—

Improved handling of cancelled and aborted delta
copies by introducing support for partial
updates.

The algorithm now remembers how far along the file
it managed to proceed and stashes this information
for the next run.

When the copying is cancelled partway through,
the file is marked as out-of-sync, but the delta
state is updated to capture what has been already
done. This comes very handy and speeds things up
when updating very large files.

—

Improved handling of large file counts.

Version 1 maintained an in-memory index of all
delta copied files and, surprisingly, this didn't
scale well. It also kept all delta state information
in a single folder, which too led to some pain and
suffering when the file count grew large.

Feeling creative? Make your own version, save as
a PNG, drop it in a config directory and -
voilà - your systray looks just like you
want it to.

There are two sets of icons on the sprite sheet.
The first three columns is the XP style, which
is a bit puffy and matches well older Windows
styling. The last three columns is a flat version
that goes better with native W7 tray icon style.

Each icon comes in 3 sizes - 16x16, 20x20, 24x24.
This is needed to accommodate 3 common DPI levels,
see
this earlier entry for
details.

"Message pending" indication, whereby a message
can be either an error, a warning or a notice
and these are differentiated with the ! color.

I realize it's not color-blind friendly,
but the goal is to attract the attention and
make user bring the app window up to read the
message --(therefore)--> reflecting the message
type via an icon level is not exactly critical.

Custom pop-out button that stays nice, flat
and out of the way when idle, but
acts like a regular button otherwise. Not
exactly a UI innovation, but it was nice
to be able to use it in a real project :)

Took out the engine mode switch from the Preferences
and into a dialog of its own. Still need to understand
if it's worth mentioning that there's an engine per se
or just stick to more amorphous "application mode"...

Done with the split-view mode for the main window.
It's one of those things that everyone takes for
granted and that, in reality, maps to quite a bit
of work if done right.

For example, resizing window by the top edge should
resize the top pane and leave the bottom part alone.

Likewise, resizing by the bottom edge should
expand/compress the bottom pane, but not the
top one.

However, if we are compressing window by the top edge
and the top pane is at its minimal height, then we
should the bottom pane starting to compress.

All good so far, but, say, we change our mind and start
expanding the window back (without releasing the mouse
button).

We should see the bottom pane brought back to its original
size first, followed by the expansion of the top part...
even though we are, technically, dragging by the top edge.

This sort of UX detail is something that few people will
notice if it's done right, because it feels natural and
aligns well with expected behavior. But cut some corners -
and you bet everyone will notice.

Basically the idea for the log is to support
multiple detail levels, each capturing
progressively more context that its parent.

With all details collapsed, the log gives a
quick overview of when the job was running
and if it completed OK or with errors.

Expanding details allows for a drill-down -
to look at the details such as the file and
byte counts, the timing and performance data
and the details of any errors that have
occured.

I'm still going to shuffle things around
the window, e.g. push the -/+ buttons to the
right, tweak the styling too, but the general
hierarchical format will stay. It is a very
convenient way of organizing the log info.

Finished a piece of code that determines
effective timestamp resolution of a file
system.

—

Every file system keeps track of when a file
was created and when it was last modified.
These timestamps are the natural properties
of a file and they are readily available for
inspection in the Windows Explorer or any
file manager of your choice.

The less obvious aspect of timestamping is
that timestamps have resolution. For example,
FAT tracks file modification with a measly 2
second precision. In comparison, NTFS uses
the 100 ns (nano-second) precision.

—

It is very typical for a backup software to
rely on timestamps to determine if a file
has been modified and requires copying. But
if the source file sits on an NTFS volume
and the backup goes onto a FAT disk, then
comparing timestamp directly simply won't
work, because FAT timestamp will be
rounded-up to a 2-second mark.

In other words, the timestamp granularity needs
to be taken into an account when comparing the
timestamps. The question is how to determine
what it is exactly.

—

First of all, the granularity for creation
and modification times can be different.
FAT has them at 10 ms and 2 s respectively,
but NTFS has them both at 100 ns.

To complicate matters, there appears to be
NAS devices that look like NTFS boxes, but
with the granularity that is not 100 ns.

So what I did is added code to probe the file
system and determine effective resolution for
both timestamps. This involves dropping a
temporary file and then trying to set its
timestamps to this, that and 3rd and see what
they end up at. From that it's possible to
deduce the resolution.

In cases when such probing fails, the app
falls back to guessing resolution by the
file system name. It is also possible to
override the resolution values from the
config, just in case.

Done with the backup scheduler. Comes with a
couple of neat improvements over v1.

—

The Go and Stop buttons work almost exactly
as in v1. If the backup is running, click
Stop once to Pause it, click again to Cancel,
click third time to disable the job.
If the job is disabled, click once to enable
it, click again to manually start it. And so
on, you get the idea.

—

First improvement over v1 are the Go!
and Stop! commands that are activated by
holding Ctrl down when clicking on the
toolbar buttons.

Second improvement is that multiple
jobs can now be run in parallel. Generally,
this has a negative impact on the overall
performance, but there are certain cases
when this is desirable. One example would
be running a smaller backup in parallel
with a much larger and slower one.

There are three ways to do this:

With the Go! command.

With a per-backup setting that makes
the job always run right away, even if
there's another job is running.

With an app-level setting that does the
same, but for all jobs.

—

PS. I considered adding something more
elaborate, e.g. allowing assigning jobs to
"run groups" and then allowing only one
active job per group, but that seemed like
an overkill, so I dropped that.

Specifically, I want to get rid of the V1's
Summary panel,
trim the information contained in it a bit and fold it
directly into the main window items.

In conjunction with this I'm also considering having two
modes for displaying backup jobs on the list - "terse"
one-liners and "extended" multi-liners. What I still need
to figure out if this should be a per-item preference or
an app-wide setting and how one would switch between these
display modes.

Trying to understand how to indicate if
a selected path points at a removable storage.

The app tracks storage devices being plugged into
the computer and it adjusts backup paths if a known
device re-appears under a different drive name.

I can, of course, not bother with this at all and
simply spit out a notification when (and if) the
drive change is detected, but then I think it's an
important functionality and it'd better be mentioned
during the setup.

Unlike the v1 that always stored the backup
interval as a number of seconds, the v2 adds
a time unit specifier.

This simplifies quite a few things in the UI
and yet it keeps the configuration flexible
enough to support 1.5 hour backups by allowing
to specify a 90 minute interval.

—

On an unrelated note, killed few hours trying
to utilize native Windows controls for the
interval configuration. The amount of work
required to make them look reasonably
good is as depressing as it is remarkable.

For example, the editbox control doesn't
natively support vertically centering the
input text. It just does not. It is
meant to look like this -

Want "120" aligned to the label on the left?
Sure thing, that'd be 2 hours of plowing
through documentation only to realize that
it must be... drum-roll... a multiline
control that you will then need to manually
pad the right amount at the top to push the
line to the middle.

—

But that's peanuts compared to trying to
beat some sense into the dropdown box. Long
story, but making a box taller and keeping
the line centered requires taking over the
control drawing functions, which in presence
of Windows themes is a delightfully unpleasant
experience.

In any case, turning the top control line
into the bottom one -

costs exactly a day of work, while it really
shouldn't. Welcome to Windows.

Initially, it was meant for the config
window that hosts periodic backup settings
(as seen in a couple of posts below), but
having talked to others and slept on the
issue I think that the natural language
input (i.e. requiring to enter "1 week"
or "15 minutes") is not the best option
for this case.

It provided a good at-a-glance overview of the
backup configuration, split settings logically
and tucked away the minute details. It did have
one problem - it was tall. So tall, in fact,
that I got several reports that it didn't fit
on screen :-|

—

V2 adds several new options to the backup
config, so this doesn't make things any
easier. After looking at few options, including
paged, tabbed and two-column, screenful layouts,
I circled back to the v1 layout.

To compress it vertically I've done two things -
got rid of the group boxes (frames) and collapsed
radio button groups into a single line with the
current selection.

This gives us the middle screenshot
(pic) that shows
the config window in the "overview" state.

—

Clicking on a "Change ↓" button expands the
radio button group and allows changing the
setting without leaving the page. Latter
is pretty nice to have, because ... well ...
because it's simply faster and more convenient.

Expanding a section gives the screenshot on the
right
(pic).
To collapse the section, there's a little
button in the section header on the right hand
side. I'm not 100% convinced it's really needed
nor that it's the best solution, but that's how
it is at the moment.

—

I forgot the "how to handle deleted files"
section, but it will be in the beta.
Not to worry.

The Backup Engine section - as simple as it looks,
it is fronting a couple of months worth of work
and it's easily one of the most complex part of
the v2. This is what detaches the user interface
from the guts of the program and lets latter run
as a separate process.

—

The Maintenance section -
I figured that with the backup being an important
piece of software, very few users would be interested
in fully automatic updates. Furthermore, quite a few
people like to retain even tighter control over the
software and, for example, do not like it when the
software talks to the Internet whenever it wants.
Hence the new option -

Don't check, but remind to check for updates

It's basically a kitchen timer that goes "Ding!"
every week or two (configurable) and prompts you
to let the app check for a new version. We'll see
how it goes, but I think it's a sensible
compromise between the privacy and functionality.

—

The More button transforms the window into a
comprehensive list of all available options,
close to two dozens now and still counting.
The idea however is to keep the defaults
reasonable so that there won't be much need
to venture into this part of the app.

—

Lastly, note how the Apply/Close buttons change
when the settings are modified. I have no idea
why this is not a standard behavior, because it
really should be.

#2 - A system tray notification. This is less intrusive,
but perhaps it's too out of the way. Also, the
systray is a shared area so the program is always
competing with other apps for the same screen real
estate.

#3 - Something else that is right in your field of view
if you are looking at the app, but that doesn't get
in a way of whatever you are doing at the moment.

—

Cue in the message bar, now an integral part of the
Bvckup UI. It is used for displaying various status
notifications right after the launch, after switching
the engine to/from the system service mode, when
discovering software updates and few other things.

I have also tried the self-closing version of the bar
that dismisses itself on a timeout. Like so -
and so -
but having lived with this for a couple of days,
it now seems like a solution in a search of a
problem. So I shelved it for now. If there's any
a real need for this, it'd be very easy to add
later on.

The most unconventional part of the setup
is how it handles the configuration of the
installation parameters.

The configuration page (see Page 3) is very
bare and focuses on how the program is going
to be used instead of being a mixed
bag of the minute details of the setup.

The target location is made into a secondary
setting with its value controlled by the usage
preference. There's no field for the Start
Menu group name. There's no ever popular
"Run the program after the setup completes"
option either. Instead, exiting the setup
automatically runs the program (see Page 4),
unless the installer window is closed through
an (x) at the top.

This leads to a simpler and more intuitive
setup flow... or at least that's the idea :)

If you are logged in as a regular user and
have no rights to modify C:\Program Files,
there's no reason why you shouldn't still
be able to install a software, just for
yourself.

While other installers routinely ask if to
install the software for everyone or just
for one user, it's a superficial question.
It merely controls if the Start Menu group
is created under All Users or your personal
profile. The program files still go into a
system directory and this requires
administrative privileges.

Bvckup implements true system- and user-
deployment, which is akin to deploying the
program in /usr/bin or ~/bin in Unix terms.

In fact, on a multi-user system it is
perfectly possible to install one instance
of Bvckup for each user and they will all
happily co-exist with each other.

This is one of more difficult parts of the application,
even though on the surface it looks uncomplicated. If
we run the installer and it detects that the app is
already installed, it should just update the
installation. How hard could it be?

Let's see.

If the program is not running, then it's indeed a simple
matter of replacing the program file with a newer version.
The fun starts if the program is running.

On Windows, running a program locks its file, preventing
any changes from being made to it. This is not required
per se and there are operating systems that don't do this.
There are also custom
PE
loaders for Windows that can take
an executable and create a running process out of it without
any locking fuss. For example,
bo2k
could do that back in 1999. See its source code for
details if interested.

—

To complicate matters, there might be more than one
process running off the same executable. For example,
if several users are logged into a Windows box and
each runs a copy of a program.

Therefore, the first order of business for any installer is
to shut down all running instances of the program. A
well-behaved installer should also restart all these
instances after it's done with updating.

—

The shutting down part is not a big deal. There's more
than one way to do this and all of them are fairly
straight-forward.

It's the restarting that is contrived.

Not only program instances may be running under different
user accounts, they may also be running with custom
command-line arguments, as Administrators or system
services. Restarting this cornucopia of awesomeness
is not an envious task.

—

Here's how Bvckup works around this mess.

Every Bvckup executable has a corresponding named event in
the Global namespace. The event is created by
the first instance of the program when it's launched, and it
is opened by all other instances.

The name of the event includes the
file ID of the executable file.
This groups running instances by the file they are using,
because, after all, the installer is interested in a
specific executable and nothing else.

For example, the following is an event name used by the
Bvckup installation on my development box -

Global\Bvckup2.e65a96a8001f000000013cba.Update

—

With this in place, the installer starts off by signaling
the event.

In response, a running program makes a temporary copy
of its executable file and starts it in a special
"relauncher" mode. And then it exits.

The "relauncher" process opens the "update" event and
simply spins there, waiting for it to become unsignaled.

Meanwhile the installer waits for the program file to
become unlocked. Once all running instances replace
themselves with the relaunchers, the executable frees
up and the installer proceeds with the update. It then
clears the "update" event and exits.

This releases the relaunchers, they restart program
instances and - ta-da - the update is completed.

—

Techincally, instead of adding a special "relauncher"
mode to the program it's possible just to have a
small standalone relauncher program. It comes down to
a matter of taste. Some people don't like to spawn
from temporary executables, and I like to keep my
file count to a bare minimum. To each his own.

This is my first time not using pixel images to
decorate a website. Instead, all icons are saved
in the
SVG
format and converted to a webfont, which is then
used to style regular HTML characters into icons.

The advent of webapps like
IcoMoon and
Fontello
not only made the process of building @font-face kits
an absolute no-brainer, it also gave an easy access
to hundreds of existing vector icons, many of
which are free and of production quality.

When coupled with now virtually universal support for
CSS text-shadow, box-shadow and gradients, this almost
completely eliminates the need for raster form of
smaller decorative and nuancing elements.

Probably missed few things, but that's easy
to add later on. The biggest time sink was
trying to inject some visual hierarchy into
the list, but without going against overall
minimalist look and feel of the site.

Might still revise it after sleeping on it
for a couple of days :) Won't be the first
time.

Plodding through some grunt work with the
new website. An almost equal split between
working on a copy, javascript and styling.

As simple as the screencap looks, it came
with two hurdles that needed solving.

The header font, Alte Haas Grotesk, didn't
display correctly after it was
converted
to the @font-face kit. Specifically, the
lowercase 'o' looked like this:

The font's free, but I couldn't find a copy
of its license, so I started tracking down
font's author to ask a permission to edit
the outlines.

While waiting for a reply, I found another
workaround which involved generating the
webfont kit from OTFs instead of TTFs. Not a
big deal, but it still took over 3 hours to
resolve.

—

The second issue was with the slide-out,
fade-in animation. Sometimes the slide-out
didn't extend all the way down and sometimes
it overshot. This traced back to the fact
that slide-out height was computed before
the OpenSans was fully loaded and the script
fused basic sans-serif to compute the
height.

Again, not too complicated, but, again, cost
an hour to work through.

:-|

Little bit of this, little bit of that -
all unplanned and it all adds up.

It appears that using as many threads as there are CPU cores
and traversing the tree in a depth-first fashion (scan child
directories first and sibling directories second) yields the
best scanning times.

This is a
sprite
sheet of all raster images that are going into the
run-as-a-service demo.

Interestingly enough, the total size of individual
images saved as 32bit PNGs is 63,127 bytes, while
the size of this sprite sheet (without checkered
backdrop) is 63,829. After factoring in the size
of the code that is needed to parse the sprite sheet,
it really becomes a no brainer to keep every image
in its own file and embed them into .exe's resources
that way.

Added a "crash" handling and reporting facility
that includes several important changes compared to
the original version:

1. The program now differentiates between three types of
errors and handles them slightly differently.

First, there are unexpected operating
system errors, i.e. when a core Windows function
fails in a way that it shouldn't be failing.
Classic example (that everyone is aware of) is
TrackPopupMenu that is declared as returning
boolean, but actually doesn't shy away from
returning an integer under certain
circuimstances.

Secondly, there are inconsistencies in app's own
code, whereby one part assumes certain usage
that is not observed by another part. These are
so-called
"assert" failures. The original version has
three critical issues, all of this particular
kind.

Lastly, there are straight-forward crashes. This
is when a program is trying to operate with data
that is no longer there or corrupted. This is the
worst of them all as it typically indicates a major
screw up on programmer's part, but in some cases
it may also be an indication of failing hardware,
like a faulty memory stick.

—

2. The program now generates a
minidump
- a small-ish
file in a standard Windows format that captures
the context of a failure. Minidump files
dramatically simplify investigation of a
problem on the developer's end.

—

3. Finally, the minidumps can now be uploaded to the
support server right from the error notification
window with one click. The transfer is based on
WinInet API as this is supposed to improve
the chances of getting the report through in
proxied and otherwise "enterprised" environments.

For cases when the upload fails, the error dialog
also offers an option of opening a link in a
regular browser and reporting a problem that way.

64bit version is stored inside of the 32bit
executable, from where it is extracted and
launched to replace the 32bit version if
latter detects that it is being run on a
64bit system.

Pro:
a single executable to distribute
Con:
the exe comes out weighing twice as
much... or does it?

The size of a 32-bit executable can be easily
reduced with the help of tools called
executable packers. They work by turning
the exe file into a self-extracting archive
that also automatically launches the original
exe after the extraction. It may sound simple
but in reality it involves bundling a custom
PE loader with the packed executable (as
it might not be possible to extract the exe
into a temporary file before running it).

In any case, exe packing is routinely used by
developers who care about the size of their
products, with uTorrent being one of most
notable examples. Original Bvckup was also
compressed, owning its tiny size in part to
the magic of
UPX.

—

The problem lies with 64-bit executables.
For a variety of reasons there are presently
no exe packers that support them.

However if we store 64bit binary inside of
the 32bit executable and then pass latter
through a compressor, we end up with both
being compressed. Ta-da!

But there's more.

—

When an .exe is compressed, the compressor
leaves the primary application icon intact.
This is done so that the app would keep its
original appearance in Windows Explorer and
not sport a generic "program" icon.

Starting with Vista, the app is expected to
support a variety of icon sizes that quickly
add up. For example, bvckup's icon weighs
in at about 110KB. That's almost half as
big as the actual code, the code that does
quite a few meaningful things on top of just
looking great :)

So, if we do the 64-in-32bit packaging, we
end up compressing 64bit version in its
entirety, including its icon, and arriving
at a greater size reduction if we were to
compress the exes separately.

Test of a fading transition, part of a small library
that deals with supporting UI animations.

Fading animation dissolves a part of a window into
background, in preparation for transitioning the
window to another state, e.g. moving to the next
page in a wizard or resizing the window into a
more compact form.

As simple as it sounds, these sort of animation
is fairly complex to implement on top of the
Windows API.

Windows includes native support for window
transparency in a form of
layered windows,
but it comes with a list of limitations that
aren't exactly practical or easy to work,
except for several specific cases.

The simplest working solution involves taking
a screenshot of the window, creating an overlay
window and then using it as a canvas to render
the transition from an empty background to the
screenshot. All the while making sure that
Windows doesn't sneak in an unsolicited window
refresh that would lead to a flicker, which in
itself is a very "fun" thing to debug.

Every application icon that is exposed to the
operating system must exist in at least nine
sizes.

16, 32, 48, 256 are more or less expected.
These are used in "small", "icon", "tile"
and "extra-large " views in Windows Explorer.

20, 22 and 26 are possible sizes of an icon
used in the window caption. The exact size
depends on screen's DPI level.

And 40 (along with 32 and 48) is the size of
an icon used in Alt-Tab view on W7, again
depending on the screen's DPI.

If you don't provide an appropriately sized
icon, the OS will grab one that is slightly
bigger and shrink it down, ruining the pixel
polish.

Lastly, each icon ideally must be saved in
32, 8 and 4 bits per pixel formats, so to
be compatible... you know... with
EGA cards

It would've not that big of a deal if it
weren't for the size of a resulting .ico
file. For example, the above icon ends up
being ~ 120KB, whereas the actual .exe it
is attached to and that does quite a few
meaningful things sits at around 96KB.

Similarly, every bit of graphics in the app
must be done in at least three different sizes
- one for each common screen DPI level.

DPI stands for dots-per-inch and it's a metric
of a pixel density on the screen. The smaller
the screen, the more pixels are packed into a
square inch, assuming the resolution stays the
same. The difference in DPI between a large
LCD panel and a 12" notebook screen can easily
be a factor of two or more.

This means that if a program renders 9px text
on both screens, on one it will appear twice as
small as on the other one, and might command
a need for a good magnifying glass.

So naturally it's a good idea to make the OS
aware of the actual DPI of the monitor in use.
If it's too high, the OS can do few things to
make things visible with a naked eye again.
Like increase size of the default system font
and make windows generally larger.

XP defaults to 96 DPI and with some difficulty
it can be changed to 120 DPI. Vista and newer
versions are better. They come with four
predefined levels - 96, 120, 144 and 192 -
and they also support custom settings.

This brings us back to needing multiple versions
of every graphic in the app.

—

Vista is the first Windows version to
routinely use monitor-specific DPI values
out of the box. As such 96 is no longer
a dominant DPI level and the apps are now
expected to take the DPI into an account
when rendering their content.

The higher the DPI, the larger the graphics
and the fonts the app should use.
The penalty for not doing this is an automatic
stretching of the app window that inevitably
leads to a blurry appearance and a host of
other lovely issues.

In other words - not complying with OS
expectations is not really an option. Hence
the need for 3+ differently sized copies of
the same pixel art.

The UI sketch for run-as-a-service demo.

Trying to find a balance between using custom
styling and not getting to far away from the
native OS look and feel.
See also
8 other options leading up to this one.