Building Accurate Visual Diffs

AtSpotbot, we’re all about helping software teams ship faster. A great way to do this is by using visual diffs . Visual diffs show you the difference between how a page used to look and how it looks now. When they work well, they draw your eye to exactly what changed so engineers, designers, and PMs can all work together to make sure that the product is up to their standards.

The thing is, they have to work well. If they don’t, they’re just another distraction. Since distractions only hurt a teams cadence, we knew that to build a helpful tool, we had to make them work well.

Pixel Diffs Are Brittle

All that changed in that page is a banner at the top and the dates. But because the banner at the top pushed everything down 20px, it looks like everything changed.

Luckily there’s a better way.

Structural Diffing is Sturdy

By using the structure of the page instead of the raw pixels, we can generate a much more accurate diff.

Create a tree that represents the page. The tree looks like a DOM tree, but it includes information like the bounding rectangle of the node and all the calculated styles.

Diff the new tree with the old tree. This is unfortunately NP-hard , but we can fake it (details below).

Circle the nodes that have changed (if the user hasn’t previously indicated that they don’t care about those changes).

Simple and effective. We can now point out things that moved and figure out what got inserted where when the rest of the document stayed the same.

Heavybit fixed one of their pages at lower resolutions. This would have been a “sea of red” if it was a pixel diff.

The Gory Details of Creating a Structural Diff

The goal of a tree diff is to describe the minimal set of operations (add, remove, move, and modify) to transform tree A into tree B. There has been a lot of workdone on diffing trees, and it’s not a trivial problem. The crux of the difficulty is that a node somewhere in tree A may be anywhere in tree B, and the only way to verify that it’s not is to check all of them (that is waaaay oversimplified, read the papers for the details).

It’s possible to take some shortcuts to simplify the problem. One is to somehow uniquely identify the nodes. Another is to only identify moves among siblings in a tree. If a node has moved to a different parent, screw it, it’s a removal and an insert.

The React developers have published a nice post on how they take these shortcuts in React , where they enforce unique keys among siblings . React can get away with this because React is responsible for creating the tree in the first place, so it has knowledge of every node and can id them itself.

Where React has control over the input, Spotbot does not. When diffing arbitrary trees where you don’t have control over the identifiers, you need to come up with some way of uniquely identifying a node.

Our tree is a JSON structure that represents the page once it’s loaded. It contains not only DOM information but also styling.

At Spotbot, we use an heuristic inspired by BULD (bottom up, lazy down, described in part 5 of this paper ) to build an identifier that we add to these node objects.

First we compute the signatures for each node. The signature is a hash of the contents (if it’s a text node), the id, each of the classes, and the combination of its children’s hashes (if it’s an element node).

There are many possible signatures, so a set of the possible signatures for the node is created. Then each signature is matched against the signatures in the other tree to find the unique one. If a unique signature is found, the corresponding node B is matched with this node A and we use the signature as the node A’s id.

Now that we have unique identifiers, we just need to come up with that minimal set of operations that will transform tree A into tree B. At Spotbot we use the excellent JSONDiffPatch library alongside the node identifiers we built with the bottom up, lazy down technique. This basically does two things:

Identifies nodes that are the same, but have different attributes. We note the attributes that are different and mark the node as changed.

Identifies children that have been inserted, deleted, or moved. JSONDiffPatch does this by finding the longest common subsequence in the array of children. These nodes are also marked as changed.

And that’s it! Now we’ve got a new tree B with nodes marked with the changes that were made to old tree A to get to the newer state. We can then use that information to decide whether the change is relevant (it’s visible and hasn’t been ignored). All relevant changes get boxes drawn around them in the UI so the user can jump between changes and see exactly what changed.

Flaws in Structural Diffing

As with any engineering decision, there are tradeoffs. Structural diffing is not perfect and there’s still a lot of work for us to do at Spotbot before it works flawlessly.

Added DOM wrappers. Since we only identify moves amongst children, adding a wrapper div can cause us to think vast sections of a page were added or removed when in fact there was just a wrapper element added. This may be improved by using XYdif f instead of the simple JSONDiffPatch algorithm which mostly relies on LCS.

Browser inconsistencies. Sometimes IE draws a box to be 100px wide, and sometimes that box is 101px wide. ¯/_(ツ)_/¯

Cache busting urls. It’s pretty hard to tell if an image changed because the image source was changed or if a cache-busting parameter was added. Layering pixel diffing on top of structural diffing may solve this particular problem.

Those problems are all solvable, or at least manageable, and, at Spotbot, we’re getting better every day. Check out Spotbot today and catch issues at a glance!