Tested

Tested -

When scores were based solely on safety features and code hygiene, Windows was in the lead, with Linux trailing. Linux did so much better in terms of code complexity, however, that even though that's just 20% of the overall score, the total scores are now fairly close. You can browse the different evaluation categories below to determine which elements are more important to you.

We are including a desktop operating system here for comparison purposes, as both smart tvs were running Linux variants. Rather than comparing to off the shelf Linux, we are comparing to a version where a modest hardening effort ($500 paid to a consultant) has been made, as we would expect a tv vendor to spend at least the cost of one tv on hardening their firmware. While neither tv was up to the safety standards of the hardened Linux, Visio had a clear security and safety advantage over LG, which did surprisingly poorly.

Here we consider two versions of Microsoft Office for OS X. How does the newer release compare to the old?

Safety Features

Of the three operating systems reviewed, Windows 10 has the most consistent use of safety features, and is the only one to broadly deploy CFI. Linux is well behind MacOS and Windows when it comes to its use of ASLR. MacOS takes a hit because of its diminishing use of the heap protection flag, and because it lags behind Linux in the use of source fortification.

Directly comparing scores across different operating systems is tricky in this category, because each OS has different safety capabilities and liabilities. Some safety features are only available or applicable in one operating system, which is why some OSes are absent entirely from some sets. Safe SEH and RELRO, for example, are addressing security concerns specific to their operating systems, so they are only applicable for one OS each. Others, like CFI, are newer features that are available in some OSes and not others.

Use of DEP and ASLR was consistent across the board, but Edge far outstripped the other browsers in terms of CFI, with Opera in second place. Firefox's use of stack guards lagged a bit behind the others, and Chrome didn't have as many 64 bit binaries as expected.

Chrome is the only one of the browsers to use the heap protection flag, but it loses that advantage due to the slightly lower values for use of 64 bit binaries, DEP, and ASLR. Firefox does the best in terms of fortification, and Opera does the best for stack guard usage. Safari is the worst in both of these categories, getting it the worst safety feature score overall.

Both of the smart tvs were entirely 32 bit, but Visio did much better than LG in terms of safety features. Where Visio's scores were comparable to hardened Linux for all categories (aside from being 32 bit), LG had no ASLR, and almost no use of source fortification or stack guards.

Code Hygiene

Some functions are historically difficult to use without introducing vulnerabilities. We have broken these into categories, based on how much of a risk they pose. In many cases, newer, safer versions of these functions have been introduced to replace them - these are the "good" category of functions. Windows got the highest code hygiene score because they showed the most frequent use of good functions, and least frequent use of the very risky category. MacOS came in second, and Linux came in third. Instances of extremely risky function use were very rare.

Edge did much better than the other browsers in terms of use of good functions, while Opera did the best at minimizing the use of the riskier categories.

Firefox was the only browser to show significant use of good functions, but looses that advantage and then some by having the most "very risky" functions as well, by a significant margin. Chrome and Opera end up being nearly identical in their ratios, and Safari ends up being the best of the pack.

When looking at code hygiene, all three browsers performed similarly.

Neither tv used many good functions, and both used more extremely risky functions than we generally expect to see in commercial code. LG had the largest ratio of very risky functions.

In terms of code hygiene, Office 2016 saw an increased ratio of "good" functions, resulting in a slightly improved score.

Ratio: Good-Risky Functions

The circular bar charts to the left show how many good functions were present in comparison to the number of risky functions of varying degrees. If more of the circle is brightly colored, then that indicates that a greater part of all of the categorized functions were from the "good" category. Darker shades of grey indicate higher levels of risk.

Good

Risky

Very Risky

Extremely Risky

Code Complexity

Linux links to fewer libraries than either of its competitor OSes, and has a much smaller code base, so it scores much better than either in terms of complexity. Windows has fewer linked libraries than MacOS, but a larger overall code base by far, so it gets the worst complexity score.

The browsers with the largest code sizes tended to use the least libraries, and vice versa, which resulted in these scores all averaging out to being very close, with Opera having the advantage.

Safari has the smallest code base, but the most linked libraries by far. Firefox had the fewest linked libraries, and was the smallest of the other three, so it gets the best complexity score.

On the whole, all three browsers weigh in with similar code sizes.

We expect the desktop operating system to be much more complex than the one on a smart tv, and when it comes to code size, that turns out to be true. When it comes to linked libraries, though, LG uses almost as many libraries as a full deployment of Linux. Visio had the smallest code size and used the fewest libraries.

With the upgrade to Office 2016, one notable change is the increase in code size.

Code Size

# of Libraries

Security Aids

Some product features aren't currently captured in our scores (usually because we can't automatedly measure them), but could be relevant to decision-making. If a tool or operating system is open source, then it's possible to fix some of its security flaws yourself, and create a customized version for your personal or organizational use (eg, recompile with safer settings than the off the shelf version). If an application engages in sandboxing, then while parts of the software might be vulnerable, attacks on those parts could be contained within a designated "sandbox".

Is Open Source?

Has Sandboxing?

* A yellow check mark represents partial support. This usually means the software is using an open source library, but has custom additions that are not open source.

I Use Low-Scoring Software

What Should I Do?

If You're An Average User

Switch to a secure choice

See whether a better-scoring option has the features you need.

If You're A Technically Advanced User

Re-Compile the Software

If it's open source, recompile with better safety features. If it's not, contact the vendor and ask them to enable the missing safety features.