Quotes on the Internet

In 2012, Lukas Mathis took a quote out of the context of a blog post by Marco Arment and ran with it. The result was a though-provoking essay. Key quote:

This is a sentiment you often hear from people: casual users only need «entry-level» devices. Even casual users themselves perpetuate it: «Oh, I’m not doing much on my computer, so I always just go with the cheapest option.» And then they buy a horrid, underpowered netbook, find out that it has a tiny screen, is incredibly slow, the keyboard sucks, and they either never actually use it, or eventually come to the conclusion that they just hate computers.

In reality, it’s exactly backwards: proficient users can deal with a crappy computer, but casual users need as good a computer as possible.

Lukas fully develops the idea in his post, Crappy Computers. Go ahead and read it now if you haven't already. This blog will still be here when you come back.

Floating-point libraries

The idea expressed by Lukas Mathis applies identically in a much more specific setting: developing with floating-point computations. The developers who most need accurate floating-point libraries are those who least care about floating-point. These developers will themselves tell you that it is all the same to them. They do not know what an ULP (“unit in the last place”) is, so what difference is it to them if they get two of them as error where they could have had one or half of one?

In this, they are just as wrong as the casual computer users who pick horrid netbooks for themselves.

Floating-point-wise, programming environments are not born equal

All recent processors for desktop computers provide basic operations +, -, *, / and square root for IEEE 754 single- and double-precision floating-point numbers. Each operation has its assembly instruction, and since the assembly instruction is the fastest way to implement the operation, compilers have no opportunity to mess things up in a misguided attempt at optimizing for speed.

Who am I kidding? Of course compilers have plenty of opportunities to mess things up.

It may seem to a compiler that a compile-time computation is even faster than the assembly instruction provided by the processor, so that if the program computes x / 10.0, the compiler may compute 1 / 10.0 at compile-time and generate assembly code that multiplies x by this constant instead. This transformation causes the result to be less accurate in some rare cases.

Or a compiler may simplify source-code expressions as if floating-point operations were associative when they aren't. It may for instance optimize a carefully crafted floating-point expression such as a + b - a - b into 0.

Nevertheless, there has been much progress recently in standard compliance for compilers' implementations of floating-point. Overall, for programs that only use the basic operators, the situation has never been better.

The situation is not as bright-looking when it comes to mathematical libraries. These libraries provide conversion to and from decimal, and transcendental elementary functions implemented on top of the basic operations. They are typically part of the operating system. Implementations vary wildly in quality from an operating system to the next.

Expert developers know exactly what compromise between accuracy and speed they need, and they typically use their own functions instead of the operating system's. By way of illustration, a famous super-fast pretty-accurate implementation of the inverse square root function is used in Quake III and has been much analyzed.

The casual developer of floating-point programs, on the other hand, will certainly use the functions provided by the system. Some of eir expectations may be naive, or altogether impossible to reconcile with the constraints imposed by the IEEE 754 standard. Other expectations may be common sense, such as a sin() function that does not return-2.76.

For such a developer, the mathematical libraries should strive to be as accommodating and standard-compliant as possible, because ey needs it, regardless of what ey thinks.

An example

To illustrate, I have written a string-to-long conversion function. It could have been an entry in John Regehr's contest, but since the deadline is passed, I have allowed the function to expect only positive numbers and to fail miserably when the input is ill-formed.

I wrote my function to work for all well-formed inputs that fit in a long (but I only tested it for four values, so do not embed it in your space shuttle, please). Some of the reasons why I expect it to work are implicit: for one, powers of ten up to 10^22 are exactly representable as double-precision floating-point numbers. Also, I happen to know that on the system I use, the mathematical library is one of the best available.

I am not, in fact, a floating-point expert. I could be completely illiterate with respect to floating-point and have written the exact same function. In fact, this happened to StackOverflow user1257. (I am not saying that StackOverflow user1257 is illiterate with respect to floating-point, either. Ey wrote a function similar to mine, after all.)

User1257's function returned 122 when applied to the string "123" !!!

This was so troubling that user1257 suspected a compiler bug. The reality is more likely that on eir computer, the statement long pow10 = pow(10, 2); sets variable pow10 to 99. The function pow() only needs to be inaccurate by 1ULP for this result to come up because of C's truncation behavior (towards zero) when converting from floating-point to integer.

Conclusion

My str2long() function would fail just the same if it was run in user1257's compilation environment. I still think that my function is correct and that I should be able to expect results to the ULP from the math library's pow() function. A floating-point expert would never even encounter the issue at all. I might be able to diagnose it and to cope. But the floating-point beginner simply needs an environment in which long pow10 = pow(10, 2); sets pow10 to 100.

If you program, and if you use floating-point at all, beware of relying on the math library equivalent of a crappy netbook.

I used to think the same way about floating-point until I seriously tried to make a static analyzer for predicting anything a program containing floating-point computations can do (or should be able to do. The extent of the differences was one lesson learned. The sin(3.0) fiasco was one example, and there was another about decimal-to-floating-point conversions; they have been documented in this blog).

Looking back, most of the unpredictability I expected to trip on was rooted in my own superstition. There are rules, and they are generally well-followed. The rules are not particularly intuitive, but it's nothing that someone who has managed to learn how to write C programs cannot learn too.

I did not realize it, but I was actually writing about the “principle of least astonishment”. I think this is a good way to think about the issue. If a program contains exp(1), the developer would be least astonished to get a number that behaves like the mathematical constant e, but that's not possible because this constant cannot be represented as a floating-point number (unless one was willing to introduce some serious astonishment elsewhere). However, 100.0 is a floating-point number, and there is no less astonishing result than this one for pow(10, 2). For what I know of IEEE 754, this principle has obviously been a strong concern. But the committee also had to deal with real world constraints and could not require from future implementors feats that no-one knew how to achieve at the time.