notes.public

Science is hard to do right and not always the best tool in the toolbox

You’ve heard about how people lost in the desert tend to unintentionally walk in circles, right? Either one leg is slightly longer than the other, or their gait is slightly uneven, but for whatever reason, people have trouble walking in a straight line without a frame of reference.

A while back I read about a study to confirm this finding. It took a bunch of people and blindfolded them and had them try to walk straight. And of course they were constantly getting turned around. The conclusion was that it is impossible for people to walk in a straight line without a visual frame of reference.

But if you conducted a similar study to determine whether people could play the violin, and took a few dozen people off the street and gave them a violin to see if they could play it, you would probably conclude that it is impossible for people to play the violin.

Rather than normal scientific experiments, we have ways to determine whether and how well people can play an instrument. They are called auditions and recitals. More generally, if you want to find the limit of human capability, the appropriate structure is competitive, such as a tournament.

If you’re concerned about being able to find your way out of a desert, here are a few ideas. Obviously the simplest way is to use the sun and stars to navigate. If you happen to be blindfolded, perhaps use the heat of the sun on your skin, or the direction of the prevailing winds. Figure out in advance the diameter of the circle you tend to walk in, and adjust for it every 1,000 paces. I’m sure that someone who practiced walking in a straight line competitively would come up with even better tricks.

Similarly, if you want to figure out how to lose weight or get in shape, don’t ask a nutritionist or dietitian. Ask a successful bodybuilder or athlete. Of course they won’t know everything about health, and there will be some “bro science” mixed in, but at least their results show they’re doing something right. (Realistically, the answer is probably, “invest more time and effort than you are willing to invest.”)

If you want to understand people, emotions, beauty, etc., don’t ask science. Ask art. Of course art is less certain than physics, but it’s probably more certain than psychology. The successes in art are often remarkably self-evident.

If you want to learn the best study tricks, look at the most successful students, not studies comparing averages across all the students. Even if the study is right, the benefit will only be marginal, because the best students use a lot of different tricks in concert.

If you want to design a scientific study to show that people are incompetent and don’t know anything, that’s very easy to do. Negative reproductions are hard to do right, because it’s so easy to fail for the wrong reason.

I also don’t give scientists a pass for trying to prove the obvious, because “at least now we know for sure.” No, either we knew before with heaps of real-world experience, or we still don’t know despite what this one random study says.

Using something means you’re also testing it. For example, if one of our assumptions about the laws of physics with regard to the internal combustion engine were wrong, we’d have found out quickly because there are billions of ICEs being used all over the world. That isn’t to say that we’d find all the edge cases (which by definition don’t happen in normal operating conditions), but at least the basic ideas are solid.

Conversely, this implies that any scientific hypothesis that is difficult to test must also be difficult to use. If the higgs boson had any implications for happy meal toys, we could just make some and check whether they work. But since it is so expensive and difficult to even detect, it simply can’t have any practical applications for the foreseeable future.

That’s not to say the Large Hadron Collider is bad or a waste of time. But instead of testing the standard model, we’re really just testing ourselves.

In conclusion, science is hard to do right and not always the best tool in the toolbox. It is far from the only way of knowing things, unless you want to stretch the definition beyond reason.