These books have started to change my thinking about the nature of reality itself: I see now that randomness and uncertainty have an essential role to play. Interestingly, I shunned probability and statistics, the sciences of randomness and uncertainty, in college because I was steeped in Euclid, logic, and Buckminster Fuller’s “generalized principles” in Synergetics. I wanted to design destiny with deliberate application of knowledge … to worship at the altar of scientific determinism. Fortunately, Bucky taught me to “dare to be naïve” so I have been open to the new evidence about randomness. Now I suspect that Bucky and I were a little off about this subtle subject. It isn’t surprising, probability and statistics are among the newer branches of mathematics having developed mostly after the calculus was well established. They have not had enough time to pervade our collective consciousness.

Do you think the world is fundamentally deterministic or random? What influences have shaped your thinking and biases about the subjects of randomness, uncertainty, probability, and statistics? Do you think the increasing focus on the role of randomness and uncertainty in our lives is an important trend?

Randomness Rules Our Lives

Is Mlodinow’s thesis that randomness rules our lives really so convincing? Evidently so. Mlodinow finds dramatic evidence of randomness in our economic lives. He retells the poignant story of Sherry Lansing who led Paramount Pictures to huge successes in seven consecutive phenomenal years. Then after three years of bad results, she left the company. Did Paramount let her go too quickly? Evidently so because the pipline she left behind was full of new hits that restored Paramount’s revenue and market share. Shouldn’t seven years of success earn the right to forgive a few bad years? What if another great leader happened to have their three consecutive bad years at the beginning of their tenure? Do we replace them before their ship comes in? Mlodinow cites many other examples including the fact that “And to Think That I Saw It on Mulberry Street” was rejected by publishers some 27 times before Dr. Seuss’ career launched. Mlodinow also shows that student grades are often random and independent of their skill and knowledge.

Should we insist that our students, our schools, and our business leaders perform, perform, and perform with no “bad” years allowed? Do you believe that performance results are somewhat random? We invest a lot in exam and executive performance. Given the evidence, is that wise?

One part of Kahneman’s Nobel-prize winning work addressed the conjunction fallacy. Let A, B, and C be statements represented by a colored circle in the venn diagram to the right. The only case in which they can be simultaneously true is in the small area where all three colors overlap. So it is much less likely (less area) for three statements to be simultaneously true than for any one of them to be true. However, when someone weaves a story filled with a lot of concrete details, it seems more vivid and hence more believable than the statements considered separately: that’s the conjunction fallacy. Evidence of people falling for this fallacy has been documented widely even in medicine and the court room. We humans are easily duped by a good story!

It is surprising that the Nobel prize for the work showing how “blind” humans are to the elementary logic of the conjunction fallacy was only awarded one decade ago! Humanity has only just yesterday identified this basic weakness in our cognitive function! Add to the conjunction fallacy the many other fallacies and biases that Taleb, Lehrer, and Mlodinow show us to be subject to and one can see that Emanuel Lasker who was world chess champion for 27 years got it right: “In life we are all duffers”!

What is the significance of our weakness in understanding uncertainty? Do these weaknesses of the human mind subject us to the ravages of randomness? Are they a consequence of an inherent randomness in reality? Or do they simply lead to the appearance of randomness?

Human perception … is not a direct consequence of reality but rather an act of imagination. Perception requires imagination because the data people encounter in their lives are never complete and always equivocal.

Mlodinow illustrates the problem by explaining that the human visual system sends “the brain a shaky, badly pixelated picture with a hole in it” (due to the relative weakness of our vision outside the fovea and the blind spot). In addition to conjunction bias, the sharp shooter effect, the hot-hand fallacy, availability bias, confirmation bias, and more, it becomes evident that “When we look closely, we find that many of the assumptions of modern society are based … on shared illusions.” And his conclusion

It is important in our own lives to take the long view and understand that streaks and other patterns that don’t appear random can indeed happen by pure chance. It is also important, when assessing others, to recognize that among a large group of people it would be very odd if one of them didn’t experience a long streak of successes or failures.

What shared illusions do we hold? How often are our lives subject to pure chance events? How important is serendipity? Do you believe that a long series of failures or successes is just the result of luck? When is it luck and when is it skill? How can we tell the difference?

The problem of randomness is deeper still: even machine-enhanced human sensing and measurement are fundamentally random! In Walter Lewin’s excellent video introducing physics and measurement in MIT OCW’s Physics I course, he says “Any measurement that you make without any knowledge of the uncertainty is meaningless.” Understanding uncertainty is at the heart of scientific measurement. No physics experiment ever found an exact match between theory and the laws of nature: data points always appear at random! Then add in effects like Heisenberg’s uncertainty principle and we see that randomness and uncertainty are vital elements of experience: they are pervasive.

In view of the elementary role of uncertainty in our perceptual and physical experience, what can we say about reality? What is reality if experience is so imprecise, fuzzy, uncertain, and fallible?

In chapter III of D’Arcy Wentworth Thompson’s great magnum opus On Growth and Form many of the issues involved with the law of error are eloquently discussed. Mlodinow’s chapter 7 covers the same material at a more introductory level and with a wider range of vivid examples. But the two authors reach different conclusions. Thomspon says what I perceive to be society’s orthodoxy:

[We] are sometimes told `chance’ reigns, and `uncertainty’ is the rule, but such phrases as ‘mere chance’ or ‘at random’ have no meaning at all except with reference to the knowledge of the observer and a thing is a ‘pure matter of chance’ when it depends on laws which we do not know or are not considering.

Buckminster Fuller seems to agree with Thompson that randomness is an illusion:

What do you think? Is experience basically random? Uncertain? Deterministic? Both? Neither? How do you parse experience?

Exploring the relationships between reality and randomness

Reality it seems to me is concrete experience. Experience is “what happened” in contradistinction to our story or interpretation about what happened. But we are subject to forgetfulness, “invented” recollections, and our “creative” imagination may even get the concrete details wrong (how would we know?). Illusions and delusions fool us. Even when we are extraordinarily careful about observing data as in scientific measurement, each datum varies noticeably and randomly from the next. If raw experience is inundated with randomness and uncertainty, it is evidently essential. Mlodinow’s thesis that randomness rules our lives seems clearly justified!

Buckminster Fuller often pointed out that from time-to-time our minds can find patterns that are common in all experience. Then we say we have found a “truth” and in time it may even be recognized as a generalized principle. We are pretty confident about these scientific truths: they work reliably in experiment and in engineering practice. They make sense and they interaccommodate. They are represented by exquisite mathematics. They have been thoroughly vetted in a vigorous debate with alternative theories. From this perspective, randomness and uncertainty are seen as “challenges with messy data” that need to be overcome to reveal the order in nature. Indeed, if a datum differs from what a generalized principle would suggest, we examine our measurement to find the cause of the error. That is, we trust the science and question the errant data. Universe, it seems, rigorously follows the laws of nature … it is deterministic!

But science depends on the tools of statistics to determine the validity of its data and to develop incisive understanding of the interrelationships in the variables under study. Statistics is the tool through which scientists parse experience to find the order in the chaos. Probability and statistics are the technology used to overcome the inherent biases that blind us. They quantify uncertainty and show again that randomness is fundamental in the sciences. It is pivotal in the framework used to identify the mathematical relations that form the laws of nature.

Might the assumption of uncertainty simply be an analytical tool and not the essence of ontology (the nature of being and existence)? Or is uncertainty at the heart of the matter and “determinism” merely the rare statistical relationship with a 95% confidence interval?

Perhaps, both perspectives are illusions induced by the mathematical framework used to parse reality: probability theory vs. deductive reasoning. Reality could be simply the interrelationships in experience per se. The ontological distinction might simply be a story … an interpretation … philosophical sugar. Perhaps the uncertainty-determinism duality is just another example of the fundamental both-neither-ness inherent in Universe? [Credit to Tom Miller for introducing me to the concept of “both neither” at the Synergetics Collaborative’s third Summer Workshop at SUNY Oswego in July 2005.]

How do you interpret the “tension” between uncertainty and determinism? What ontology do you favor? Do you lean toward determinism or uncertainty or both-neither-ness or something else? Why?

Some OER (Open Educational Resources) for learning more about probability and statistics

Mlodinow’s excellent story-telling approach makes The Drunkard’s Walk an easy and entertaining way to learn the story of randomness. If you want to go into more depth by studying on-line, I recommend the introductory Probability and Statistics course at Carnegie Mellon’s OLI (Open Learning Initiative). The course is self-paced and includes exercises to test out and practice using the concepts to build intuition and facility in using the material. The OLI Statistics course supports solving problems on the computer using several statistical packages including my favorite, the R project for statistical computing. Like most statistics resources that I have seen, the course explains how to “do statistics” and sometimes fails to give justifications for the methods. The course lacks video lectures and requires free registration.

Video courses are better at relaying context for the concepts and for surveying a field. Building skills or striving for mastery of the subject requires more disciplined practice of the material by either working the exercises or building an application of personal interest (a method that is currently underutilized in formal education). I found several video courses available (including ones at UCLA, Stanford, Berkeley, Iowa State, and IIT Kharagpur). Of course, Khan Academy is great for supplementing the understanding of specific concepts). The video course Sets, Counting, and Probability at Harvard looks very promising (I hope to find time to check it out … eventually — for now I’m working to finish the OLI course).

Do you know of any other good OER materials on probability and statistics? Can anyone review one of the video courses?

20 Responses to “Are Randomness and Uncertainty fundamental and pervasive?”

R.W.Gray on 22 April 2011 at 8:02 am

Random and Randomness is the canvas upon which order is painted. Order arises from randomness while not eliminating the randomness. Randomness occurs because of our limited preception. We are in synch with Universe at a particular scale and at a particular rate of time. At different scales, things seem random because they occur more and more outside our zone of synch.

It’s easy to conclude that randomness rules… until you try to write a computer program that generates a random number. It’s so difficult, we tend to settle for tests of randomness that merely try to measure how unpredictable something is, rather than whether it’s truly random.

Which is another way of saying that randomness is a function of resolution (in the sense of the word as used in the printing industry, i.e., how close you have to get to an image before you can see the dots of which it’s made). It depends on where you’re standing, a concept that has been scientifically described as the mathematics of fractals.

I have a friend who began to notice a phenomenon he calls “clumping,” in which strikingly similar events occur seemingly without causal relationships. The defining feature of a “clump” is that it generates a frisson of weirdness. He may have put his finger on a central feature of the human psyche: that feeling of strangeness that signifies the perception of randomness.

Two words missing from your post are accuracy and precision. How do you account for so many instances of accuracy and precision in our universe of randomness is so pervasive?

I am a card player. Clearly, little is as random as the shuffle of the deck, yet there are many times where I know in an instant before the cards are dealt that I need to modify my bet up or down knowing the cards about to be dealt are favorable or not. We call it “gut instinct”, but perhaps it is a tie in to the universe at a level we cannot yet explain.

Rather than opine that randomness somehow underlies the science that we can demonstrate as accurate or precise, I would rather consider that what we currently see as random is much more orderly than we have the capability to observe or measure.

There is a fundamental theorem of mathematics that proves any logical system is either incomplete or inconsistent. We cannot expect the Cosmos to be logical. We cannot even expect our own brains to be logical.

Still, we can and do repeat processes, and those we can manage. Demming’s work with the Japanese automakers has shown that statistics can truly enable us to measure and improve quality. What you need is a system that is under statistical control. In such a system, strangely enough, the variations truly are random, but not exceptional. That is, they have common causes of randomness. People often use measures to predict outcomes on processes that are not systems and not under statistical control because they have special causes of variation. It is those special causes you are talking about here. In many cases they are not randomly sourced, just overlooked.

Demming goes into a number of common management mistakes, such as tinkering with processes. Because management controls the processes, management must understand how and when to measure them and any potential changes they make to them.

R.W., I love your poetic comment. The idea that order emerges from primordial randomness rings true in many ways.

When you say “Randomness occurs because of our limited preception.” Do you mean preception as a precept or a doctrine that is taught? That works for me. But I suspect you meant perception or becoming aware through the senses and the brain’s integration of said sense data. That works a little better for me. Is it what you meant?

I like the idea that we need to be in sync with what we are observing to see the order. Of course, how to get in sync and how to know that you are in sync are not always easy questions!

Mlodinow addresses the issue of the difficulty (if not impossibility) of generating randomness. I chose to focus on the intuitive ambiguity: there is some sense of randomness that is real and pervasive AND there is definitely order and pattern that is real and pervasive. I am curious about the relationship between these two seemingly antagonistic behaviors in Univese.

The generation problem gets at another distinction: the one between the two major interpretations of randomness. The subjective interpretation concerns the way the sample is generated. Charles Saunders Pierce championed the frequency interpretation which concerns the observed relative frequency of a sample over the long run. Our interpretations make all the difference!

Perhaps your friend’s notion of “clumping” is the pattern-matching “skill” of the dopamine system in our brains identifying “related” events even when there is no relationship? It is the same idea that caused such an uproar with the random shuffle function in the original ipod. HowStuffWorks explains that confusion by referencing the birthday paradox. Our pesky dopamine-mediated pattern recognition system is often problematical!

Accuracy and precision can only be measured as Walter Lewin indicates in the MIT video that I reference to within a measure of uncertainty. No human measurement is capable of 100% accuracy or precision. In fact, if a scientific journal sees data that is 100% precise, they immediately deduce fraud! Indeed anyone who claims anything with absolute certainty is probably fraudulent.

Humanity has achieved dramatically accurate and precise results: the computer that I’m using has a bunch of them built-in. But they all include random imperfections. So understanding uncertainty becomes important too.

As to “gut instincts”: Jonah Lehrer’s book covers the subject very well. It turns out that our instincts and intuitions are products of our emotional dopamine-mediated pattern recognition system. They are a powerful and effective faculty of human intelligence. However, instincts and intuitions are fallible. They are especially fallible when it comes to uncertainty.

You have tuned your “gut instincts” to be good at cards. However, I’m sure that if you carefully measure your experience you will find that it often fails: it is not a 100% accurate faculty. Indeed, if your cards instinct is “right” even 55% of the time, you will be doing great because most people’s sense of uncertainty is so bad that Monkeys and Rats could often perform better (read Taleb, Lehrer & Mlodinow for the details). Note: I am particularly bad in judging cards and other uncertain situations. I have only just begun a study of probability and statistics so I can learn to think more accurately about these subjects.

Finally, I do not mean to suggest that randomness supplants order. There is fabulous, intricate order in Universe. The order can be characterized by deterministic mathematical relationships (E=mc2 and others). But order is pervaded by randomness: in our measurements, in our perception, in our modeling, in our design and engineering.

I love the way Universe permeates deterministic order with randomness and uncertainty. They are not separate, but pervade and interfuse with each other! That’s what I find fascinating about randomness and uncertainty.

I am quite certain that if I add 1 plus 1, that I will yield a sum of 2. I am even willing to bet (being the card player that I am) that I can do this with complete precision 100% of the time.

You might be aware that I served in the US Marine Corps. A USMC Sniper can hit a 12 oz can of soda from 1 mile away. While this may not be precise, it is accurate in that they can do this repeatedly by “controlling” a huge number of seemingly random variables such as velocity, bullet weight, grains of powder, windage, elevation, et cetera.

Much of our universe, right down to the time it takes to write your blogs is finite and measurable. Time is a great indicator of so much in our universe right down to the concept of evolution. Evolution in its true sense is not random, but a predictable and measurable adaptation that occurs over a period of time.

Whether we are talking about the earth’s rotation around the sun, the change from night to day and vice versa, the nanosecond it takes for chemical processes to occur, we can observe and measure time in ways today that were not even fathomable 200 years ago.

What will our measurement capability be in another 50 years? I stand by my position that the universe is much more finite and measurable than it is random. Even though we may not have the means to observe and measure now, this does not mean that the order is not there.

Gödel’s incompleteness theorems: an excellent example of fundamental mathematical uncertainty. So it is not just our brains that we cannot expect to be logical, perhaps, we cannot even expect mathematics to be “certain”! I hope to revisit this idea in a future post.

Thanks for giving a pragmatic example of using statistics in citing Deming’s work. Randomness and uncertainty CAN and ARE used as tools for building precise and accurate business and engineering systems. More evidence of the pervasiveness of randomness.

The Universe is finite, I agree there. We have been historically able to measure with increasing accuracy, I agree there and I also see no reason for the trend to end. And you seem to agree that Universe has its random component. So the question seems to be how to judge which is more important, agreed?

My point was that determinism and randomness are “both-neither” phenomena. That is one can look at a situation from the deterministic frame and see the randomness impinging on it (errors in measurement, the butterfly effect, etc.). Alternatively, one can assume randomness and then find implicate order emerging. They are simply two frameworks or mathematical models to elucidate a given situation.

Judging if the deterministic or statistical viewpoint is more important or if new analytical frameworks might eclipse them both is something that, for the moment, I’m not concerned about.

To me it is simply important to note that the sciences of randomness and uncertainty provide a fresh perspective to uncover previously hidden relationships and enrich our understanding of experience. My invitation is to appreciate this relatively new form of thinking, even if it proves to be just another stepping stone on the long journey of Humanity’s quest for intellectual understanding in Universe.

Great inquiry, CJ. Thanks for raising the question. I really enjoyed the comments as well on this topic at the razor’s edge.

To your question on how we perceive the nature of experiential reality – I would answer dynamic. I think that quantum physics are beginning to expose that edge between the present moment – and the moment about to unfold. The model that speaks to me – based on my current knowledge, the latest research and hints from “on the edge” schools of thought – is that reality unfolds like a frothy wave on the beach. For any given water molecule, its location in the next “now” moment is really a probability field. General vectors of force, momentum, mass, etc. will have a big impact on that field – but can we say with certainty that they full determine that field?

I like the idea that “reality unfolds like a frothy wave”. Most of the time, reality seems less chaotic. That could just be our minds focusing on the parts that we can understand and ignoring the endemic “noise”.

I do not understand how the Noosphere project can make the leap from patterns in quantum random number generators to effects of global consciousness. If their data interpretation is correct (and I’m suspicious), then we have new quantum phenomena that need to be explained. I do think there is global consciousness, I just don’t like their way of trying to prove it.

Rupert Sheldrake’s morphic fields idea seems interesting. I have not read any of his books or articles.

I value speculative work like the Noosphere project and Sheldrake’s work. I think we need to test the limits of established wisdom with bold new ideas. I am skeptical of them all … including the scientifically established ideas (history shows that they change over time too). The scientific models tend to provide the best leverage and so deserve significant consideration.

I think it is important to look at situations with multiple models. If a proposed solution works even when tested against several diverse models, then our confidence in the proposal should strengthen. That is, the different models help us vet hypotheses, designs, solutions & interpretations. Some models are best for inspiration, imagination, thought experiments or just aesthetic value. Other models are known to be incorrect, but provide excellent approximations (my favorite example is the use of Euclidean Geometry in Carpentry despite the fact that we live on a sphere which has a non-Euclidean Geometry with no parallel lines, etc.). Science tends to emphasize those models that survive “a contest of alternatives to become something that then gets operationally accepted as truth” as Stephen Stearns put it. Science also prefers quantitative models. But conceptual models are important for developing intuition as to how the system works. Approximate models are essential for simulations to test out scenarios to evaluate design options to solve special-case problems.

I conclude that most models, if they elicidate part of the truth, have value. Our challenge is to understand the strengths and weaknesses of the different models and to creatively apply them to our local problems.

“Do you think the world is fundamentally deterministic or random?” Does it have to be one or the other? In my life, I’ve noticed tons of cause and effect. I’ve also noticed many many surprises. When we problem solve, we get more problems. Problems are unanswered questions effecting our happiness and our health. Remember, Synergy is behavior unpredicted. Synergy is surprise behaviors discovered out of problem solving. So surprises are just as much of our lives as the un-surprises, the predicted behavior of our only-and-always-true generalized principles. Probably determinism needs indeterminism. http://www.youtube.com/watch?v=dLrMVous0Ac

Dick,
You are so right about surprises. Just as our life time is a segment, surprises are inevitable. I’ve noticed people divide into two groups. Those who embrace surprise and those who avoid surprise. Roughly speaking there are ten avoiders to one embracer. Maybe it is 100 to one.
Don

The Fourth Quadrant: A Map of the Limits of Statistics by Nassim Nicholas Taleb is a good essay on a part of randomness that my post did not directly address. Taleb explains how rare events, so called “black swans”, are unpredictable by statistics. It is a good essay to fortify one’s intellectual toolkit to avoid becoming a “Turkey”: “A Turkey is fed for a 1000 days—every day confirms to its statistical department that the human race cares about its welfare ‘with increased statistical significance'”. As DavidHume pointed out past experience is no guarantee that the future will continue in the same way! Too often in life we delude ourselves that a theory that seems to work well will guarantee future success. It is nice to have someone like Taleb reminding us that many things continue to be unpredictable.

I don’t really know what the difference is. If I don’t know the cause then I can’t tell if it’s ‘random’ or ‘chosen’.

From a macro level people suffer because they choose the vantage that it’s random and as such they are, to whatever degree, victims.

This is a conventional perspective. But I offer that since people operate far better with certainty, and that existentially everything is already, always, certain, that people learn to ‘switch modes’ when experiencing a conventional-stressor (“I don’t know what she’ll say when I ask her out.” “I don’t know what the boss will say about my decision.” etc.).

Practically speaking this means that when you’re uncertain about something (stress, angst, frustration, whatever you like) perhaps you could ask yourself on a scale of 1 to 10 how stressful it is. Lets say it’s a five.

Now ask yourself how certain you are of that conventional measure of stress. In other words, how certain are you of whatever certainty you’re experiencing?

Well it’s always a 10. Everyone, always, is 100% certain about their uncertainty. (If you’re a programmer you can capitalize on the analogy of pointer and handle.)

Now I would ask you focus on thought that you now have certainty. Switch from thinking about the uncertainty to being certain. Now observe your level of suffering. When I re-focus on the certainty I find the suffering is gone.

Rinse and repeat.

I think ‘self-actualization’, practically speaking, the skill of being able to ‘remove oneself’ from participant (conventional-suffering) to observer (existential-freedom). This is why I say all transformational thinking axioms are really just a rebranding/repurposing of existentialism.

Commenting on this interesting and crucial question, I come down on the side of determinism. Of course, the problem is we don’t see the whole spectrum of what contributes to the events we witness so we never seem to quite know enough to get in it right all the time.

The question I ask, is what position can consciousness take that will know all the givens involved in a given situation. With a universe that is organized by fractal hierarchical fields, the obvious need is to base one’s perceptions on what the field tells us rather than the field objects, that is the objects in the field.

Fields are complete energetic distributional system wherein the field and the field objects are in balance with the energy spread so as to have stasis. Field objects on the other hand, cannot in an of themselves balance forces, hence everything but the tetrahedron (and its derivitives) are unstable. Rarely to we deal with tetrahedrons figuratively or factually, so we are usually in a more complex structural form. With out recognizing the field, the field objects experiences randomness, uncertainty and arbitary parameters making for the experience that chaos rules with only probabilities left to provide a semblance of order.