(The post is written for beginners. Send the link to your friends! Regular Less Wrong readers may want to jump to the Stanovich material.)

The last 40 years of cognitive science have taught us a great deal about how our brains produce errors in thinking and decision making, and about how we can overcome those errors. These methods can help us form more accurate beliefs and make better decisions.

Long before the first Concorde supersonic jet was completed, the British and French governments developing it realized it would lose money. But they continued to develop the jet when they should have cut their losses, because they felt they had "invested too much to quit"1 (sunk cost fallacy2).

John tested positive for an extremely rare but fatal disease, using a test that is accurate 80% of the time. John didn't have health insurance, and the only available treatment — which his doctor recommended — was very expensive. John agreed to the treatment, his retirement fund was drained to nothing, and during the treatment it was discovered that John did not have the rare disease after all. Later, a statistician explained to John that because the disease is so rare, the chance that he had had the disease even given the positive test was less than one in a million. But neither John's brain nor his doctor's brain had computed this correctly (base rate neglect).

Mary gave money to a charity to save lives in the developing world. Unfortunately, she gave to a charity that saves lives at a cost of $100,000 per life instead of one that saves lives at 1/10th that cost, because the less efficient charity used a vivid picture of a starving child on its advertising, and our brains respond more to single, identifiable victims than to large numbers of victims (identifiability effect3 and scope insensitivity4).

During the last four decades, cognitive scientists have discovered a long list of common thinking errors like these. These errors lead us to false beliefs and poor decisions.

How are these errors produced, and how can we overcome them? Vague advice like "be skeptical" and "think critically" may not help much. Luckily, cognitive scientists know a great deal about the mathematics of correct thinking, how thinking errors are produced, and how we can overcome these errors in order to live more fulfilling lives.

Rationality

First, what is rationality? It is not the same thing as intelligence, because even those with high intelligence fall prey to some thinking errors as often as everyone else.5 But then, what is rationality?

Cognitive scientists recognize two kinds of rationality:

Epistemic rationality is about forming true beliefs, about getting the map in your head to accurately reflect the territory of the world. We can measure epistemic rationality by comparing the rules of logic and probability theory to the way that a person actually updates their beliefs.

Instrumental rationality is about making decisions that are well-aimed at bringing about what you want. Due to habit and bias, many of our decisions don't actually align with our goals. We can measure instrumental rationality with a variety of techniques developed in economics, for example testing whether a person obeys the 'axioms of choice'.6

In short, rationality improves our choices concerning what to believe and what to do.

Ever since Aristotle spoke of humans as the "rational animal," we've had a picture of ourselves as rational beings that are hampered by shortcomings like anger and fear and confirmation bias.

Cognitive science says just the opposite. Cognitive science shows us that humans just are a collection of messy little modules like anger and fear and the modules that produce confirmation bias. We have a few modules for processing logic and probability and rational goal-pursuit, but they are slow and energy-expensive and rarely used.

Type 1 processes are fast, do not require conscious attention, do not need input from conscious processes, and can operate in parallel.

Type 2 processes are slow, require conscious effort, and generally only work one at a time.

Type 1 processes provide judgments quickly, but these judgments are often wrong, and can be overridden by corrective Type 2 processes.

Type 2 processes are computationally expensive, and thus humans are 'cognitive misers'. This means that we (1) default to Type 1 processes whenever possible, and (2) when we must use Type 2 processes, we use the least expensive kinds of Type 2 processes, those with a 'focal bias' — a disposition to reason from the simplest model available instead of considering all the relevant factors. Hence, we are subject to confirmation bias (our cognition is focused on what we already believe) and other biases.

So, cognitive miserliness can cause three types of thinking errors:

We default to Type 1 processes when Type 2 processes are needed.

We fail to override Type 1 processes with Type 2 processes.

Even when we override with Type 2 processes, we use Type 2 processes with focal bias.

But the problem gets worse. If someone is going to override Type 1 processes with Type 2 processes, then she also needs the right content available with which to do the overriding. For example, she may need to override a biased intuitive judgment with a correct application of probability theory, or a correct application of deductive logic. Such tools are called 'mindware'.9

Thus, thinking can also go wrong if there is a 'mindware gap' — that is, if an agent lacks crucial mindware like probability theory.

Finally, thinking can go wrong due to 'contaminated mindware' — mindware that exists but is wrong. For example, an agent may have the naive belief that they know their own minds quite well, which is false. Such mistaken mindware can lead to mistaken judgments.

Types of errors

Given this understanding, a taxonomy of thinking errors could begin like this:10

The circles on the left capture the three normal sources of thinking errors. The three rectangles to the right of 'Cognitive Miserliness' capture the three categories of error that can be caused by cognitive miserliness. The rounded rectangles to the right of 'Mindware Gap' and 'Corrupted Mindware' propose some examples of (1) mindware that, if missing, can cause a mindware gap, and (2) common contaminated mindware.

The process for solving a reasoning task, then, may look something like this:11

First, do I have mindware available to solve the reasoning problem before me with slow, deliberate, Type 2 processes? If not, my brain must use fast but inaccurate Type 1 processes to solve the problem. If I do have mindware available to solve this problem, do I notice the need to engage it? If not, my brain defaults to the cheaper Type 1 processes. If I do notice the need to engage Type 2 processes and have the necessary mindware, is sustained (as opposed to momentary) 'Type 2 override' required to solve the problem? If not, then I use that mindware to solve the problem. If sustained override is required to solve the reasoning problem and I don't have the cognitive capacity (e.g. working memory) needed to complete the override, then my brain will default back to Type 1 processes. Otherwise, I'll use my cognitive capacities to sustain Type 2 override well enough to complete the reasoning task with my Type 2 processes (mindware).

That may be something like how our brains determine how to solve a reasoning task.

If this is right, then we may talk of three different 'minds' at work in solving reasoning problems:

The autonomous mind, made of unconscious Type 1 processes. There are few individual differences in its operation.

The algorithmic mind, made of conscious Type 2 processes. There are significant individual differences in fluid intelligence in particular and cognitive capacity in general — that is, differences in perceptual speed, discrimination accuracy, working memory capacity, and the efficiency of the retrieval of information stored in long-term memory.12

The reflective mind, which shows individual differences in the disposition to use rationality mindware — the disposition to generate alternative hypotheses, to use fully disjunctive reasoning, to engage in actively open-minded thinking, etc.13

Rationality Skills

But it is not enough to understand how the human brain produces thinking errors. We also must find ways to meliorate the problem if we want to have more accurate beliefs and more efficiently achieve our goals. As Milkman et al. (2010) say:

...the time has come to move the study of biases in judgment and decision making beyond description and toward the development of improvement strategies.

Stanovich (2009) sums up our project:

To jointly achieve epistemic and instrumental rationality, a person must display judicious decision making, adequate behavioral regulation, wise goal prioritization, sufficient thoughtfulness, and proper evidence calibration. For example, epistemic rationality — beliefs that are properly matched to the world — requires probabilistic reasoning and the ability to calibrate theories to evidence. Instrumental rationality — maximizing goal fulfillment — requires adherence to all of the axioms of rational choice. People fail to fulfill the many different strictures of rational thought because they are cognitive misers, because they lack critical mindware, and because they have acquired contaminated mindware. These errors can be prevented by acquiring the mindware of rational thought and the thinking dispositions that prevent the overuse of the strategies of the cognitive miser.

This is the project of 'debiasing' ourselves14 with 'ameliorative psychology'.15

What we want is a Rationality Toolkit: a set of skills and techniques that can be used to overcome and correct the errors of our primate brains so we can form more accurate beliefs and make better decisions.

Our goal is not unlike Carl Sagan's 'Baloney Detection Kit', but the tools in our Rationality Toolkit will be more specific and better grounded in the cognitive science of rationality.

I mentioned some examples of debiasing interventions that have been tested by experimental psychologists in my post Is Rationality Teachable? I'll start with those, then add a few techniques for ameliorating the planning fallacy, and we've got the beginnings of our Rationality Toolkit:

A simple instruction to "think about alternatives" can promote resistance to overconfidence and confirmation bias. In one study, subjects asked to generate their own hypotheses are more responsive to their accuracy than subjects asked to choose from among pre-picked hypotheses.16 Another study required subjects to list reasons for and against each of the possible answers to each question on a quiz prior to choosing an answer and assessing the probability of its being correct. This process resulted in more accurate confidence judgments relative to a control group.17

Training in microeconomics can help subjects avoid the sunk cost fallacy.18

Because people avoid the base rate fallacy more often when they encounter problems phrased in terms of frequencies instead of probabilities,19 teaching people to translate probabilistic reasoning tasks into frequency formats improves their performance.20

Warning people about biases can decrease their prevalence. So far, this has been demonstrated to work with regard to framing effects,21 hindsight bias,22 and the outcome effect,23 though attempts to mitigate anchoring effects by warning people about them have produced weak results so far.24

Research on the planning fallacy suggests that taking an 'outside view' when predicting the time and resources required to complete a task will lead to better predictions. A specific instance of this strategy is 'reference class forecasting',25 in which planners project time and resource costs for a project by basing their projections on the outcomes of a distribution of comparable projects.

Unpacking the components involved in a large task or project helps people to see more clearly how much time and how many resources will be required to complete it, thereby partially meliorating the planning fallacy.26

One reason we fall prey to the planning fallacy is that we do not remain as focused on the task at hand throughout its execution as when we are planning its execution. The planning fallacy can be partially meliorated, then, not only by improving the planning but by improving the execution. For example, in one study27 students were taught to imagine themselves performing each of the steps needed to complete a project. Participants rehearsed these simulations each day. 41% of these students completed their tasks on time, compared to 14% in a control group.

But this is only the start. We need more rationality skills, and we need step-by-step instructions for how to teach them and how to implement them at the 5-second level.

3 People are more generous (say, in giving charity) toward a single identifiable victim than toward unidentifiable or statistical victims (Kogut & Ritov 2005a, 2010; Jenni & Loewenstein 1997; Small & Loewenstein 2003; Small et al. 2007; Slovic 2007), even though they say they prefer to give to a group of people (Kogut & Ritov 2005b).

Once upon a time, three groups of subjects were asked how much they would pay to save 2000 / 20000 / 200000 migrating birds from drowning in uncovered oil ponds. The groups respectively answered $80, $78, and $88 [Desvousges et al. 1992]. This is scope insensitivity or scope neglect: the number of birds saved — the scope of the altruistic action — had little effect on willingness to pay.

Cognitive scientists recognize two types of rationality: instrumental and epistemic... [We] could characterize instrumental rationality as the optimization of the individual’s goal fulfillment. Economists and cognitive scientists have reﬁned the notion of optimization of goal fulfillment into the technical notion of expected utility. The model of rational judgment used by decision scientists is one in which a person chooses options based on which option has the largest expected utility...

The other aspect of rationality studied by cognitive scientists is termed epistemic rationality. This aspect of rationality concerns how well beliefs map onto the actual structure of the world. Instrumental and epistemic rationality are related. The aspect of beliefs that enter into instrumental calculations (i.e., tacit calculations) are the probabilities of states of affairs in the world.

How can we measure an individual's divergence from expected utility maximization if we can't yet measure utility directly? One of the triumphs of decision science is the demonstration that agents whose behavior respects the so-called 'axioms of choice' will behave as if they are maximizing expected utility. It can be difficult to measure utility, but it is easier to measure whether one of the axioms of choice are being violated, and thus whether an agent is behaving instrumentally irrationally.

Kahneman (1986). Comments on the contingent valuation method. In Cummings, Brookshie, & Schulze (eds.), Valuing environmental goods: a state of the arts assessment of the contingent valuation method. Roweman and Allanheld.

The beginning of this post (the list of concrete, powerful, real/realistic, and avoidable cases of irrationality in action), is probably the best introduction to x-rationality I've read yet. I can easily imagine it hooking lots of potential readers that our previous attempts at introduction (our home page, the "welcome to LW" posts, etc) wouldn't.

In fact, I'd nominate some version of that text as our new home page text, perhaps just changing out the last couple sentences to something that encompasses more of LW in general (rather than cogsci specifically). I mean this as a serious actionable suggestion.

For the sake of constructive feedback though, I thought that much of the rest of the article was probably too intense (as measured in density of unfamiliar terms and detailed concepts) for newcomers. It sort of changes from "Introduction for rationality beginners" to "Brief but somewhat technical summary for experienced LWers". So it could be more effective if targeted more narrowly.

I think the intensity of the second half of this article may be compensated by the sheer amount of footnotes and references. I mean, that much bibliographic work amounts to a quite compelling argument from authority¹. Someone who dozes out and scrolls fast will notice this, and may think that so much work is worth the effort to try and read.

Meta: I would recommend distinguishing between citation-notes and content-notes. Scrolling a long way down to find a citation is annoying and distracting, but so is the feeling that I might be missing some content if I don't scroll down to look.

That would be useful. I guess the way I've seen this done sometimes is to use symbols like * † ‡ for content footnotes and numbers for citation endnotes? But that usually works on paper, where pages are short and you can see the content of the footnote at a glance. I'm not sure I've seen a solution for this that works on the web. It'd be nice to have an integrated Less Wrong footnote system so that we could test different ways of displaying the content. Maybe a hover-over-the-footnote-to-read-its-contents feature?

Until that sort of feature is implemented, what about footnote links to the content while having text (no link) to the references? Also helpful would be a "return to where this number is in the text" function. I anticipate this solution taking less time while being less robust.

Here's an example. The body text footnote numbers link to the bottom, and a return arrow links you back to the citation. Major problem on the linked website is that the page seems to have to reload. I don't know of any way to make citations such as these without the process being time-intensive unless you write your own citation manager or contact the linked-to blogger.

I've been very pleased with it on gwern.net; might be a little tricky on LW because it relies on the footnotes all having a particular name which the Javascript can then blindly load a related footnote in the popup, or whatever, and LWers seem to use various tools to generate footnoted-HTML (when they do at all).

You could write footnote 1, where the number 1 is a link pointing only to this comment. Hovering over the 1 shows some text. I couldn't seem to cancel the link formatting, so that might not be too useful unless you can somehow arrange that the footnotes are the first comment in your own thread.

I played in the sandbox and noticed that some things work differently there than here.

Unfortunately, if I remember correctly, there are gaps in browser support for
it. Also IIRC, using <a> works in more browsers, but the text will show up
styled like a link unless some CSS tweaking is done.

That's just a title tooltip isn't it? You can set those in Markdown easily enough (eg. [display](http://hyperlink "hover text")), and you're not allowed any sort of markup inside the tooltip, and they have severe length limitations too. So it'd be a major compromise. (I have, painfully, added them to the frontpage of gwern.net, but no one has ever commented on them or given any sign that they are useful, so I've never bothered with putting them elsewhere.)

Unpacking the components involved in a large task or project helps people to see more clearly how much time and how many resources will be required to complete it, thereby partially meliorating the planning fallacy.

But experiment has shown that the more detailed subjects' visualization, the more optimistic (and less accurate) they become. (In saying this, EY cites the work of Buehler, 2002. [1])

Is there something from your citation (#26) that overrides the conclusions of Buehler? [2] In fact, #5 was the conclusion proposed in "Planning Fallacy," which I thought was made specifically because examining all the details was so unreliable. In other words, #5 seems to say: forget about all the details; just find similar projects that actually happened and base your timeline on them.

I've been reading OB and LW for years. I'm sure I've read at least 3/4 of what was posted on OB and 1/2 of highly voted posts on LW. I wouldn't call myself a beginner, and I still really appreciated this post.

Every once in awhile it's nice to be able to step back and be reminded what it's all about, so thanks for writing this!

I point people who are new to this to the Sequences but that always feels like I'm dumping too much on them. Hopefully, this post will make a better introduction to rationality and why we need it.

At first glance I thought this would be an awesome post to introduce normal people to rationality. However it quickly becomes theoretical and general, ending pretty much with "to make actual use of all this you need to invest a lot work."

So... why isn't there some kind of short article along the lines of "xyz is a cognitive bias which does this and that, here's an easy way to overcome said bias, this is your investment cost, and these are your expected returns" or something? Could be as short as half a page, maybe with a few links to short posts covering other biases, and most importantly without any math. You know, something that you could link a manager or CEO to while saying "this might interest you, it allows you to increase the quality of your economic and otherwise decisions."

In a magazine article or textbook on cognitive science, the author might describe the marvelous mechanisms we have for recognizing faces and refer to this as "a remarkable aspect of human intelligence." Likewise, a book on popular science might describe how we have mechanisms for parsing syntax when we process language and also refer to this as "a fascinating product of the evolution of the human intellect." Finally, a textbook on evolutionary psychology might describe the remarkably intelligent mechanisms of kin recognition that operate in many animals, including humans. Such processes-face recognition, syntactic processing, detection of gaze direction, kin recognition -are all parts of the machinery of the brain. They are also sometimes described as being part of human intelligence. Yet none of these processes are ever tapped on intelligence tests. What is going on here? Is there not a contradiction? In fact, there is not a contradiction at all if we understand that intelligence tests assess only those aspects of cognitive functioning on which people tend to show large differences. What this means is that intelligence tests will not routinely assess all aspects of cognitive functioning. There are many kinds of Type i processing that are important for us as a species, but on which there tend not to be large differences between people in the efficiency of functioning. Face recognition, syntactic processing, gaze direction detection, and kin recognition provide four examples of such domains.' This is why such processes are not assessed on intelligence tests.

As stated, still hearsay. I understand that there are probably references somewhere, but it's still unclear even what standards of performance are considered ("syntactic processing"?). And of course, the Type I processes relevant to this post are not like those listed, which are way too specialized to serve the role of the general default decision-making.

But I assume you're asking me to go one step deeper. Stanovich has a more detailed discussion of this in his book Rationality and the Reflective Mind. In particular, footnote 6 on page 37 cites the following sources on individual differences (and the general lack thereof) in the autonomous mind:

I recently found an interesting study that bears on the doctor example. Christensen and Bushyhead (1981) find that when asked to make clinical judgments, doctors usually take base rates into account quite accurately, even though when they are asked to explicitly do statistical problems involving base rates they usually get them wrong.

I've seen some rather detailed discussion of the specific case of the enlarged human cranium being a rather tight squeeze through the pelvis, but I don't recall any more general discussion of selective pressures acting against intelligence.

Interesting post, but I think there is a typo : « Type 1 processes are computationally expensive » Shouldn't it be type 2 ?

Also, for the Concorde story, what I always heard (being a french citizen) is that « Yes, we know, Concorde was losing money, but it is great for the image, it's just a form of advertising. It gives a good image of France (and UK) aerospatial industry, and therefore makes it easier for Airbus to sell normal planes, or for companies like Air France to sell tickets on normal planes. » Now, how much of it is about a posterior rationalization of why they felled to the sunk cost fallacy (because they couldn't admit to public opinion they did it), and how they really decided to continue the program for that effect, I've no idea...

I grew up in Long Island 20 miles from JFK airport. We could see the Concorde once in a while at JFK airport and if we were very lucky we would see it landing or taking off. The amount of mindspace in the world occupied by that beautiful plane was gigantic compared to that occupied by most other planes. Whether the Concorde was still a net deficit to the UK and France would require, I think, a calculation similar to figuring the deficit or surplus to the U.S. of putting people on the moon.

You might be right - as I never saw one - but the project didn't start with a plan to built a spectacular flying sculpture. So they fell first to the planning fallacy (which may not be so much a psychological cognitive bias but the very structure of possible outcomes of everything - the top of the frequency distribution is to the right of the "arrival" time), then to sunk costs which later were half acknowledged, thus making them highly suspicious of trying to resolve a cognitive dissonance (rationalization).

One has to take into account the original prediction to make a probabilistic interpretation...

Perhaps what people have in mind when they say that are betting odds. If you bet money on an unlikely event then the odds are quoted with the high number (your reward if the event occurs) first, which seems sensible from an advertising perspective.

Biased response: as a native American English speaker I can assure you that "a million to one" is idiomatically correct.

I suspect that the more complete non-idiomatic version would be "a million to one against" and that the "against" is implicit because the idiom is highly established as expressing a very low probability.

Epistemic rationality is about forming true beliefs, about getting the map in your head to accurately reflect the world out there.

Since the map describes itself as well, not just the parts of the world other than the map, and being able to reason about the relationship of the map and the world is crucial in the context of epistemic rationality, I object to including the "out there" part in the quoted sentence. The map in your head should accurately reflect the world, not just the part of the world that's "out there".

But it can be taken as meaning to exclude the mind. I'm clearly not arguing with Luke's intended meaning, so the intended meaning is irrelevant to this issue, only possible interpretations of the text as written.

Ahh, don't say "understanding" when you mean "containing a simulation"!

It's true that a computer capable of storing n bits can't contain within it a complete description of an arbitrary n-bit computer. But that's not fundamentally different to being unable to store of a description of the 3^^^3 ×n-bit world out there (the territory will generally always be bigger than the map); and of course you don't have to have a miniature bit-for-bit copy of the territory in your head to have a useful understanding of it, and the same goes for self-knowledge.

(Of course, regardless of all that, we have quines anyway, but they've already been mentioned.)

The naive concept of understanding includes everything we've already learned from cognitive psychology, and other sciences of the brain. Knowing, for example, that the brain runs on neurons with certain activation functions is useful even if you don't know the specific activation states of all the neurons in your brain, as is a high-level algorithmic description of how our thought processes work.

This counts as part of the map that reflects the world "inside our heads", and it is certainly worth refining.

In the context of a computer program or AI such "understanding" would include the AI inspecting its own hardware and its own source code, whether by reading it from the disk or esoteric quining tricks. An intelligent AI could make useful inferences from the content of the code itself -- without having to actually run it, which is what would constitute "simulation" and run into all the paradoxes with having not enough memory to contain a running version of itself.

"Understanding" is then usually partial, but still very useful. "Simulating" is precise and essentially complete, but usually computationally intractable (and occasionally impossible) so we rarely try to do that. You can't get around logical certainty, but that just means you'll sometimes have to live with incomplete knowledge, and it's not as if we weren't resigned to that anyway.

The "map" is NEVER comlete. So our map of the map is an incomplete map of the map. In engineering terms, the remarkable feature of the human mind's gigantically oversimplified map of the world both in it and around it is that it is as effective as it is in changing the world.

On the other hand, since we are not exposed to anything with a better map than ours, it is difficult to know what we are missing. Must be a cognitive bias or three in there as well.

More accurately, the map should worry about mapping its future states, to plan the ways of setting them up to reflect the world, and have them mapped when they arrive, so that knowledge of them can be used to update the map (of the world further in the future, including the map, and more generally of relevant abstract facts).

(More trivially, there are quines, programs that know full own source code, as you likely know.)

(More trivially, there are quines, programs that know full own source code, as you likely know.)

I don't think being able to quine yourself really has anything to do with fully understanding yourself. I could get a complete printout of the exact state of every neuron in my brain, that wouldn't give me full understanding of myself. To do something useful with the data, I'd need to perform an analysis of it at a higher level of abstraction. A quine provides the raw source code that can be analyzed, but it does no analysis by itself.

"The effects of untried mutations on fifteen million interacting shapers could rarely be predicted in advance; in most cases, the only reliable method would have been to perform every computation that the altered seed itself would have performed... which was no different from going ahead and growing the seed, creating the mind, predicting nothing." (Greg Egan).

In any case, if we are talking about the brains/minds of baseline unmodified humans, as we should be in an introductory article aimed at folks outside the LW community, then XiXiDu's point is definitely valid. Ordinary humans can't quine themselves, even if a quine could be construed as "understanding".

His point is not valid, because it doesn't distinguish the difficulty of self-understanding from that of understanding the world-out-there, as nshepperd points out. (There was also this unrelated issue where he didn't seem to understand what quines can do.) A human self-model would talk about abstract beliefs first, not necessarily connecting them to the state of the brain in any way.

His point is not valid, because it doesn't distinguish the difficulty of self-understanding from that of understanding the world-out-there, as nshepperd points out. (There was also this unrelated issue where he didn't seem to understand what quines can do.)

I don't? Can you elaborate on what exactly I don't understand. Also, "self" is a really vague term.

More trivially, there are quines, programs that know full own source code, as you likely know.

Something has to interpret the source code (e.g. "print"). The map never equals the territory completely. At some point you'll get stuck, any self-replication is based on outside factors, e.g. the low entropy at the beginning of the universe.

I probably shouldn't talk about concepts of which I know almost nothing, especially if I don't even know the agreed upon terminology to refer to it. The reason for why I replied to your initial comment about the "outside world" was that I felt reasonable sure that the concept that is called a quine does not enable any agent to feature a map that equals the territory, not even in principle. And that's what made me believe that there is an "outside world" to any agent that is embedded in a larger world.

As far as I know a Quine can be seen as an artifact of a given language rather than a complete and consistent self-reference. Every Quine is missing some of its own definition, e.g. "when preceded by" or "print" need external interpreters to work as intended.

You wrote that you can pack a whole Linux distribution "in there". I don't see how that gets around the problem though, maybe you can elaborate on it. Even if the definitions of all functions were included in the definition of the quine, only the mechanical computation, the features of the actual data processing done at the hardware level "enable" the Linux kernel. You could in theory extent your quine until you have a self-replicating Turing machine, but in the end you will either have to resolve to mathematical Platonism or run into problems like the low-entropy beginning of the universe.

For example, once your map of the territory became an equal copy of the territory it would miss the fact that the territory is made up of itself and a perfect copy. And once you would incorporate that fact into your map then your map would be missing the fact that the difference between itself and the territory is the knowledge that there is a copy of it. I don't see how you can sidestep this problem, even if you accept mathematical Platonism.

Unless there is a way to sidestep that problem there is always an "outside world" (I think that for all practical purposes this is true anyway).

Let's say there are two agents, A and B, that try to defeat each other. If A was going to model B to predict its next move A(B), and vice versa B(A), then each of them would have to update on the fact that they are simulating each other, A(B(A)) and B(A(B)). That they will have to simulate themselves as well, A(A(B(A)), B(A)), does only highlight the ultimate problem, they are going to remain ignorant of some facts in the territory ("outside world"). Among other reasons, the observer effect does not permit them to attain certainty about the territory (including themselves). If nothing else, logical uncertainty will remain.

But as you probably know, I still lack the math and general background knowledge to tell if what I am saying makes any sense, I might simply be confused. But I have a strong intuition that what I am saying might not be completely wrong.

That is my own pet theory of "god." That the complexity of the operation of the universe requires something at least as complex as the universe to understand it, and the only real candidate when it comes down to it is the universe itself. Either in some sense the universe comprehends itself, or it doesn't. (Either the universe is a conscious universe, or it is a p-zombie universe.)

I suspect we are more than a few steps away from making a map of this part of the territory that is helpful beyond providing good beer conversations and maybe some poetry.

Every program runs on some kind of machine, be it an intel processor, an abstract model of a programming language or the laws of the universe. A program can know its own source code in terms it can execute, i.e. commands that are understood by the interpreter.

But I am not sure what point you are trying to make exactly in the above comment.

But I am not sure what point you are trying to make exactly in the above comment.

Vladimir Nesov was criticizing lukeprog's phrase "world out there" claiming that the "map in your head should accurately reflect the world, not just the part of the world that's "out there"". I agree, but if you are accurate then you have to admit that it isn't completely possible to do so.

Note that having type 1 processes is not an error. AIs will also need to have type 1 processes. Being perfectly rational is never optimal if time and resources are an issue. Being perfectly rational would make sense only given infinite time and resources and no competition.

I heard you bring this up in person a few times last weekend too. I wanted to follow up with you because I think I'm starting to understand the reason for your disagreements with others on this matter. It is not the case that you're fighting the good fight and we are all just off in crazy land on this issue. I think instead, you're conflating "rationality" with "deliberation".

Naive "perfect rationality" -- being infinitely deliberate -- is of course a mistake. But that's not what Yudkowsky, Omohundro and other careful thinkers are advocating when they discuss wanting to build an AI that is rational. They mean things like having a deliberate rational process that won't knowably violate the axioms of choice or other basic sanity checks that our kludgey, broken minds don't even try to do on their own.

Also, it's possible to design algorithms that are ideally rational within time and resource constraints (ie, AIXItl, Godel Machines). There isn't a false dichotomy between "quick and dirty" heuristic kludges and infinite rationality. You can use "quick and clean" methods which converge towards rationality as they compute, rather than "type 1 processes" which are unrelated to rational deliberation.

Basically, I completely disagree with your first two sentences and would agree with your second two if you replaced "perfectly rational" with words closer to what you actually mean, like "infinitely deliberate".

Is the reflective, algorithmic, autonomous hierarchy specifically for "reasoning problems" as in tools for solving non-personal puzzle questions? If yes it seems dangerous to draw too many conclusions from that for every day rationality. If not, what's the evidence that there are "few continuous individual differences" concerning the autonomous mind? For example, people seem to differ a lot in how they are inclined to respond when pressured to something, some seem to need conscious effort to be able to say no, some seem to need conscious effort to avoid responding with visible anger.

The word "Bias" is often associated with the word "prejudice" which has become loaded with rather negative associations. (People don't like others to think of them as "prejudiced") Especially as I am not a native english speaker until a week ago (I read LW since a month) I didn't make a distinction between bias and prejudice as in my language the 2 words translate more or less the same. Maybe the process of "debiasing" should include to associate the word "bias" with "cognitive bias : a pattern of poor judgment" which every human brain has and there is nothing to be ashamed of.

Bias is a systematic error in estimation. In the case of cognitive biases (as opposed to, for instance, statistically biased samples) we're talking about cases where people reliably make certain errors in estimation or prediction, for instance in estimating how long a project will take or whether an investment of effort or money is worthwhile.

This is a good introductory "big picture" post that describes motivation for developing our craft, some mechanisms that underlie its development, and glimpses into possible future directions. It also locates its topic in the literature.

(Not sure why it's being so weakly upvoted. One reason could be that it poses itself as presenting no new material, and so people skip it, and skipping it, refrain from voting.)

Nice article! I was wondering though whether there were any theories on why our brain works the way it does. And did we make the same mistakes, say 1000 years ago? I am new here, so I don't know what the main thoughts on this are, but I did read HPMOR which seems to suggest that this was always the case. However, it seems to me that people are becoming less and less critical, perhaps because they are too busy or because the level of education has steadily decreased over the last decennia. How could we actually prove that these fallacies aren't caused by some external factors?

What would you expect to see if people have become steadily less "critical" over the last thousand years?
What would you expect to see if people have become steadily more "critical" over the last thousand years?
What would you expect to see if people have remained equally "critical" over the last thousand years?
What would you expect to see if people's "critical"ness has varied in non-steady ways over the last thousand years?

I don't believe that people have become steadily less or more critical, but it seems plausible that this has varied in non-steady ways, increasing or decreasing depending on the circumstances. I would expect that in this case the degree to which people make common thinking errors also varies. In fact, I suspect that the 2 4 6 test yields different results depending on the subjects' education, regardless of whether they know of positive bias.

I see how that might have been confusing. The 1000 years ago was simply an example to question whether people have always made the same thinking errors. The criticalness is based only on recent history, mostly on the people around me and is an attempt to argue in favor of possible external factors.

That would be a part of the whole Nature vs. Nurture debate, wouldn't it? I think it would be very hard to prove that any given thinking error is biologically hardwired (as opposed to culturally); and even if a bias is biologically hard-wired, an opposing cultural bias might be able to counter that.

Many people are largely exposed to only a single culture; widespread, pervasive errors in that culture would, I expect, be indistinguishable from hardwired biases.

I've always wanted to know how it feels for students/"experts" of cognitive science how they feel when they realize their limits w r t perceptual speed, discrimination accuracy, working memory capacity etc.

As a student of the field in the late 80s, the most pervasive effect was constantly being forced to realize that "because X is actually Y" is not actually an answer to "why does X seem Y to me?" That is, not just that it's sometimes false, but that whether it's true or not has nothing whatsoever to do with the question I asked.

It's a nice introduction to rationality that someone could present to their friends/family, though I do still think that someone who has done no prior reading on it would find it a bit daunting. Chances are, before introducing it to someone, a person might want to make them a little more familiar with the terms used.

I wish i had this back when I was teaching gen-ed science courses in college. I tried to do something similar, but at a much smaller scale. Some random observations, that would help flesh the content out:

A big reason "type 1" reasoning is so often wrong is these decision making modules evolved under very different conditions than we currently live in.

I always liked Pinker's description (from "How the Mind Works") of the nature of the conscious mind by reverse-engineering it: it is a simulation of a serial process running on parallel wetware, for handling problems of astronomic combinatorial difficulty. This relates nicely to the cost of type 2 processes.

The modular nature of the brain and the conscious mind's primary role as a confabulator in tying together type 1 responses is nicely highlighted in Ramachandran's books, especially in his descriptions of hemispatial neglect.

You start the piece with some nice specific examples of cognitive errors, which gives the beginner reader an idea of precisely the sort of mistakes he/she can avoid by learning more... a great way to stimulate interest. Considering how important 'type 1' and 'type 2' processes are for the rest of the piece, I think it would be improved by beginning that section with similar concrete examples. Catchy names other than 'type 1' and 'type 2' would also help... maybe 'Colbert' vs. 'Stewart'?

Definite Upvote for filling the depressingly barren niche that is Introductory Postings! On a blog as big and interconnected as this one, it's hard to know where to start introducing the idea to other people. The new front page was a good start at drawing people in, and I admire your spirit in continuing that pursuit.

I love this post, personally. It starts off very well, with a few juicy bits that prompt serious thinking in the right direction right off the bat. Only problem is, my internal model of people who do not have an explicit interest in rationality got confused/bored halfway through "Types of Errors" and clicked out to play angry birds. It stopped reading like common-sense and started reading like a textbook--or worse, a manual. (Granted, a trouble-shooting manual for the human mind does not sound like a bad idea, but it's still the sort of thing that most people do not like to read until circumstances force them to.)

What is the main evidence that deliberate reasoning is computationally expensive and where would I go to read about it (books, keywords etc.)? This seems to be a well accepted theory, but I am not familiar with the science.

There are all kinds of studies showing that when working memory is occupied then system 2 but not system 1 processes are interrupted, and so on. The Frankish intro should at least point you to the sources you may be looking for.

Type 2 processes are computationally expensive, and thus humans are 'cognitive misers'. This means that we (1) default to Type 1 processes whenever possible, and (2) when we must use Type 2 processes, we use the least expensive kinds of Type 2 processes, those with a 'focal bias' — a disposition to reason from the simplest model available instead of considering all the relevant factors. Hence, we are subject to confirmation bias (our cognition is focused on what we already believe) and other biases.

I don't follow how this results in confirmation bias. Perhaps you could make it more explicit?

(Also, great article. This looks like a good way to introduce people to LW and LW-themed content.)

Ok. Simple question: what has cognitive science done for the world that has made it more rational than any other given field of psychology has to made people more rational?

For anyone curious, I am not a cognitivist.

""

Penrose uses Gödel’s incompleteness theorem (which states that there are mathematical truths which can never be proven in a sufficiently strong mathematical system; any sufficiently strong system of axioms will also be incomplete) and Turing’s halting problem (which states that there are some things which are inherently non-computable) as evidence for his position.
Searle has developed two arguments, the first (well known through his Chinese room thought experiment) is the ‘syntax is not semantics' argument—that a program is just syntax, while understanding requires semantics; therefore programs (hence cognitivism) cannot explain understanding. Such an argument presupposes the controversial notion of a private language. The second, which Searle now prefers but is less well known, is his ‘syntax is not physics’ argument—nothing in the world is intrinsically a computer program except as applied, described or interpreted by an observer, so either everything can be described as a computer and trivially a brain can but then this does not explain any specific mental processes, or there is nothing intrinsic in a brain that makes it a computer (program). Detractors of this argument might point out that the same thing could be said about any concept-object relation, and that the brain-computer analogy can be a perfectly useful model if there is a strong isomorphism between the two.[citation needed] Both points, Searle claims, refute cognitivism.""

And not for either of these reasons, but simply cause I've never seen a cognition or heard it and needed to use it as a term to achieve anything so it seems like a waste memory.