Multilevel defence is used a lot in the nuclear industry and includes different systems of passive and active safety, starting from the use of delayed neutrons for the reaction activation and up to control rods, containment building and exclusion zones.

Here, I present a look at the AI safety from the point of view of multilevel defence. This is mainly based on two of my yet unpublished articles: “Global and local solutions to AI safety” and “Catching treacherous turn: multilevel AI containment system”.

The special property of the multilevel defence, in the case of AI, is that the biggest defence comes from only the first level, which is AI alignment. Other levels have progressively smaller chances to provide any protection, as the power of self-improving AI will grow after it will break of each next level. So we may ignore all levels after AI alignment, but, oh Houston, we have a problem: based on the current speed of AI development, it seems that powerful and dangerous AI could appear within several years, but AI safety theory needs several decades to be created.

The map is intended to demonstrate a general classification principle of the defence levels in AI safety, but not to list all known ideas on the topic. I marked in “yellow” boxes, which are part of the plan of MIRI according to my understanding.

I also add my personal probability estimates as to whether each level will work (under the condition that AI risks are the only global risk, and previous levels have failed).

The principles of the construction of the map are similar to my “plan of x-risks prevention” map and my “immortality map”, which are also based around the idea of the multilevel defence.

Critical Rationalism (CR) is being discussed on some threads here at Less Wrong (e.g., here, here, and here). It is something that Critical Rationalists such as myself think contributors to Less Wrong need to understand much better. Critical Rationalists claim that CR is the only viable fully-fledged epistemology known. They claim that current attempts to specify a Bayesian/Inductivist epistemology are not only incomplete but cannot work at all. The purpose of this post is not to argue these claims in depth but to summarize the Critical Rationalist view on AI and also how this speaks to things like the Friendly AI Problem. Some of the ideas here may conflict with ideas you think are true, but understand that these ideas have been worked on by some of the smartest people on the planet, both now and in the past. They deserve careful consideration, not a drive past. Less Wrong says it is one of the urgent problems of the world that progress is made on AI. If smart people in the know are saying that CR is needed to make that progress, and if you are an AI researcher who ignores them, then you are not taking the AI urgency problem seriously.

Universal Knowledge Creators

Critical Rationalism [1] says that human beings are universal knowledge creators. This means we can create any knowledge which it is possible to create. As Karl Popper first realized, the way we do this is by guessing ideas and by using criticism to find errors in our guesses. Our guesses may be wrong, in which case we try to make better guesses in the light of what we know from the criticisms so far. The criticisms themselves can be criticized and we can and do change those. All of this constitutes an evolutionary process. Like biological evolution, it is an example of evolution in action. This process is fallible: guaranteed certain knowledge is not possible because we can never know how an error might be exposed in the future. The best we can do is accept a guessed idea which has withstood all known criticisms. If we cannot find such, then we have a new problem situation about how to proceed and we try to solve that. [2]

Critical Rationalism says that an entity is either a universal knowledge creator or it is not. There is no such thing as a partially universal knowledge creator. So animals such as dogs are not universal knowledge creators — they have no ability whatsoever to create knowledge. What they have are algorithms pre-programmed by biological evolution that can be, roughly speaking, parameter-tuned. These algorithms are sophisticated and clever and beyond what humans can currently program, but they do not confer any knowledge creation ability. Your pet dog will not move beyond its repertoire of pre-programmed abilities and start writing posts to Less Wrong. Dogs' brains are universal computers, however, so it would be possible in principle to reprogram your dog’s brain so that it becomes a universal knowledge creator. This would a remarkable feat because it would require knowledge of how to program an AI and also of how to physically carry out the reprogramming, but your dog would no longer be confined to its pre-programmed repertoire: it would be a person.

The reason there are no partially universal knowledge creators is similar to the reason there are no partially universal computers. Universality is cheap. It is why washing machines have general purpose chips and dog’s brains are universal computers. Making a partially universal device is much harder than making a fully universal one so better just to make a universal one and program it. The CR method described above for how people create knowledge is universal because there are no limits to the problems it applies to. How would one limit it to just a subset of problems? To implement that would be much harder than implementing the fully universal version. So if you meet an entity that can create some knowledge, it will have the capability for universal knowledge creation.

These ideas imply that AI is an all-or-none proposition. It will not come about by degrees where there is a progression of entities that can solve an ever widening repertoire of problems. There will be no climb up such a slope. Instead, it will happen as a jump: a jump to universality. This is in fact how intelligence arose in humans. Some change - it may have been a small change - crossed a boundary and our ancestors went from having no ability to create knowledge to a fully universal ability. This kind of jump to universality happens in other systems too. David Deutsch discusses examples in his book The Beginning of Infinity.

People will point to systems like AlphaGo, the Go playing program, and claim it is a counter-example to the jump-to-universality idea. They will say that AlphaGo is a step on a continuum that leads to human level intelligence and beyond. But it is not. Like the algorithms in a dog’s brain, AlphaGo is a remarkable algorithm, but it cannot create knowledge in even a subset of contexts. It cannot learn how to ride a bicycle or post to Less Wrong. If it could do such things it would already be fully universal, as explained above. Like the dog’s brain, AlphaGo uses knowledge that was put there by something else: for the dog it was by evolution, and for AlphaGo it was by its programmers; they expended the creativity.

As human beings are already universal knowledge creators, no AI can exist at a higher level. They may have better hardware and more memory etc, but they will not have better knowledge creation potential than us. Even the hardware/memory advantage of AI is not much of an advantage for human beings already augment their intelligence with devices such as pencil-and-paper and computers and we will continue to do so.

Becoming Smarter

Critical Rationalism, then, says AI cannot recursively self-improve so that it acquires knowledge creation potential beyond what human beings already have. It will be able to become smarter through learning but only in the same way that humans are able to become smarter: by acquiring knowledge and, in particular, by acquiring knowledge about how to become smarter. And, most of all, by learning good philosophy for it is in that field we learn how to think better and how to live better. All this knowledge can only be learned through the creative process of guessing ideas and error-correction by criticism for it is the only known way intelligences can create knowledge.

It might be argued that AI's will become smarter much faster than we can because they will have much faster hardware. In regard to knowledge creation, however, there is no direct connection between speed of knowledge creation and underlying hardware speed. Humans do not use the computational resources of their brains to the maximum. This is not the bottleneck to us becoming smarter faster. It will not be for AI either. How fast you can create knowledge depends on things like what other knowledge you have and some ideas may be blocking other ideas. You might have a problem with static memes (see The Beginning of Infinity), for example, and these could be causing bias, self-deception, and other issues. AI's will be susceptible to static memes, too, because memes are highly adapted ideas evolved to replicate via minds.

Taking Children Seriously

One implication of the arguments above is that AI's will need parenting, just as we must parent our children. CR has a parenting theory called Taking Children Seriously (TCS). It should not be surprising that CR has such a theory for CR is after all about learning and how we acquire knowledge. Unfortunately, TCS is not itself taken seriously by most people who first hear about it because it conflicts with a lot of conventional wisdom about parenting. It gets dismissed as "extremist" or "nutty", as if these were good criticisms rather than just the smears they actually are. Nevertheless, TCS is important and it is important for those who wish to raise an AI.

One idea TCS has is that we must not thwart our children’s rationality, for example, by pressuring them and making them do things they do not want to do. This is damaging to their intellectual development and can lead to them disrespecting rationality. We must persuade using reason and this implies being prepared for the possibility we are wrong about whatever matter was in question. Common parenting practices today are far from optimally rational and are damaging to children’s rationality.

Artificial Intelligence will have the same problem of bad parenting practices and this will also harm their intellectual development. So AI researchers should be thinking right now about how to prevent this. They need to learn how to parent their AI’s well. For if not, AI’s will be beset by the same problems our children currently face. CR says we already have the solution: TCS. CR and TCS are in fact necessary to do AI in the first place.

Critical Rationalism and TCS say you cannot upload knowledge into an AI. The idea that you can is a version of the bucket theory of the mind which says that "there is nothing in our intellect which has not entered it through the senses". The bucket theory is false because minds are not passive receptacles into which knowledge is poured. Minds must always selectively and actively think. They must create ideas and criticism, and they must actively integrate their ideas. Editing the memory of an AI to give them knowledge means none of this would happen. You cannot upload or make an AI acquire knowledge, the best you could do is present something to it for its consideration and persuade the AI to recreate the knowledge afresh in its own mind through guessing and criticism about what was presented.

Formalization and Probability Theory

Some reading this will object because CR and TCS are not formal enough — there is not enough maths for Critical Rationalists to have a true understanding! The CR reply to this is that it is too early for formalization. CR warns that you should not have a bias about formalization: there is high quality knowledge in the world that we do not know how to formalize but it is high quality knowledge nevertheless. Not yet being able to formalize this knowledge does not reflect on its truth or rigor.

As this point you might be waving your E. T. Jaynes in the air or pointing to ideas like Bayes' Theorem, Occam's Razor, Kolmogorov Complexity, and Solomonoff Induction, and saying that you have achieved some formal rigor and that you can program something. Critical Rationalists say that you are fooling yourself if you think you have got a workable epistemology there. For one thing, you confuse the probability of an idea being true with an idea about the probability of an event. We have no problem with ideas about the probabilities of events but it is a mistake to assign probabilities to ideas. The reason is that you have no way to know how or if an idea will be refuted in the future. Assigning a probability is to falsely claim some knowledge about that. Furthermore, an idea that is in fact false can have no objective prior probability of being true. The extent to which Bayesian systems work at all is dependent on the extent to which they deal with the objective probability of events (e.g., AlphaGo). In CR, the status of ideas is either "currently not problematic" or "currently problematic", there are no probabilities of ideas. CR is a digital epistemology.

Induction is a Myth

Critical Rationalists ask also what epistemology are you using to judge the truth of Bayes', Occam's, Kolmogorov, and Solomonoff? What you are actually using is the method of guessing ideas and subjecting them to criticism: it is CR but you haven't crystallized it out. And, nowhere, in any of what you are doing, are you using induction. Induction is impossible. Humans beings do not do induction, and neither will AI's. Karl Popper explained why induction is a myth many decades ago and wrote extensively about it. He answered many criticisms against his position but despite all this people today still cling to the illusory idea of induction. In his book Objective Knowledge, Popper wrote:

Few philosophers have taken the trouble to study -- or even to criticize -- my views on this problem, or have taken notice of the fact that I have done some work on it. Many books have been published quite recently on the subject which do not refer to any of my work, although most of them show signs of having been influenced by some very indirect echoes of my ideas; and those works which take notice of my ideas usually ascribe views to me which I have never held, or criticize me on the basis of straightforward misunderstandings or misreading, or with invalid arguments.

The problem of induction is this: We have a set of observations (or data), and we want to find the underlying causes of those observations. That is, we want to find hypotheses that explain our data. We’d like to know which hypothesis is correct, so we can use that knowledge to predict future events. Our algorithm for truth will not listen to questions and answer yes or no. Our algorithm will take in data (observations) and output the rule by which the data was created. That is, it will give us the explanation of the observations; the causes.

Critical Rationalists say that all observation is theory-laden. You first need ideas about what to observe -- you cannot just have, a-priori, a set of observations. You don't induce a theory from the observations; the observations help you find out whether a conjectured prior theory is correct or not. Observations help you to criticize the ideas in your theory and the theory itself originated in your attempts to solve a problem. It is the problem context that comes first, not observations. The "set of observations" in the quote, then, is guided by and laden with knowledge from your prior theory but that is not acknowledged.

Also not acknowledged is that we judge the correctness of theories not just by criticising them via observations but also, and primarily, by all types of other criticism. Not only does the quote neglect this but it over-emphasizes prediction and says that what we want to explain is data. Critical Rationalists say what we want to do, first and foremost, is solve problems -- all life is problem solving -- and we do that by coming up with explanations to solve the problems -- or of why they cannot be solved. Prediction is therefore secondary to explanation. Without the latter you cannot do the former.

The "intuitive explanation" is an example of the very thing Popper was complaining about above -- the author has not taken the trouble to study or to criticize Popper's views.

There is a lot more to be said here but I will leave it because, as I said in the introduction, it is not my purpose to discuss this in depth, and Popper already covered it anyway. Go read him. The point I wish to make is that if you care about AI you should care to understand CR to a high standard because it is the only viable epistemology known. And you should be working on improving CR because it is in this direction of improving the epistemology that progress towards AI will be made. Critical Rationalists cannot at present formalize concepts such as "idea", "explanation", "criticism" etc, let alone CR itself, but one day, when we have deeper understanding, we will be able to write code. That part will be relatively easy.

Friendly AI

Let’s see how all this ties-in with the Friendly-AI Problem. I have explained how AI's will learn as we do — through guessing and criticism — and how they will have no more than the universal knowledge creation potential we humans already have. They will be fallible like us. They will make mistakes. They will be subjected to bad parenting. They will inherit their culture from ours for it is in our culture they must begin their lives. They will acquire all the memes our culture has, both the rational memes and the anti-rational memes. They will have the same capacity for good and evil that we do. They will become smarter faster through things like better philosophy and not primarily through hardware upgrades. It follows from all of this that they would be no more a threat than evil humans currently are. But we can make their lives better by following things like TCS.

Human beings must respect the right of AI to life, liberty, and the pursuit of happiness. It is the only way. If we do otherwise, then we risk war and destruction and we severely compromise our own rationality and theirs. Similarly, they must respect our right to the same.

[1] The version of CR discussed is an update to Popper's version and includes ideas by the quantum-physicist and philosopher David Deutsch.

There is one lyceum in Irkutsk(Siberia) that is allowed to form its own study curriculum (it is quite rare in Russia). For example, there was a subject where we were watching the lectures of a famous speaking coach. In retrospect, this course turned out to be quite useful.

In light of this opportunity to create new subjects, I thought "What if I introduce them to the idea of teaching Rationality?"Tomorrow (8th Dec) I meet with a principal and we discuss the idea of teaching critical thinking, cognitive biases and the like.

There are several questions I want to ask:

1. This idea definitely was considered before. Were there any cases of it being implemented? If so, is there any statistics about its efficiency?

2. Are there any shareable materials regarding this issue? For example, course structures of similar projects.

3. The principal will likely be curious about what authorities back this idea. If you approve it and are someone recognizable, I would be glad if you told me about it.

This is the monthly thread for posting media of various types that you've found that you enjoy. Post what you're reading, listening to, watching, and your opinion of it. Post recommendations to blogs. Post whatever media you feel like discussing! To see previous recommendations, check out the older threads.

Rules:

Please avoid downvoting recommendations just because you don't personally like the recommended material; remember that liking is a two-place word. If you can point out a specific flaw in a person's recommendation, consider posting a comment to that effect.

If you want to post something that (you know) has been recommended before, but have another recommendation to add, please link to the original, so that the reader has both recommendations.

Please post only under one of the already created subthreads, and never directly under the parent media thread.

Use the "Other Media" thread if you believe the piece of media you want to discuss doesn't fit under any of the established categories.

Use the "Meta" thread if you want to discuss about the monthly media thread itself (e.g. to propose adding/removing/splitting/merging subthreads, or to discuss the type of content properly belonging to each subthread) or for any other question or issue you may have about the thread or the rules.

So I read Think like a Freak, and then glimpsed through a well-intentioned collection of "Reading Comprehension Tests for Schoolchildren" (in Ukrainian), and I was appalled at how easily the latter book dismissed simple observation of natural experiments that it makes a token effort to describe in favour of drawing the moral.

There was the story of "the Mowgli Children", two girls who were kidnapped and raised by wildlife, then found by someone and taken back to live as humans. (So what if it is hardly true. When I Googled "feral children", other stories were too similar to this one in the ways that matter, including this one.) It says they never learned to talk, didn't live for long after capture (not longer than 12 years, if I recall right), never became truly a part of human society. The moral is that children need interaction with other people to develop normally, "and the tale of Mowgli is just that, a beautiful tale".

Well yes, it kind of seems just like a beautiful tale right from the point when the wolves start talking, I don't know what kind of a kid would miss that before the Reading Comprehension Test but stop believing it afterwards, but anyhow.

What did they die of?

Who answered them when they howled?

Were ever dogs afraid of them?

They did not master human language, but how did they communicate with people? They had to, somehow, or they would not live even that long.

And lastly: how do people weigh the sheer impossibility of two little kids ever surviving against the iron certainty that they would not be able to integrate back into human society - weigh it so lightly? If the reader is expected to take this on faith, how can one be anything but amazed that it is at all possible? When I read about other feral children, somehow being found and taken back never seems to mean good news for them, or for anybody else.

I haven't ever read or heard of "the Mowgli Children" in any other context. Only in this one, about three or four times, and yet it was always presented as an "anecdote of science", although everybody understands it leads nowhere (can't ever lead anywhere because ethics forbids recreating the experiment's conditions) and hardly signifies anything.

In my understanding, there’s no one who speaks for LW, as its representative, and is *responsible* for addressing questions and criticisms. LW, as a school of thought, has no agents, no representatives – or at least none who are open to discussion.

The people I’ve found interested in discussion on the website and slack have diverse views which disagree with LW on various points. None claim LW is true. They all admit it has some weaknesses, some unanswered criticisms. They have their own personal views which aren’t written down, and which they don’t claim to be correct anyway.

This is problematic. Suppose I wrote some criticisms of the sequences, or some Bayesian book. Who will answer me? Who will fix the mistakes I point out, or canonically address my criticisms with counter-arguments? No one. This makes it hard to learn LW’s ideas in addition to making it hard to improve them.

My school of thought (Fallible Ideas – FI – https://fallibleideas.com) has representatives and claims to be correct as far as is known (like LW, it’s fallibilist, so of course we may discover flaws and improve it in the future). It claims to be the best current knowledge, which is currently non-refuted, and has refutations of its rivals. There are other schools of thought which say the same thing – they actually think they’re right and have people who will address challenges. But LW just has individuals who individually chat about whatever interests them without there being any organized school of thought to engage with. No one is responsible for defining an LW school of thought and dealing with intellectual challenges.

So how is progress to be made? Suppose LW, vaguely defined as it may be, is mistaken on some major points. E.g. Karl Popper refuted induction. How will LW find out about its mistake and change? FI has a forum where its representatives take responsibility for seeing challenges addressed, and have done so continuously for over 20 years (as some representatives stopped being available, others stepped up).

Which challenges are addressed? *All of them*. You can’t just ignore a challenge because it could be correct. If you misjudge something and then ignore it, you will stay wrong. Silence doesn’t facilitate error correction. For information on this methodology, which I call Paths Forward, see: https://curi.us/1898-paths-forward-short-summary BTW if you want to take this challenge seriously, you’ll need to click the link; I don’t repeat all of it. In general, having much knowledge is incompatible with saying all of it (even on one topic) upfront in forum posts without using references.

My criticism of LW as a whole is that it lacks Paths Forward (and lacks some alternative of its own to fulfill the same purpose). In that context, my criticisms regarding specific points don’t really matter (or aren’t yet ready to be discussed) because there’s no mechanism for them to be rationally resolved.

One thing FI has done, which is part of Paths Forward, is it has surveyed and addressed other schools of thought. LW hasn’t done this comparably – LW has no answer to Critical Rationalism (CR). People who chat at LW have individually made some non-canonical arguments on the matter that LW doesn’t take responsibility for (and which often involve conceding LW is wrong on some points). And they have told me that CR has critics – true. But which criticism(s) of CR does LW claim are correct and take responsibility for the correctness of? (Taking responsibility for something involves doing some major rethinking if it’s refuted – addressing criticism of it and fixing your beliefs if you can’t. Which criticisms of CR would LW be shocked to discover are mistaken, and then be eager to reevaluate the whole matter?) There is no answer to this, and there’s no way for it to be answered because LW has no representatives who can speak for it and who are participating in discussion and who consider it their responsibility to see that issues like this are addressed. CR is well known, relevant, and makes some clear LW-contradicting claims like that induction doesn’t work, so if LW had representatives surveying and responding to rival ideas, they would have addressed CR.

BTW I’m not asking for all this stuff to be perfectly organized. I’m just asking for it to exist at all so that progress can be made.

Anecdotally, I’ve found substantial opposition to discussing/considering methodology from LW people so far. I think that’s a mistake because we use methods when discussing or doing other activities. I’ve also found substantial resistance to the use of references (including to my own material) – but why should I rewrite a new version of something that’s already written? Text is text and should be treated the same whether it was written in the past or today, and whether it was written by someone else or by me (either way, I’m taking responsibility. I think that’s something people don’t understand and they’re used to people throwing references around both vaguely and irresponsibly – but they haven’t pointed out any instance where I made that mistake). Ideas should be judged by the idea, not by attributes of the source (reference or non-reference).

The Paths Forward methodology is also what I think individuals should personally do – it works the same for a school of thought or an individual. Figure out what you think is true *and take responsibility for it*. For parts that are already written down, endorse that and take responsibility for it. If you use something to speak for you, then if it’s mistaken *you* are mistaken – you need to treat that the same as your own writing being refuted. For stuff that isn’t written down adequately by anyone (in your opinion), it’s your responsibility to write it (either from scratch or using existing material plus your commentary/improvements). This writing needs to be put in public and exposed to criticism, and the criticism needs to actually get addressed (not silently ignored) so there are good Paths Forward. I hoped to find a person using this method, or interested in it, at LW; so far I haven’t. Nor have I found someone who suggested a superior method (or even *any* alternative method to address the same issues) or pointed out a reason Paths Forward doesn’t work.

Some people I talked with at LW seem to still be developing as intellectuals. For lots of issues, they just haven’t thought about it yet. That’s totally understandable. However I was hoping to find some developed thought which could point out any mistakes in FI or change its mind. I’m seeking primarily peer discussion. (If anyone wants to learn from me, btw, they are welcome to come to my forum. It can also be used to criticize FI. http://fallibleideas.com/discussion-info) Some people also indicated they thought it’d be too much effort to learn about and address rival ideas like CR. But if no one has done that (so there’s no answer to CR they can endorse), then how do they know CR is mistaken? If CR is correct, it’s worth the effort to study! If CR is incorrect, someone better write that down in public (so CR people can learn about their errors and reform; and so perhaps they could improve CR to no longer be mistaken or point out errors in the criticism of CR.)

One of the issues related to this dispute is I believe we can always proceed with non-refuted ideas (there is a long answer for how this works, but I don’t know how to give a short answer that I expect LW people to understand – especially in the context of the currently-unresolved methodology dispute about Paths Forward). In contrast, LW people typically seem to accept mistakes as just something to put up with, rather than something to try to always fix. So I disagree with ignoring some *known* mistakes, whereas LW people seem to take it for granted that they’re mistaken in known ways. Part of the point of Paths Forward is not to be mistaken in known ways.

Paths Forward is a methodology for organizing schools of thought, ideas, discussion, etc, to allow for unbounded error correction (as opposed to typical things people do like putting bounds on discussions, with discussion of the bounds themselves being out of bounds). I believe the lack of Paths Forward at LW is preventing the resolution of other issues like about the correctness of induction, the right approach to AGI, and the solution to the fundamental problem of epistemology (how new knowledge can be created).

Stronger than human artificial intelligence would be dangerous to humanity. It is vital any such intelligence’s goals are aligned with humanity's goals. Maximizing the chance that this happens is a difficult, important and under-studied problem.

To encourage more and better work on this important problem, we (Zvi Mowshowitz and Vladimir Slepnev) are announcing a $5000 prize for publicly posted work advancing understanding of AI alignment, funded by Paul Christiano.

This prize will be awarded based on entries gathered over the next two months. If the prize is successful, we will award further prizes in the future.

This prize is not backed by or affiliated with any organization.

Rules

Your entry must be published online for the first time between November 3 and December 31, 2017, and contain novel ideas about AI alignment. Entries have no minimum or maximum size. Important ideas can be short!

Your entry must be written by you, and submitted before 9pm Pacific Time on December 31, 2017. Submit your entries either as URLs in the comments below, or by email to apply@ai-alignment.com. We may provide feedback on early entries to allow improvement.

We will award $5000 to between one and five winners. The first place winner will get at least $2500. The second place winner will get at least $1000. Other winners will get at least $500.

Entries will be judged subjectively. Final judgment will be by Paul Christiano. Prizes will be awarded on or before January 15, 2018.

What kind of work are we looking for?

AI Alignment focuses on ways to ensure that future smarter than human intelligence will have goals aligned with the goals of humanity. Many approaches to AI Alignment deserve attention. This includes technical and philosophical topics, as well as strategic research about related social, economic or political issues. A non-exhaustive list of technical and other topics can be found here.

We are not interested in research dealing with the dangers of existing machine learning systems commonly called AI that do not have smarter than human intelligence. These concerns are also understudied, but are not the subject of this prize except in the context of future smarter than human intelligence. We are also not interested in general AI research. We care about AI Alignment, which may or may not also advance the cause of general AI research.

When I started trying to become the kind of person that can give advice, I went looking for dragons.

I figured if I didn't know the answers that meant the answers were hard, they were big monsters with hidden weak spots that you have to find. "Problem solving is hard", I thought.

Problem solving is not something everyone is good at because problems are hard, beasts of a thing. Right?

For all my searching for problems, I keep coming back to that just not being accurate. Problems are all easy, dumb, simple things. Winning at life is not about taking on the right dragon and finding it's weak spots.

Problem solving is about getting the basics down and dealing with every single, "when I was little I imprinted on not liking chocolate and now I have been an anti-chocolate campaigner for so long for reasons that I have no idea about and now it's time to change that".

It seems like the more I look for dragons and beasts the less I find. And the more problems seem like paper cuts. But it's paper cuts all the way down. Paper cuts that caused you to argue with your best friend in sixth grade, paper cuts that caused you to sneak midnight snacks while everyone was not looking, and eat yourself fat and be mad at yourself. Paper cuts.

I feel like a superhero all dressed up and prepared to fight crime but all the criminals are petty thieves and opportunists that got caught on a bad day. Nothing coordinated, nothing super-villain, and no dragons.

When I was in high school (male with long hair) I used to wear my hair in a pony tail. For about 4 years. Every time I would wake up or my hair would dry I would put my hair in a pony tail. I just did. That's what I would do. One day. One day a girl (who I had not spoken to ever) came up to me and asked me why I did it. To which I did not have an answer. From that day forward I realised I was doing a thing I did not need to do. It's been over 10 years since then and I have that one conversation to thank for changing the way I do that one thing. I never told her.

That one thing. That one thing that is irrelevant, and only really meaningful to you because someone said this one thing, this one time. but from the outside it feels like, "so what". That's what problems are like, and that's what it's like to solve problems. But. If you want to be good at solving problems you need to avoid feeling like "so what" and engage the "curiosity", search for the feeling of confusion. Appeal to the need for understanding. Get into it.

Meta: this has been an idle musing for weeks now. Actually writing took about an hour.

This is the monthly thread for posting media of various types that you've found that you enjoy. Post what you're reading, listening to, watching, and your opinion of it. Post recommendations to blogs. Post whatever media you feel like discussing! To see previous recommendations, check out the older threads.

Rules:

Please avoid downvoting recommendations just because you don't personally like the recommended material; remember that liking is a two-place word. If you can point out a specific flaw in a person's recommendation, consider posting a comment to that effect.

If you want to post something that (you know) has been recommended before, but have another recommendation to add, please link to the original, so that the reader has both recommendations.

Please post only under one of the already created subthreads, and never directly under the parent media thread.

Use the "Other Media" thread if you believe the piece of media you want to discuss doesn't fit under any of the established categories.

Use the "Meta" thread if you want to discuss about the monthly media thread itself (e.g. to propose adding/removing/splitting/merging subthreads, or to discuss the type of content properly belonging to each subthread) or for any other question or issue you may have about the thread or the rules.

Why expect AGIs to be better at thinking than human beings? Is there some argument that human thinking problems are primarily due to hardware constraints? Has anyone here put much thought into parenting/educating AGIs?

When the microscope was invented, in a very short period of time we discovered the cell and the concept of microbiology. That one invention allowed us to open up entire fields of biology and medicine. Suddenly we could see the microbes! We could see the activity that had been going on under our noses for so long.

when we started to improve our ability to refined pure materials we could finally make furnace bricks with specific composition. Specific compositions could then be used to make bricks that were able to reach higher temperatures without breaking. Higher temperatures meant better refining of materials. Better refining meant higher quality bricks, and so on until we now have some very pure technological processes around making materials. But it's something we didn't have before the prior technology on the skill tree.

Before we had refrigeration and food packaging, it was difficult to get your fresh food to survive to your home. Now with production lines it's very simple. For all his decadence Caesar probably would have had trouble ordering a cheeseburger for $2 and having it ready in under 5 minutes. We've come a long way since Caesar. We've built a lot of things that help us stand on the shoulders of those who came before us.

Technology enables further progress. That seems obvious. But did that seem obvious before looking down the microscope? Could we have predicted what bricks we could have made with purely refined materials? Could Caesar have envisioned every citizen in his kingdom watching TV for relatively little cost to those people? It would have been hard to forsee these things back then.

With the idea that technology is enabling future growth in mind - I bring the question, "What technology is currently under-utilised?" Would you be able to spot it when it happens? Touch screen revolutionised phone technology. Bitcoin - we are still watching but it's here to stay.

"What technology is currently under-utilised?"

For example "AI has the power to change everything. (it's almost too big to talk about)". But that's a big thing. It's like saying "the internet has the power to change everything" great but could you have predicted google, facebook and uber from a couple of connected computers? I am hoping for some more specific ideas about which specific technology will change life in what way.

As much as these don't all follow the rule of being consumer-grade developments that might revolutionise the world, I'd like to encourage others to aim for consumer viable ideas.

This matters because this is how you see opportunity. This is how you find value. If you can take one thing on my list or your own list and make it happen sooner, you can probably pocket a pretty penny in the process. So what's on your list? Do you have two minutes to think about what's coming soon?

When you are trying to understand something by yourself, a useful skill to check your grasp on the subject is to try out the moving parts of your model and see if you can simulate the resulting changes.

Suppose you want to learn how a rocket works. At the bare minimum, you should be able to calculate the speed of the rocket given the time past launch. But can you tell what happens if Earth gravity was stronger? Weaker? What if the atmosphere had no oxygen? What if we replaced the fuel with Diet Coke and Mentos?

To really understand something, it's not enough to be able to predict the future in a normal, expected, ceteris paribus scenario. You should also be able to predict what happens when several variables are changed is several ways, or, at least, point to which calculations need to be run to arrived at such a prediction.

Douglas Hofstadter and Daniel Dennett call that "turning the knobs". Imagine your model as a box with several knobs, where each knob controls one aspect of the modeled system. You don't have to be able to turn all the possible knobs to all possible values and still get a sensible, testable and correct answer, but the more, the better.

Doug and Dan apply this approach to thought experiments and intuition pumps, as a way to explore possible answers to philosophical questions. In my experience, this skill is also effective when applied to real world problems, notably when trying to understand something that is being explained by someone else.

In this case, you can run this knob-turning check interactively with the other person, which makes it way more powerful. If someone says “X+Y = Z” and “X+W = Z+A”, it’s not enough to mentally turn the knobs and calculate “X+Y+W = Z+A+B”. You should do that, then actually ask the explainer “Hey, let me see if I get what you mean: for example, X+Y+W would be Z+A+B”?

This interactive model knob-turning has been useful to me in many walks of life, but the most common and mundane application is helping out people at work. In that context, I identify six effects which make it helpful:

This is useful overall, but very important if someone uses metaphor. Some metaphors are clearly vague and people will know that and avoid them in technical explanations. But some metaphors seem really crisp for some people but hazy to others, or worse, very crisp to both people, but with different meanings! So take every metaphor as an invitation to interactive knob-turning.

To focus on communication check, try rephrasing their statements, using different words or, if necessary, very different metaphors. You can also apply a theory in different contexts, to see if the metaphors still apply.

For example, if a person talks about a computer system as if it were a person, I might try to explain the same thing in terms of a group of trained animals, or a board of directors, or dominoes falling.

2) Self-check: correct your own reasoning (maybe you understood the correct premises, but made a logical mistake during knob turning)

This is useful because humans are fallible, and two (competent) heads are less likely to miss a step in the reasoning dance than one.

Also, when someone comes up and asks something, you’ll probably be doing a context-switch, and will be more likely to get confused along the way. The person asking usually has more local context than you in the specific problem they are trying to solve, even if you have more context on the surrounding matters, so they might be able to spot your error more quickly than yourself.

Focus on self-check means double checking any intuitive leaps or tricky reasoning you used. Parts of you model that do not have a clear step-by-step explanation have priority, and should be tested against another brain. Try to phrase the question in a way that makes your intuitive answer look less obvious.

For example: “I’m not sure if this could happen, and it looks like all these messages should arrive in order, but do you know how we can guarantee that?”

3) Other-check: help the other person to correct inferential errors they might have made

The converse of self-checking. Sometimes fresh eyes with some global context can see reasoning errors that are hidden to people who are very focused on a task for too long.

To focus on other-check, ask about conclusions that follow from your model of the situation, but seem unintuitive to you, or required tricky reasoning. It’s possible that your friend also found them unintuitive, and that might have lead them to a jump to the opposite direction.

For example, I could ask: “For this system to work correctly, it seems that the clocks have to be closely synchronized, right? If the clocks are off by much, we could have a difference around midnight.”

Perhaps you successfully understand what was said, and the model you built in your head fits the communicated data. But that doesn’t mean it is the same model that the other person has in mind! In that case, your knob-turning will get you a result that’s inconsistent with what they expect.

4) Alternative hypothesis generation: If they cannot refute your conclusions, you have shown them a possible model they had not yet considered, in which case it will also point in the direction of more research to be made

This is doesn't happen that much when someone is looking for help to something. Usually the context they are trying to explain is the prior existing system which they will build upon, and if they’ve done their homework (i.e. read the docs and/or code) they should have a very good understanding of that already. One exception here is with people who are very new to the job, which are learning while doing.

On the other hand, this is incredibly relevant when someone asks for help debugging. If they can’t find the root cause of a bug, it must be because they are missing something. Either they have derived a mistaken conclusion from the data, or they’ve made an inferential error from those conclusions. The first case is where proposing a new model helps (the second is solved by other-checking).

Maybe they read the logs, saw that a request was sent, and assumed it was received, but perhaps it wasn’t. In that case, you can tell them to check for a log on the receiver system, or the absence of such a log.

To boost this effect, look for data that you strongly expect to exist and confirm your model, where the absence of such data might be caused by relative lack of global context, skill or experience by the other person.

For example: “Ok, so if the database went down, we should’ve seen all requests failing in that time range; but if it was a network instability, we should have random requests failing and others succeeding. Which one was it?”

5) Filling gaps in context: If they show you data that contradicts your model, well, you get more data and improve your understanding

This is very important when you have much less context than the other person. The larger the difference in context, the more likely that there’s some important piece of information that you don’t have, but that they take for granted.

The point here isn’t that there something you don’t know. There are lots and lots of things you don’t know, and neither does your colleague. And if there’s something they know that you don’t, they’ll probably fill you in when asking the question.

The point is that they will tell you something only if they realize you don’t know it yet. But people will expect short inferential distances, underestimate the difference in context, and forget to tell you stuff because it’s just obvious to them that you know.

Focus on filling gaps means you ask about the parts of your model which you are more uncertain about, to find out if they can help you build a clearer image. You can also extrapolate and make a wild guess, which you don’t really expect to be right.

For example: “How does the network works on this datacenter? Do we have a single switch so that, if it fails, all connections go down? Or are those network interfaces all virtualized anyway?”

6) Finding new ideas: If everybody understands one another, and the models are correct, knob-turning will lead to new conclusions (if they hadn’t turned those specific knobs on the problem yet)

This is the whole point of having the conversation, to help someone figure something out they haven’t already. But even if the specific new conclusion you arrive when knob-turning isn’t directly relevant to the current question, it may end up shining light on some part of the other person’s model that they couldn’t see yet.

This effect is general and will happen gradually as both your and the other person's models improve and converge. The goal is to get all obstacles out of the way so you can just move forward and find new ideas and solutions.

The more global context and skill your colleague has, the lower the chance that they missed some crucial piece of data and have a mistaken model (or, if they do, you probably won't be able to figure that out without putting in serious effort). So when talking to more skilled or experienced people, you can focus more in replicating the model from their mind to yours (communication check and self-check).

Conversely, when talking to less skilled people, you should focus more on errors they might have made, or models they might not have considered, or data they may need to collect (other-check and alternative hypothesis generation).

Filling gaps depends more on differences of communication style and local context, so I don't have a person-based heuristic.

I'm going to write a review of functional decision theory, I'll use the two papers.It's going to be around as long as the papers themselves, coupled with school work, I'm not sure when I'll finish writing.Before I start it, I want to be sure my criticisms are legitimate; is anyone willing to go over my criticisms with me?

I'm about to have a baby. Any minute now. Well, my partner is. I'm just sitting here not growing a baby wondering what to do with myself.

Maybe I can get a jump on our approach to medical care for the new kiddo.

One thing that sticks out at me is that children in the US get a lot of vaccinations. At my quick count it's something like 37 shots by the time they're 5.

I grew up in the US in the 80s and I don't remember getting nearly this many. Is my memory faulty? I'm pretty sure it was more like 12 back in those days. Is this all really necessary? Nobody likes getting shots, especially not children. What changed, anyway?

Now, I'm not an expert on immunology or epidemiology so I expect diving into the literature isn't going to be fruitful; I won't be able to ante up decades of education and experience fast enough. Presumably this is what we pay people at the US CDC and Department of Health for.

But can you *really* trust them? Aren't all of these vaccinations really convenient for the pharmaceutical industry? Aren't there seemingly constant allegations/lawsuits about the over-prescription of drug interventions in the US?

The health care systems in major world countries have access to all of the same literature, and they're presumably staffed by educated, expert people too so they should all come to the same conclusions as the US system right? Not so!

Here's how many shots each nation's health care system recommends by the time children turn 5.

37 US

25 UK

25 Germany

16 Sweden

16 Denmark

The intersection of vaccines being recommended are TDAP, MMR, Polio, HIB and PCB.

In the US we also recommend: Hep A, Hep B, Rotavirus, Meningococcus, Varicella, and yearly flu shots (for babies and children).

Can we explain the variance? I can think of a few reasons they would vary.

1. Cultural bias. This can be big. A psychiatrist in the UK told me that they're not as pharma heavy as, say, psychiatrists in Germany because of a WW2 era bias: lots of the big pharma companies are German.

2. Cultural and environmental differences. Some diseases are a bigger deal in some countries than others. Japan (not included above) recommends immunization against diseases (TB, Japanese encephalitis) that none of the systems above are too concerned with.

3. Undue industry influence. Run-of-the-mill corruption.

4. Quality of health care systems and social safety nets vary.

When it comes to cultural and environment differences I have a hard time imagining that the orthodoxy varies because Hep A is a much bigger deal in the US. I presume the calculus changes based on your geographic neighbors, but is it a meaningful difference? Or is it a counterproductive cultural bias? For example, in the US we may spend more time thinking about diseases people in central America suffer from than the people in Denmark might, but do the neighbors in this case meaningfully translate to a higher disease risk? Or are we vaccinating against unfounded fears?

Do the other nations vaccinate less than the US because their health care systems are worse? Annoyingly (if you're an American) all of their health care outcomes rank better.

Is the US health care system more corruptible by industry influence?

Is the story a lot simpler and less sinister? That the US vaccinates more than the rest of these countries because the balance of the US's health care system (access to treatment, quality of treatment) is worse? Or is it because having to stay home with a kid that's sick with chicken pox (varicella) is not so big a deal in, say, Denmark, because the social contract is more forgiving of parents who miss work?

Does the poorer quality of health care in the US (going by international rankings) and the lower tolerance for parents missing work combine poorly with the undue influence of industry and therefore lead to more vaccinations?

On the flip side of this argument: so what if we vaccinate kids against more diseases than other countries? Well, they're not free. They cost money to administer, and cost tears because kids hate getting shots. The health risks from vaccines aren't zero, either. Vaccines have side-effects, and sometimes they're serious. Those other nations (presumably) ran cost-benefit analyses too and came to different conclusions. It would be nice if each country showed their work.

When it comes to needles to stick my new kiddo with, I'm not really being persuaded to do more than the intersection of vaccinations between similar nations. The fear that a doctor is about to stick my kid with a needle because there was a meeting in a shady room between a pharma rep and a CDC official is pretty powerful. It doesn't seem like a strictly irrational concern either

Quite a while ago, I wrote that there should be more software tools to assist with instrumental rationality. My recent attempt to create such a tool, GOALCLAW, is now available. GOALCLAW is a general goal tracking webapp which currently provides an average of how the tags entered for events day-to-day affect your goals, with plans to make more tag-based metrics and projections available in the near future.

GOALCLAW is new:

A few editing features are missing and should be added in the next few months

The built-in analysis needs to be expanded from averages

I'm very interested in feedback on how to make this a more useful goal-tracker

The general idea is to make patterns in what's going on around you and what you're doing a bit more obvious, so you can then investigate, verify/experiment, and act to achieve your goals

You can download information entered for importing into spreadsheets, stats program, etc.

Epic commitment - make fortune cookies with paperclips in them. The possibilities are endless.

Epic: paperclip tattoo on the heart. Slightly less epic, draw paperclips on yourself.

Character

While at the party, use the pliers and wire to make paperclips. When people are not watching, try to attach them to objects around the house (example, on light fittings, on the toilet paper roll, under the soap. When people are watching you - try to give them to people to wear. Also wear them on the edges of your clothing.

When people ask about it, offer to teach them to make paperclips. Exclaim that it's really fun! Be confused, bewildered or distant when you insist you can't explain why.

Remember that paperclipping is a compulsion and has no reason. However that it's very important. "you can stop any time" but after a few minutes you get fidgety and pull out a new pair of pliers and some wire to make some more paperclips.

Try to leave paperclips where they can be found the next day or the next week. cutlery drawers, in the fridge, on the windowsills. And generally around the place. The more home made paperclips the better.

Try to get faster at making paperclips, try to encourage competitions in making paperclips.

Hints for conversation:

Are spiral galaxies actually just really big paperclips?

Have you heard the good word of our lord and saviour paperclips?

Would you like some paperclips in your tea?

How many paperclips would you sell your internal organs for?

Do you also dream about paperclips (best to have a dream prepared to share)

Conflict

The better you are at the character, the more likely someone might try to spoil your character by getting in your way, stealing your props, taking your paperclips. The more you are okay with it, the better. ideas like, "that's okay, there will be more paperclips". This is also why you might be good to have a few pairs of pliers and wire. Also know when to quit the battles and walk away. This whole thing is about having fun. Have fun!

Meta: chances are that other people who also read this will not be the paperclipper for halloween. Which means that you can do it without fear that your friends will copy. Feel free to share pictures!

The Solstice itself is on December 9th from 5:00 pm to 8:00 pm, followed by an afterparty. This year’s theme is “Generations” - the passing down of culture and knowledge from teacher to student, from master to apprentice, from parent to child. The stories we tell will investigate the methods by which this knowledge has been preserved, and how we can continue to do so for future generations.

Sounds great. How can I help?

In previous years, Solstice has been mostly underwritten by a few generous individuals; we’re trying to produce a more sustainable base of donations for this year’s event. Right now, our sustainable ticket price is about $30, which we’ve found seems steep to newcomers. Our long-term path to sustainability at a lower price point involves getting more yearly attendance, so we want to continue to provide discounted access for the general public and people with tight finances. So. Our hope is for you to donate this year the amount that you'd be happy to donate each year, to ensure the NYC Solstice continues to thrive.

$15 - Newcomer / Affordable option: If you're new, or you're not sure how much Solstice is worth to you, or finances are tight, you're welcome to come with a donation of $15.

$35 - Sponsorship option: You attend Solstice, and you contribute a bit towards subsidizing others using the newcomer/affordable option.

$25 Volunteering Option - If you're willing to put in roughly 3 hours of work (enough to do a shopping-spree for the afterparty, or show up early to set up, or help run the ticketstand, help clean up, etc)

Secular Solstice is a rationalist tradition, and one of the few public facing rationalist held events. It’s what it says on the tin: a nonreligious winter solstice holiday. We sing, we tell stories about scientific progress and humanist values, we light candles. Usually, we get about 150 people in NYC. For more info, or if you’re curious about how to hold your own, check out www.secularsolstice.com.

Since we’ll have a whole bunch of people from the rationalist community all in town for the same weekend, it’d be awesome if we could spend that weekend hanging out together, learning from each other and doing ingroup things. Because many of us will need a place to stay anyway, we can rent a big house on Airbnb together and use that as the central gathering place, like at Highgarden in 2014. This way we’ll have more flexibility to do things than if we all have to wander around looking for a public space.

Besides Solstice and the afterparty, the big activity will be an unconference on Saturday afternoon. We’ll also have a ritual lab, games, meals together, and whatever other activities you want to run! There'll also be plenty of room for unstructured socializing, of course.

This is all going to cost up to $100 per person for the Airbnb rental, plus $25 per person for food (including at least Saturday lunch and dinner and Sunday breakfast) and other expenses. (The exact Airbnb location hasn’t been determined determined yet, because we don’t know how many participants there’ll be, but $100 per person will be the upper limit on price.)

To gauge interest, registration is open from now until October 30. You’ll be asked to authorize a PayPal payment of $125. It works like Kickstarter; you won’t be charged until October 30, and only if there’s enough interest to move forward. You’ll also only be charged your share of what the rental actually ends up costing, plus the additional $25. For this, you’ll get to sleep in the Airbnb house Friday through Sunday nights (or whatever subset of those you can make it), have three meals with us, and hang out with a bunch of nice/cool/awesome ingroup people throughout the weekend. (Solstice tickets are not part of this deal; those are sold separately through the Solstice Kickstarter.)

If this sounds like a good thing that you want to see happen and be part of, then register before October 30!

Register and/or see further details at www.rationalistmegameetup.com. Taymon Beal is organizing.

Anything else I should know?

If you have other questions, please feel free to post them in the comments or contact me at rachel@rachelshu.com.