If you add an opponent with opposite values from you into a decision theory problem, you are adding a component that is approximately as hard to compute the behavior of as it is to solve the problem as a whole (since it depends on a computation by the opponent which is of roughly the same form).

Wait—may I ask for some remedial instruction on why zero-sum game-theory isn’t a strict subset of decision theory, just with pessimistic assumptions about some of the variables? I can turn any decision problem into a game-theory problem by adding an opponent with opposite values from me, right? I strongly suspect that _both_ are NP generally, and I can give many specific examples of both that can be solved in polynomial time.

I’m confused by your “cognitive modes” idea as well—it may be true, but I don’t see how it follows, if decision theory is more difficult (NP) than game theory (P), that

In decision theory, you plan with no regard to any opponents interfering with your plans, allowing you to plan on arbitrarily long time scales. In zero-sum game theory, you plan on the assumption that your opponent will interfere with your plans (your ∃s are interleaved with your opponent’s ∀s), so you can only plan as far as your opponent lacks the ability to interfere with these plans

This seems to lean in the opposite direction—in decision theory, you can easily answer as far out as you like, in zero-sum game-theory you’re far more limited. It is also more about certainty of game state than about motivation or existence of opponent. A plan with no opponents but also a bunch of randomness (or unknown starting state; same thing for these purposes) is MORE impossible to follow than a plan with known state and an active opponent.

I do somewhat agree with your final paragraph, but I don’t agree with your premise nor reasoning to get there.

One polite way to respond to people using words you prefer they not use is “[Word] upsets me for [Reason], can you use [Replacement Word] instead?” If they can’t (because they’re not a native English speaker, or they have a linguistic disability, or they are chronically sleep deprived, to name just three of the reasons that word replacement can be impossible), then you have to judge how important not being around people who use Word is for you.

You could also consider asking what they mean if you don’t know what they mean. My rough sense is something like “a cluster of chemicals the central examples of which require industrial manufacturing processes to create, did not exist before the 20th century, are not part of any culture’s traditional way of doing things, could not be manufactured in a home kitchen, and bear little resemblance to petroleum, corn, or soybeans in spite of being derived from them.”

Yeah, I don’t fully endorse the linked Tumblr post; in particular there’s certainly ways to resolve these conflicts that aren’t “abdicate the terminology yourself”. But some of it is highly relevant and well said.

A thing doesn’t have to be a natural category for people to want to talk about it and have a legitimate interest in talking about it.

I disagree with your second point and think you’re missing mine. If you don’t want to talk to someone, don’t talk to them. You don’t have to be cruel, and your desire to be cruel doesn’t make it reasonable.

If the orthogonality thesis is incorrect and the reason (some) humans are friendly is because they are not intelligent enough (as Nick Land argue here), then friendly AGIs would be impssible. I think the arguments for this position are really badly argued, but this is still a good reductio of gworley’s philosophical conservatism.

I feel like the example for “loading definitions” does, in fact, strike a word from my vocabulary without suitable replacement. I would like a word for “the aspects of masculinity that are bad”; in order to prevent the conversation turning into a bunch of complaints about my use of a particular term, I instead have to just say “masculinity.” I do not want to use “masculinity” to mean “the aspects of masculinity that are bad.” I would like to distinguish between those two things.

(While I have no moderation power, I would personally really prefer that this conversation not turn into a conversation about the merits of that particular term.)

One possible problem with the scheme is that level N assistants will have access to a lot of power, in the form of level N-1 assistants. If they have even a little desire for personal happiness, they might be tempted to trade. Such trade can make the whole bureaucracy corrupt or even harmful.

I really like when people put effort into providing alternatives-to/​critiques-of how I work, so thanks for this.

It doesn’t mean I can always reply promptly, alas. Right now I’m in the process of taking all my finals (so I can get a visa and move to the Bay), and this will continue for a few weeks :(. Oli’s just coming back from the same, while Ray’s holding the fort.

It’s also regularly the case that doing detailed public write-ups of considerations around a decision/​approach isn’t the right use of effort relative to just *building the product*, and that applies to things like detailed comments on this too. So far with LW I and Oli have spent much less time writing about where we’re going than just going there, which I think has worked well. (Ray has actually done a lot of writing of his thinking, somehow. Well done Ray.)

Then again, write-ups are a big part of building moderation, so I’ll see in a few weeks when I come to this.

I tried diving into All of Statistics but I found it to be way too concise. I didn’t get past Chapter 3. In particular Chapter 3 felt like a list of distributions and some arbitrary properties. It felt like I wasn’t really getting an intuition for what these distributions represent or why these properties are interesting. In the end I dropped the book because I felt like a wasn’t really learning anything.

My negative experience with this book is likely a result of me having no previous experience with statitistics.

The technology example reminded me of Darcey Riley’s discussion of shattered dichotomies: sometimes people think of things as being either X or Y, and then learn an argument for why this dichotomy doesn’t make sense. As a result, they might reject the dichotomy entirely, keep it but conclude that “everything is X” or “everything is Y”, or acknowledge the argument but find the dichotomy useful regardless.

For instance, back in high school philosophy class, I used to argue that “all people are selfish”. If you’re hurt, and I go to help you, it’s not because I’m altruistic. It’s because the sight of you in pain causes me to feel pain, and I, selfishly, want to relieve my own pain (or I want to avoid the guilt I’d feel for not helping). Similarly, if I give you a gift, it’s not because I’m altruistic; it’s because I selfishly want the pleasure and satisfaction that comes from gift-giving.

In high school, I thought this was a great argument. As an adult, I roll my eyes. It’s not that the argument is wrong, per se; based on the definition of “selfish”, it really is possible to classify all actions as selfish. I just don’t think it’s useful. Our folk concepts of “selfless” and “selfish” might be fuzzy and imprecise (as all concepts are), but they help us navigate a complicated world. When you realize that your friend Mike is selfish, you might decide to hang out with him less, or to avoid doing him favors because you know they won’t be reciprocated. And when you’re deciding whether to give your friend Steve a ride to the airport, you might agree to do it, because you don’t want him to think you’re selfish.

(Though maybe the shattering of this dichotomy can be useful for some people! If someone suffers from scrupulosity, and is wracked with unnecessary guilt that they’ve chosen the selfish option too often, then completely removing the distinction between “selfish” and “selfless” could be exactly what they need.)

Another shattered dichotomy I’ve encountered is the people who argue “A city is just as natural as a pristine forest, because cities were made by humans, and humans are part of nature. A city is just as much a natural structure as a bird’s nest or an anthill.” I neither agree nor disagree with this argument; it’s really just a matter of what you want the concepts to mean. And that, in turn, will depend on what you’re using them for. A lot of people (myself included) find natural landscapes beautiful, but also find industrial complexes ugly. And, while I’ll always probably find refineries ugly on a visceral level, this argument helps me appreciate them as part of the ecosystem of human activity, which I do in fact find beautiful. So in that particular instance, I appreciate the shattering of the dichotomy. But when someone says “pollution in the Shenandoah river isn’t a big deal, because industrial waste is just as natural as fish poop”, then I’m going to object, because they’re relying on the standard inference “natural ⇒ harmless”, and they’re trying to get you to classify pollution as natural so you’ll think of it as harmless as well.

P(earth is flat)=0.6 isn’t a garbage prediction, since it lets people update to something reasonable after seeing the appropriate evidence. It doesn’t incorporate all the evidence, but that’s a prior for you.

I think God and Monty Hall are both interesting examples, In particular Monty Hall is interesting because so many professional mathematicians got the wrong answer for it, and God is interesting because people disagree as to who the relevant experts are, as well as what epistemological framework is appropriate for evaluating such a proposition. I don’t think I can give you a good answer to either of them (and just to be clear I never said that I agreed with Hanson’s point of view).

Maybe you’re right that xzxq is not relevant to Hanson’s paper.

Regarding weighting, Hanson’s paper doesn’t talk about averaging at all so it doesn’t make sense to ask whether the averaging that it talks about is weighted. But the idea that all agents would update to a (weighted) species-average belief is an obvious candidate for an explanation for why their posteriors would agree. I realize my previous comments may have obscured this distinction, sorry about that.

The proposition that we should be able to reason about priors of other agents is surely not contentious. The proposition that if I learn something new about my creation, I should update on that information, is also surely not contentious, although there might be some disagreement what that update looks like.

The form is the interesting thing here. By arguing for the common prior assumption, RH is giving an argument in favor of a form of modest epistemology, which Eliezer has recently written so much against.

In the case of genetics, if I learned that I’m genetically predisposed to being optimistic, then I would update my beliefs the same way I would update them if I had performed a calibration and found my estimates consistently too high. That is unless I’ve performed calibrations in the past and know myself to be well calibrated. In that case the genetic predisposition isn’t giving me any new information—I’ve already corrected for it. This, again, surely isn’t contentious.

In Eliezer’s view, because there are no universally compelling arguments and recursive justification has to hit bottom, you don’t give up your prior just because you see that there was bias in the process which created it—nothing can be totally justified by an infinite chain of unbiased steps anyway. This means, concretely, that you shouldn’t automatically take the “outside view” on the beliefs you have which others are most likely to consider crazy; their disbelief is little evidence, if you have a strong inside view reason why you can know better than them.

In RH’s view, honest truth-seeking agents with common priors should not knowingly disagree (citation: are disagreements honest?). Since a failure of the common prior assumption entails disagreement about the origin of priors (thanks to the pre-rationality condition), and RH thinks disagreement about the origin of priors should rarely be relevant for disagreements about humans, RH thinks honest truth-seeking humans should not knowingly disagree.

I take it RH thinks some averaging should happen somewhere as a result of that, though I am not entirely sure. This would contradict Eliezer’s view.

Although I have no idea what this has to do with “species average”. Yes, I have no reason to believe that my priors are better than everybody else’s, but I also have no reason to believe that the “species average” is better than my current prior (there is also the problem that “species” is an arbitrarily chosen category).

The wording in the paper makes me think RH was intending “species” as a line beyond which the argument might fail, not one he’s necessarily going to draw (IE, he might concede that his argument can’t support a common prior assumption with aliens, but he might not concede that).

I think he does take this to be reason to believe the species average is better than your current prior, to the extent they differ.

But aside from that, I struggle to understand what, in simple terms, is being disagreed about here.

I see several large disagreements.

Is the pre-rationality condition a true constraint on rationality? RH finds it plausible; WD does not. I am conflicted.

If the pre-rationality argument makes sense for common probabilities, does it then make sense for utilities? WD thinks so; RH thinks not.

Does the pre-prior formalism make sense at all? Can rationality conditions stated with use of pre-priors have any force for ordinary agents who do not reason using pre-priors? I think not, though I think there is perhaps something to be salvaged out of it.

One pattern I’m noticing is because of the fact that because of the relative comparative advantage of citizenship in other countries, and the relative difficulty of attaining permanent residency in the United States, the communities of rationalists abroad are more stable over time because of the practical difficulty of convincing people to move to the United States. For example, having post-secondary education that is more subsidized not just in undergrad but in graduate studies as well in countries aside from the United States keeps non-American rationalists in their home countries until their mid-to-late twenties. That’s young enough I know rationalists who musing about moving to Berkeley to work on AI alignment or another community project someday, but I also know a lot of rationalists who have set down roots where they are by then, and aren’t inclined to move. Another thing if is a rationalist doesn’t have a university degree or highly in-demand skills (e.g., STEM) for big corporations, it’s difficult enough to get health insurance and visas for a lot of rationalists emigrating to the United States it doesn’t make sense to try. This first post I wrote is intended to be part of a sequence to be focused on finding solutions to problems apparently common to community organization both in Berkeley and elsewhere. I tried to end this post on a positive tone, intending to build up to optimism in the next one, with marked examples of recent success among rationalists around the world. Vancouver has a couple local EA organizations, and strategies we’ve implemented locally have dramatically increased our rate of meetups, doubled the number of rationalist houses (from 2 to 4, but in 6 months that is still significant. The same has happened in Motreal the last few months. Jan Kulveit is another rationalist who has commented on this post as well who has had reported a lot of success with local community organization in the Czech Republic, as has Toon Alfrink from the Netherlands. If we can integrate what worked for us into a single strategy for mobilizing resources in local rationality communities it could be excellent.

The good news is I think the possible global failure mode I pointed out of the rationality community being too heavily concentrated in a single geographic hub which may then collapse appears quite unlikely to come about for the foreseeable future. So while the experience of the NYC rationalist community may be similar to a lot of rationality communities, it’s not universal. I don’t know if that means much given the NYC community has lost so many people, but hopefully if something comes out of people sharing solutions we can find a way to help the NYC community as well.