ricraz

I think that’s true in the US, but not in most of Europe. E.g. in Switzerland a first year PhD student gets paid $40000 a year WITHOUT doing any teaching, and more if they teach. That’s unusually generous, but I think the setup isn’t uncommon.

Expensive in terms of time, perhaps, but almost all good universities in the US and continental Europe provide decent salaries to PhD students. UK is a bit more haphazard but it’s still very rare for UK PhDs to actually pay to be there, especially in technical fields.

A realistic example which matches this payoff matrix very well is when everyone is using the same means of transport, and so has to wait for the last to arrive before they can leave. A more speculative one: trying to create an environment of group emotional openness and vulnerability, where one person being sarcastic and prickly can ruin it for everyone.

Thanks for the excellent post, Jacob. I think you might be placing too much emphasis on learning algorithms as opposed to knowledge representations, though. It seems very likely to me that at least one theoretical breakthrough in knowledge representation will be required to make significant progress (for one argument along these lines, see Pearl 2018). Even if it turns out that the brain implements backpropagation, that breakthrough will still be a bottleneck. In biological terms, I’m thinking of the knowledge representations as analogous to innate aspects of cognition impressed upon us by evolution, and learning algorithms as what an individual human uses to learn from their experiences.

Two examples which suggest that the former are more important than the latter. The first is the “poverty of stimulus” argument in linguistics: that children simply don’t hear enough words to infer language from first principles. This suggests that ingrained grammatical instincts are doing most of the work in narrowing down what the sentences they hear mean. Even if we knew that the kids were doing backpropagation whenever they heard new sentences, that doesn’t tell us much about how that grammatical knowledge works, because you can do backpropagation on lots of different things. (You know more psycholinguistics than I do, though, so let me know if I’m misrepresenting anything).

Second example: Hinton argues in this talk that CNNs don’t create representations of three-dimensional objects from two-dimensional pictures in the same way as the human brain does; that’s why he invented capsule networks, which (he claims) do use such representations. Both capsules and CNNs use backpropagation, but the architecture of capsules is meant to be an extra “secret sauce”. Seeing whether they end up working well on vision tasks will be quite interesting, because vision is better-understood and easier than abstract thought (for example, it’s very easy to theoretically specify how to translate between any two visual perspectives, it’s just a matrix multiplication).

Lastly, as a previous commentator pointed out, it’s not backpropagation but rather gradient descent which seems like the important factor. More specifically, recent research suggests that Stochastic Gradient Descent leads to particularly good outcomes, for interesting theoretical reasons (see Zhang 2017 and this blog post by Huzcar). Since the brain does online learning, if it’s doing gradient descent then it’s doing a variant of SGD. I discuss why SGD works well in more detail in the first section of this blog post.

I’d also add to 2: an intuitive explanation of eigenvectors and their properties in general. They seem to pop up everywhere in Linear Algebra and when then do, people gesture towards a set of intuitions about them that I haven’t managed to pick up.

Nice work! I would be happy to work through some of the MIRI research guide with you. I’m particularly interested in the books on Probabilistic Inference, Provability Logic, Category Theory, and Topology. PM me if you’re planning to cover any of them any time soon.

What you’ve defined above is just morality in general: basically any moral theory can be expressed as a “nonlinear” function of some properties of individuals plus some properties of the world. For example, in deontology one nonlinearity is the fact that murdering someone is nearly-infinitely bad.

The key thing that utilitarianism does is claim that the function we should be maximising is roughly linear in well-being; my main point is clarifying that it shouldn’t be linear in “utility” (in either a desire or an economic sense).

Okay, but now you’ve basically defined “increasing utility” out of existence? If voting power is roughly normalised, then it’s roughly equally important to save the life of an immensely happy, satisfied teenager with a bright future, and a nearly-suicidal retiree who’s going to die soon anyway, as long as staying alive is the strongest relevant desire for both. In fact, it’s even worse: assuming the teenager has a strong unreciprocated crush, then I can construct situations where only 1⁄2 of their voting power will go towards saving themselves, so their life is effectively half as valuable as a loner.

That interpersonal utility comparisons are impossible in VNM utility is not some incidental fact, it is an inevitable consequence of the formalism’s assumptions.

Any consequence of a formalism’s assumptions is inevitable, so I don’t see what you mean. This happens to be an inevitable consequence which you can easily change just by adding a normalisation assumption. The wikipedia page for social choice theory is all about how social choice theorists compare utilities interpersonally—and yes, Amartya Sen did win a Nobel prize for related work. Mostly they use partial comparison, but there have been definitions of total comparison which aren’t “nonsensical”.

The first question in any such scenario has to be: “Where are these numbers coming from, and what do they mean?” If we can’t answer it in a rigorous way, then the discussion is moot.

I agree that if you’re trying to formulate a moral theory, then you need to come up with such numbers. My point is that, once you have come up with your numbers, then you need to solve the issue that I present. You may not think this is useful, but there are plenty of people who believe in desire utilitarianism; this is aimed at them.

Okay, so it’s a mistake because it’s simply undefined mathematical nonsense. Now let me define a new form of utility which differs from economic utility only by the fact that interpersonal comparisons are allowed, and occur in whatever way you think is most reasonable. How do you feel about using this new form of utility to draw moral conclusions? I think my arguments are relevant to that question.

Re mistake one:

I’m not assuming that the difference within any pair of world states which differ in a certain way is constant any more than an economist is when they say “let X be the utility that is gained from consuming one unit of good Y”. Both are approximations, but both are useful approximations.

If you’d prefer, I can formalise the situation more precisely terms of world-states. For each world-state, each member of the family assigns it utility equal to the number of family members still alive. So if they all die, that’s 0. If they all survive, that’s 5, and then the total utility from all of them is 25 (assuming we’re working in my “new form of utility” from above, where we can do interpersonal addition).

Meanwhile each loner assigns 1 utility to worlds in which they survive, and 0 otherwise. So now, if we think that maximising utility is moral, we’d say it’s more moral to kill 24 loners than one family of 5, even though each individual values their own life equally. I think that this conclusion is unacceptable, and so it is a reductio of the idea that we should maximise any quantity similar to economic utility.

It makes sense, but I find it very counterintuitive, partly because it’s not obvious to me whether the concept of “measuring desire” makes sense. Here are two ways that I might measure whether people have a stronger desire for A or B:

1) I hook up a brainwave reader to each person, and see how strongly/​emotional/​determined they feel about outcome A vs outcome B.

2) I ask each person whether they would swap outcome A for outcome B.

In the first case, it’s plausible to me that each person’s emotions are basically maxed out at the thought of either their own death, or their family’s death (since we know people are very bad at having emotions which scale appropriately with numbers). So then X = Y, and you save the 20 people.

In the second case, assume that each person involved desires to continue living, personally, at about the same strength S. But then you ask each member of the family whether they’d swap that for someone else in their family surviving, and they’d say yes. So therefore each member of the family has total desire > 5S that their family survives, whereas each loner has desire S to survive themselves, and so you save the family.

Which one is closer to your view of measuring desire? 2 seems more intuitive to me, because it matches the decisions we’d actually make, but then I find the conclusion that it’s more moral to save the family very strange.

I think your first objection is technically correct, but irrelevant to the point I was making; and your second objection is entirely consistent with my conclusion.

On “mistake one”: I am using “I assign utility U to living a happy life” as a shorthand for something like “In general, the difference in utilities I assign between worlds in which I am happily alive, and worlds in which I am not, is U, all else being equal.” This is a perfectly normal sort of phrasing; for example, the wikipedia page on utility says that it ″represents satisfaction experienced by the consumer from a good.″ Do you object to this and any other talk of utility which isn’t phrased in terms of world-states?

On “mistake two”: I should have mentioned (and will edit to add) that economists don’t endorse interpersonal comparison of economic utility. But I’m not endorsing it either: I’m explicitly flagging it as a philosophical mistake, and explaining one reason why attempts to do so are misguided. This is more useful than simply saying that it’s ill-defined, because the latter leaves us to wonder why we can’t just construct a new way to compare utilities between people—for example, in another comment cousin_it is basically arguing for economic-style utility + interpersonal comparison.

I feel like “negotiation” is very handwavey. Can you explain what that looks like in a simple zero-sum situation? For example, suppose that you can either save the lives of the family of 5 that I described above, or else save 20 loners who have no strong relationships; assume every individual has an equally strong desire to remain alive. How do we actually aggregate all their desires, without the problem of double counting?

The reason I think hedonic views are important is because desires can be arbitrarily weird. I don’t want to endorse as moral a parent who raises their child with only one overwhelmingly strong desire—that the sky remains blue. Is that child’s well-being therefore much higher than anyone else’s, since everyone else has had some of their desires thwarted? More generally, I don’t think a “desire” is a particularly well-defined concept, and wouldn’t want it to be my main moral foundation.

The thing I usually do, when asked to elicit a probability, is report a probability (usually 2 sig figs) and then also a subjective sense of how easy it would be to shift that probability by giving me more evidence /​ allowing me more time to think.

What is the correct technical way to summarise the latter quantity (ease of shifting), in an idealised setting?