I surmise from your comments that you may not be aware that Eliezer's written quite a bit on this matter; http://wiki.lesswrong.com/wiki/Complexityofvalue is a good summary/index ( is one of my favorites). There's a lot of stuff in there that is relevant to your points.

However, you asked me what I think, so here it is...

The wording of your first post in this thread seems telling. You say that "Refusing to become orgasmium is a hedonistic utilitarian mistake, full stop."

Do you want to become orgasmium?

Perhaps you do. In that case, I direct the question to myself, and my answer is no: I don't want to become orgasmium.

That having been established, what could it mean to say that my judgment is a "mistake"? That seems to be a category error. One can't be mistaken in wanting something. One can be mistaken about wanting something ("I thought I wanted X, but upon reflection and consideration of my mental state, it turns out I actually don't want X"), or one can be mistaken about some property of the thing in question, which affects the preference ("I thought I wanted X, but then I found out more about X, and now I don't want X"); but if you're aware of all relevant facts about the way the world is, and you're not mistaken about what your own mental states are, and you still want something... labeling that a "mistake" seems simply meaningless.

On to your analogy:

If someone wants to "keep her body natural", then conditional on that even being a coherent desire[1], what's wrong with it? If it harms other people somehow, then that's a problem... otherwise, I see no issue. I don't think it makes this person "kind of dumb" unless you mean that she's actually got other values that are being harmed by this value, or is being irrational in some other ways; but values in and of themselves cannot be irrational.

[Eliezer] can't stand the idea that scientific discovery is only an instrument to increase happiness, so he makes it a terminal value just because he can.

This construal is incorrect. Say rather: Eliezer does not agree that scientific discovery is only an instrument to increase happiness. Eliezer isn't making scientific discovery a terminal value, it is a terminal value for him. Terminal values are given.

In this discussion we are, of course, ignoring external effects altogether.

Why are we doing that...? If it's only about happiness, then external effects should be irrelevant. You shouldn't need to ignore them; they shouldn't affect your point.

[1]Coherence matters: the difference between your hypothetical hippie and Eliezer the potential-scientific-discoverer is that the hippie, upon reflection, would realize (or so we would like to hope) that "natural" is not a very meaningful category, that her body is almost certainly already "not natural" in at least some important sense, and that "keeping her body natural" is just not a state of affairs that can be described in any consistent and intuitively correct way, much less one that can be implemented. That, if anything, is what makes her preference "dumb". There's no analogous failures of reasoning behind Eliezer's preference to actually discover things instead of just pretend-discovering, or my preference to not become orgasmium.

That having been established, what could it mean to say that my judgment is a "mistake"? That seems to be a category error. One can't be mistaken in wanting something.

I have never used the word "mistake" by itself. I did say that refusing to become orgasmium is a hedonistic utilitarian mistake, which is mathematically true, unless you disagree with me on the definition of "hedonistic utilitarian mistake" (= an action which demonstrably results in less hedonic utility than some other action) or of "orgasmium" (= a state of maximum personal hedonic utility).[1]

I point this out because I think you are quite right: it doesn't make sense to tell somebody that they are mistaken in "wanting" something.

Indeed, I never argued that the dying hippie was mistaken. In fact I made exactly the same point that you're making, when I said:

The argument I was trying to make was, look, this hippie is using some suspect reasoning to make her decisions, and Eliezer's reasoning looks a lot like her's, so we should doubt Eliezer's conclusions. There are two perfectly reasonable ways to refute this argument: you can (1) deny that the hippie's reasoning is suspect, or (2) deny that Eliezer's reasoning is similar to hers.

These are both perfectly fine things to do, since I never elaborated on either point. (You seem to be trying option 1.) My comment can only possibly convince people who feel instinctively that both of these points are true.

All that said, I think that I am meaningfully right — in the sense that, if we debated this forever, we would both end up much closer to my (current) view than to your (current) view. Maybe I'll write an article about this stuff and see if I can make my case more strongly.

[1] Please note that I am ignoring the external effects of becoming orgasmium. If we take those into account, my statement stops being mathematically true.

That article is arguing that it's all right to value things that aren't mental states over a net gain in mental utility.[1] If, for instance, you're given the choice between feeling like you've made lots of scientific discoveries and actually making just a few scientific discoveries, it's reasonable to prefer the latter.[2]

Well, that example doesn't sound all that ridiculous.

But the logic that Eliezer is using is exactly the same logic that drives somebody who's dying of a horrible disease to refuse antibiotics, because she wants to keep her body natural. And this choice is — well, it isn't wrong, choices can't be "wrong" — but it reflects a very fundamental sort of human bias. It's misguided.

And I think that Eliezer's argument is misguided, too. He can't stand the idea that scientific discovery is only an instrument to increase happiness, so he makes it a terminal value just because he can. This is less horrible than the hippie who thinks that maintaining her "naturalness" is more important than avoiding a painful death, but it's not much less dumb.

[1] Or a net gain in "happiness," if we don't mind using that word as a catchall for "whatever it is that makes good mental states good."

[2] In this discussion we are, of course, ignoring external effects altogether. And we're assuming that the person who gets to experience lots of scientific discoveries really is happier than the person who doesn't, otherwise there's nothing to debate. Let me note that in the real world, it is obviously possible to make yourself less happy by taking joy-inducing drugs — for instance if doing so devalues the rest of your life. This fact makes Eliezer's stance seem a lot more reasonable than it actually is.

This may be a minor nit, but... is this forum collectively anti-orgasmium, now?

Because being orgasmium is by definition more pleasant than not being orgasmium. Refusing to become orgasmium is a hedonistic utilitarian mistake, full stop.[1] (Well, that's not actually true, since as a human you can make other people happier, and as orgasmium you presumably cannot. But it is at least on average a mistake to refuse to become orgasmium; I would argue that it is virtually always a mistake.)

As I understand it, Feynman's tentative explanation for why ice is slippery (which he himself qualified with "they say") has since fallen out of favor. This isn't to quibble with Feynman -- if anything, the point he alludes to here and emphasizes in a lot of other works is that science is a continuing process, always updating itself, and individual scientists are not prophets or oracles.

I don't think his explanation for why a chair pushes back on your hand is quite right, either. I've mostly been told that material solidness comes from the Pauli exclusion principle, not electrostatic repulsion.

I don't know quantum mechanics, so I don't have a good perspective on the problem, but the electrostatic explanation has always seemed lacking to me. The electric charge in a neutral atom is fairly well-approximated by a symmetric sphere of negative charge with a bunch of positive charge at the center, so two atoms shouldn't experience much electrostatic repulsion until their electron clouds overlap. At which point [I've heard] the PEP should dominate the electrostatic force.