Research

FUTURE SUFFERING & MACROSTRATEGY

Space colonization would likely increase rather than decrease total suffering. Because many people care nonetheless about humanity’s spread into the cosmos, we should reduce risks of astronomical future suffering without opposing others’ spacefaring dreams. In general, we recommend to focus on making sure that an intergalactic future will be good if it happens rather than making sure there will be such a future.

AI outcomes where something goes wrong may differ enormously in the amounts of suffering they contain. An approach that tries to avert the worst of those outcomes seems especially promising because it is currently more neglected than classical AI safety efforts which shoot for a highly specific, “best-case” outcome.

The simulation argument suggests a non-trivial chance that most of the copies of ourselves are instantiated in relatively short-lived ancestor simulations run by superintelligent civilizations. If so, when we act to help others in the short run, our good deeds are duplicated many times over. This reasoning dramatically upshifts the relative importance of short-term helping over focusing on the far future.

Discussions about the possible consequences of creating superintelligence have included the possibility of existential risk, usually understood as the risk of human extinction. We argue that suffering risks (s-risks) present comparable severity and probability. Just as with existential risks, s-risks can be caused as well as reduced by superintelligent AI.

Artificial intelligence (AI) will likely transform the world later this century. Whether uncontrolled or controlled AIs would create more suffering in expectation is a question to explore further. Regardless, the field of AI safety and policy seems to be a very important space where altruists can make a positive-sum impact along many dimensions.

This post discusses cause prioritization from the perspective of downside-focused value systems, i.e. views whose primary concern is the reduction of bads such as suffering. According to such value systems, interventions which reduce risks of astronomical suffering are likely more promising than interventions which primarily reduce extinction risks.

Two crucial questions in discussions about the risks of artificial superintelligence are: 1) How much more powerful could an AI become relative to humans, and 2) how easily could superhuman capability be acquired? To answer these questions, this article reviews the literature on human expertise and intelligence and discusses its relevance for AI.

Will we go extinct, or will we succeed in building a flourishing utopia? Discussions about the future trajectory of humanity often center around these two possibilities, which tends to ignore that survival does not always imply utopian outcomes, or that outcomes where humans go extinct could differ tremendously in how much suffering they contain.

Evaluating the effectiveness of our actions, or even just whether they're beneficial or harmful, is very difficult. One way to deal with uncertainty is to focus on actions that likely have positive effects across many scenarios. This approach often amounts to meta-level activities like promoting positive-sum institutions, reflectiveness, and effective altruism in general.

FRI’s research seeks to identify the best intervention(s) for suffering reducers to work on. Rather than continuing our research indefinitely, we will eventually have to focus our efforts on an intervention directly targeted at improving the world. This report outlines plausible candidates for FRI’s “path to impact” and distills some advice on how current movement building efforts can best prepare for them.

COOPERATION AND FORESIGHT

Some decision theorists argue that when playing a prisoner's dilemma-type game against a sufficiently similar opponent, we should cooperate to make it more likely that our opponent also cooperates. This idea, which Hofstadter calls superrationality, has strong implications when combined with the insight from modern physics that we live in a large universe or multiverse of some sort.

When agents of differing values compete, they may often find it mutually advantageous to compromise rather than continuing to engage in zero-sum conflicts. Potential ways of encouraging cooperation include promoting democracy, tolerance and (moral) trade. Because a future without compromise could be many times worse than a future with it, advancing compromise seems an important undertaking.

Fast technological development carries a risk of creating extremely powerful tools, especially AI, before society has a chance to figure out how best to use those tools in positive ways for many value systems. Suffering reducers may want to help mitigate the arms race for AI so that AI developers take fewer risks and have…

There's a decent chance that governments will be the first to build artificial general intelligence (AI). International hostility, especially an AI arms race, could exacerbate risk-taking, hostile motivations, and errors of judgment when creating AI. If so, then international cooperation could be an important factor to consider when evaluating the flow-through effects of charities.

Several arguments support the heuristic that we should help groups holding different value systems from our own when doing so is cheap, unless those groups prove uncooperative to our values. This is true even if we don't directly care at all about other groups' value systems. Exactly how nice to be depends on the particulars of the situation.

Global catastrophic risks – such as biotech disasters or nuclear war – would cause major damage in the short run, but their effects on the long-run trajectory that humanity takes are also significant. In particular, to the extent these disasters increase risks of war, they seem likely to precipitate AI arms races between nations and worsen prospects for compromise.

This article suggests a lower-bound Fermi calculation for the cost-effectiveness of promoting cooperation. The purpose of this exercise is to make our thinking more concrete about how cooperation might reduce suffering and to make its potential more tangible.

Learning is an extremely important activity for altruists. Learning can seem ineffective in the short run, but used properly, it can pay off more than most financial or single-domain-focused investments. It's important for young activists not to neglect learning in order to just "do more to help now."

ETHICS

Most ethical work is done at a low level of formality which can lead to misunderstandings in ethical discussions. In this paper, we use Bayesian inference to introduce a formalization of preference utilitarianism in physical world models. Even though our formalization is not immediately applicable, it is a first step in providing ethical inquiry with a formal basis.

Is the balance of happiness versus suffering in the future net positive or net negative (in expectation)? Is the aggregate happiness and suffering in a group of individuals positive or negative? For such questions to have factual answers that are free from value judgements, happiness and suffering would need to be objectively measurable to a very high degree. However, such a degree of measurability is widely (although not universally) rejected.

Weak negative views in ethics, such as negative-leaning utilitarianism, are said to give more weight to reducing suffering than to promoting happiness. In contrast, non-negative views such as traditional utilitarianism are said to give equal weight to happiness and suffering. However, this way of distinguishing between the views rests on controversial assumptions about the measurability of happiness and suffering.

The number of wild animals vastly exceeds that of animals on factory farms. Therefore, animal advocates should consider focusing their efforts to raise concern about the suffering that occurs in nature. In theory, engineering more humane ecological systems might be valuable. In practice, however, it seems more effective to promote the meme of caring about wild animals to other activists, academics and other sympathetic groups.

“Suffering-focused ethics” is an umbrella term for moral views that place primary or particular importance on the prevention of suffering. Most views that fall into this category are pluralistic in that they hold that other things beside suffering reduction also matter morally. To illustrate the diversity within suffering-focused ethics as well as to present a convincing case for it, this article will introduce four separate motivating intuitions.

What makes an experience valuable or disvaluable? In contrast to hedonism, which holds that pleasure is what is good and pain is what is bad, tranquilism is an “absence of desire” theory that counts pleasure as instrumentally valuable only. According to tranquilism, what matters is whether an experience is free from bothersome components. States of contentment such as flow or meditative tranquility also qualify.

It's a classic debate among utilitarians: Should we care about an organism's happiness and suffering (hedonic wellbeing), or should we ultimately value fulfilling what it wants, whatever that may be (preferences)? This article discusses various intuitions on both sides and explores a hybrid view that gives greater weight to the hedonic subsystems of brains than to other overriding subsystems.

An example of value lexicality is that an outcome with both torture and happiness is bad, regardless of the amount of happiness. Value lexicality is important partly because it can lead to suffering-focused ethics. Key topics that this essay explains include strong versus weak lexicality, value aggregation, views on large numbers and sequence arguments.

Two variables seem particularly important when trying to make informed choices about how to best shape the long-term future: One’s normative goods-to-bads ratio and one’s expected bads-to-goods ratio. This essay discusses how one could measure these variables and investigates associated challenges.

Someone who wants to do good is faced with the question how to prioritize preventing badness vs. bringing about more individuals with good lives. A relevant idea is the ‘Asymmetry,’ which roughly says that it is bad to bring into existence individuals with bad lives but not good to add individuals with good lives. One…

Artificial reinforcement learning (RL), a widely used training method in computer science, has striking parallels to reward and punishment learning in biological brains. Plausible theories of consciousness imply a non-zero probability that RL agents qualify as sentient and deserve our moral consideration, especially as AI research advances and RL agents become more sophisticated.

This essay explains my version of an eliminativist approach to understanding consciousness. It suggests that we stop thinking in terms of "conscious" and "unconscious" and instead look at physical systems for what they are and what they can do. This perspective dissolves some biases in our usual perspective and shows us that the world is…

This piece presents a hypothetical dialogue that explains why instrumental computational processes of a future superintelligence might evoke moral concern. Generally, agent-like components might emerge in many places, including the computing processes of a future civilization. Whether and how much these subroutines matter are questions for future generations to figure out, but it's good to keep an open mind to the possibility that our intuitions about what suffering is may change dramatically.

If we don't understand why we're conscious, how come we're so sure that extremely simple minds are not? I propose to think of consciousness as intrinsic to computation, although different types of computation may have very different types of consciousness – some so alien that we can't imagine them. Since all physical processes are computations,…