Tuesday, 16 May 2017

I've just been reading Aidan Lyon's fascinating paper, Collective Wisdom. In it, he mentions a result known as the Diversity Prediction Theorem, which is sometimes taken to explain why crowds are wiser, on average, than the individuals who compose them. The theorem was originally proved by Anders Krogh and Jesper Vedelsby, but it has entered the literature on social epistemology through the work of Scott E. Page. In this post, I'll generalize this result.

The Diversity Prediction Theorem concerns a situation in which a number of different individuals estimate a particular quantity -- in the original example, it is the weight of an ox at a local fair. Take the crowd's estimate of the quantity to be the average of the individual estimates. Then the theorem shows that the distance from the crowd's estimate to the true value is less than the average distance from the individual estimates to the true value; and, moreover, the difference between the two is always given by the average distance from the individual estimates to the crowd's estimate (which you might think of as the variance of the individual estimates).

Let's make this precise. Suppose you have a group of $n$ individuals. They each provide an estimate for a real-valued quantity. The $i^\mathrm{th}$ individual gives the prediction $q_i$. The true value of this quantity is $\tau$. And we measure the distance from one estimate of a quantity to another, or to the true value of that quantity, using squared error. Then:

The crowd's prediction of the quantity is $c = \frac{1}{n}\sum^n_{i=1} q_i$.

The crowd's distance from the true quantity is $\mathrm{SqE}(c) = (c-\tau)^2$.

$S_i$'s distance from the true quantity is $\mathrm{SqE}(q_i) = (q_i-\tau)^2$

The average individual distance from the crowd's estimate is $v = \frac{1}{n}\sum^n_{i=1} (q_i - c)^2$.

Given this, we have:

Diversity Prediction Theorem $$\mathrm{SqE}(c) = \frac{1}{n} \sum^n_{i=1} \mathrm{SqE}(q_i) - v$$
The theorem is easy enough to prove. You essentially just follow the algebra. However, following through the proof, you might be forgiven for thinking that the result says more about some quirk of squared error as a measure of distance than about the wisdom of crowds. And of course squared error is just one way of measuring the distance from an estimate of a quantity to the true value of that quantity, or from one estimate of a quantity to another. There are other such distance measures. So the question arises: Does the Diversity Prediction Theorem hold if we replace squared error with one of these alternative measures of distance? In particular, it is natural to take any of the so-called Bregman divergences $\mathfrak{d}$ to be a legitimate measure of distance from one estimate to another. I won't say much about Bregman divergences here, except to give their formal definition. To learn about their properties, have a look here and here. They were introduced by Bregman as a natural generalization of squared error.

Definition (Bregman divergence) A function $\mathfrak{d} : [0, \infty) \times [0, \infty) \rightarrow [0, \infty]$ is a Bregman divergence if there is a continuously differentiable, strictly convex function $\varphi : [0, \infty) \rightarrow [0, \infty)$ such that $$\mathfrak{d}(x, y) = \varphi(x) - \varphi(y) - \varphi'(y)(x-y)$$
Squared error is itself one of the Bregman divergences. It is the one generated by $\varphi(x) = x^2$. But there are many others, each generated by a different function $\varphi$.

Now, suppose we measure distance between estimates using a Bregman divergence $\mathfrak{d}$. Then:

The crowd's prediction of the quantity is $c = \frac{1}{n}\sum^n_{i=1} j_i$.

The crowd's distance from the true quantity is $\mathrm{E}(c) = \mathfrak{d}(c, \tau)$.

Thursday, 11 May 2017

The Reasoning Club
is a network of institutes, centres, departments, and groups addressing
research topics connected to reasoning, inference, and methodology
broadly construed. It issues the monthly gazette The Reasoner. (Earlier editions of the meeting were held in Brussels, Pisa, Kent, and Manchester.)

In this paper, we compare and contrast two methods for the
qualitative revision of (viz., full) beliefs. The first (Bayesian)
method is generated by a simplistic diachronic Lockean thesis requiring coherence with the agent's posterior credences after conditionalization.
The second (Logical) method is the orthodox AGM approach to belief
revision. Our primary aim will be to characterize the ways in which
these two approaches can disagree with each other — especially in the
special case where the agent's belief set is deductively cogent.

Most of the scoring rules that have been discussed and defended in
the literature are not ordinally equivalent, with the consequence that, after the very same outcome has materialized, a forecast X can be evaluated as more accurate than Y according to one model but less accurate
according to another. A question that naturally arises is therefore
which of these models better captures people’s intuitive assessment
of forecasting accuracy. To answer this question, we developed a new
experimental paradigm for eliciting ordinal judgments of accuracy concerning pairs of forecasts for which various combinations of associations/dissociations between the Quadratic, Logarithmic,
and Spherical scoring rules are obtained. We found that, overall, the
Logarithmic model is the best predictor of people’s accuracy
judgments, but also that there are cases in which these judgments —
although they are normatively sound — systematically depart from what
is expected by all the models. These results represent an empirical
evaluation of the descriptive adequacy of the three most popular scoring rules and offer insights for the development of new formal models that might favour a more natural elicitation of truthful and informative beliefs from human forecasters.

The notion of logical consequence has been approached from a variety
of angles. Tarski famously proposed a semantic characterization (in
terms of truth-preservation), but also a structural characterization (in
terms of axiomatic properties including reflexivity, transitivity,
monotonicity, and other features). In recent work, E. Chemla, B. Spector
and I have proposed a characterization of a wider class of consequence
relations than Tarskian relations, which we call "respectable" (Journal of Logic and Computation,
forthcoming). The class also includes non-reflexive and nontransitive
relations, which can be motivated in relation to ordinary reasoning
(such as reasoning with vague predicates, see Zardini 2008, Cobreros et al. 2012, or reasoning with presuppositions, see Strawson 1952, von Fintel 1998, Sharvit 2016). Chemla et al.'s
characterization is partly structural, and partly semantic, however. In
this talk I will present further advances toward a purely structural
characterization of such respectable consequence relations. I will
discuss the significance of this research program toward bringing logic
closer to ordinary reasoning.

Scientific theory choice is often characterised as an Inference to
the Best Explanation (IBE) in which a number of distinct explanatory
virtues are combined and traded off against one another. Furthermore,
the epistemic significance of each explanatory virtue is often seen as
highly case-specific. But are there really so many dimensions to theory
choice? By considering how IBE may be situated in a Bayesian framework, I
propose a more unified picture of the virtues in scientific theory
choice.