But to apply (5-2), one has to show that prob(A∨B)≠0: if this were the case, then 0=prob(A∨B)des(A∨B)+prob¬(A∨B)des¬(A∨B)=des¬(A∨B), which contradicts the ranking of ¬(A∨B).

des ¬(A∨B)=-2, since it is ranked together with ¬G.
This allows us to compute prob(A∨B) via
prob¬(A∨B)=1/(1-(des¬(A∨B)/des(A∨B)))=1/3. Since A and B are mutually exclusive, and we already know that prob(A)=prob(B) we have prob(A∨B)=2⋅prob(A) and via
1-1/3=1-prob¬(A∨B)=prob(A∨B)=2⋅prob(A) we conclude prob(A)=1/3.

Now to compute des(¬A), use prob(A)=1/(1-(des(A)/des(¬A))).
Solving this for des(¬A) yields des(¬A)=-1/2.

That does not fit the assumed order:

That seems to be an error with the exercise:
If des(¬A)=-prob(A)/prob(¬A)*1 = -prob(A)/(1-prob(A))≤-2, one can conclude prob(A)≥ 2/3. With the same argument prob(B)≥ 2/3. But A and B are supposed to be mutually incompatible, and the sum of their probabilities would exceed 1, which is impossible.

That is a good point. Assuming that problem 11 is really applicable (and it should be, if it says exactly what you have written there), the only cause of trouble I can see is (5-6). So what exactly does (5-6) say? More input would be helpful. Just to be sure, prob means probability, right?
– Jishin NobenJan 13 at 13:09

Then (5-6) is simply not applicable here. By the way, you used (5-6) nevertheless in your reasoning without showing that this assumption is satisfied. There might be an error as well. I will look into it when I get home and update my answer accordingly.
– Jishin NobenJan 13 at 15:51

1

In relation to page 86, I found a mistake in the book. The assumption of (5-6) is not that des X ≠ 1 but that prob X ≠ 1, which makes more sense. This is clear in the 1965 edition of the book.
– martinJan 17 at 19:25