A famous instance of "irrationality" reconsidered

The usual presentation

There's a rather famous demonstration of human irrationality
that goes like this. Suppose there are four index card on a
table in front of you, with the following things drawn on them:

On card 1: a square.

On card 2: a circle.

On card 3: the letter A.

On card 4: the number 42.

You're told that every card in fact has a letter or number
on one side, and a square or a circle on the other. You're
now asked to consider the statement "Every card with a letter
on one side has a square on the other.". Obviously you can't
tell whether it's true or not without more information than
you can see on the table; so you're allowed to turn some cards
over and see what's on their other sides. You should turn over
as few cards as possible. Which ones do you turn over?

If you haven't seen this before, you will probably find it
illuminating to consider it now. (Although, having been warned
that people tend to get it wrong, you'll be on your guard.)
The only real purpose of this paragraph is to make it slightly
less likely that you'll see the answer before even thinking
about the question; it would be a shame if you did that. If
you're still reading then stop doing so, decide on your answer
to the question, and then go on to the next paragraph. There's
nothing interesting in this one. Honestly. Nothing at all.
Go on, shoo. Thank you.

The usual response

OK. Almost everyone turns over card 3, with the letter A
on it. Obviously that's correct. Many people also turn card 1,
with the square on it. Very few turn cards 2 (circle) and 4
(number).

In fact, the two cards you need to turn to verify or refute
the claim are 2 (circle) and 3 (letter). You don't need card 1;
if you found a number on the other side of it, that wouldn't
say anything about the truth of "Every card with a letter
has a square". And you do need card 2; if you found
a letter on the other side of it, that would kill the hypothesis
at once.

Most people (about 90%, in fact) behave
irrationally when presented with this test.

A different version

So far, so good.
However, recently I was reading a book about irrationality which
happened to give a slightly different version of the experiment.
In this version, instead of shapes drawn on the cards, each card
has

On one side: a psychological description of a person

On the other: a drawing of a human face made by that
same person

and the statement we're asked to consider is "People with
delusions of persecution tend to draw faces with unusually
large eyes". The rest of the setup is as before: so we have
two cards showing psychological descriptions (one with
delusions of persecution, one without) and two showing
face drawings (one with large eyes, one without).

The book then described the usual findings, and made the
usual remarks about irrationality: turning card 1 is irrational,
not turning card 3 is irrational, etc.

What's wrong with this version

But, with this form of the experiment, all
the cards are worth looking at! So, not turning card 2 is
still irrational, but turning card 1 isn't irrational at all.
So is not turning card 4!

Here's why. What does "tend to" mean here? Presumably that
people with delusions of persecution are more likely
to draw faces with large eyes than people without. To tell
whether this claim is true or not, we need to know how likely
both groups are to draw large-eyed faces.

An example

Think about it like this. Instead of having just one card
of each type, you have piles containing 1000 of each
of the four kinds of card. You're told that the cards
came from a random sample of the population. (So we're
obviously interpreting "delusions of persecution" rather
broadly.) Suppose that, like a good rational
person, you turn over the allegedly relevant cards. You find
that

Of the "people with delusions of persecution" pile, exactly
half have drawn large-eyed faces.

Of the "people who have drawn small-eyed faces" pile, exactly
half have delusions of persecution.

So, now we have: 500 delusions/large-eyed;
1000 delusions/small-eyed; 500 no-delusions/small-eyed.
What do you conclude from this?

The answer, provided you're really rational,
is that it depends on what the other cards turn up. Here
are two possibilities.

Of the "people without delusions" pile, all have drawn
small-eyed faces. Of the "people who've drawn large-eyed
faces" pile, all have delusions of persecution.

Of the "people without delusions" pile, all have drawn
large-eyed faces. Of the "people who've drawn large-eyed
faces" pile, none have delusions of persecution.

In the first case, our tallies now come to:
1500 delusions/large-eyed; 1000 delusions/small-eyed;
0 no-delusions/large-eyed; 1500 no-delusions/small-eyed.
So, large-eyed drawings are much more common
among those with delusions of persecution (60% versus 0%);
and if you encounter a large-eyed drawing then you can be
absolutely certain that it came from someone with
delusions of persecution. I'd certainly describe that as
"people with d.o.p. tend to draw large-eyed faces".

In the second case, our tallies now come to:
500 delusions/large-eyed; 1000 delusions/small-eyed;
2000 no-delusions/large-eyed; 500 no-delusions/small-eyed.
In this case, large-eyed drawings are much less common
among those with delusions of persecution than among
those without (33% versus 80%); delusions of persecution
are much less common among those who draw large-eyed faces
than among those who don't (20% versus 67%).
I'd certainly never describe that as
"people with d.o.p. tend to draw large-eyed faces".

In conclusion

The answers people usually give aren't any less
irrational when applied to this modified version of
the problem, but they're irrational in a different
way. The only real conclusion one can draw is that
the authors of the book I found this in (who
didn't notice this, and made some very bogus claims
as a result) aren't too rational themselves.