Monday, April 13, 2009

Why I am not a Unicornian

The title of this post is a play on Bertrand Russel's classic essay, "Why I am not a Christian." But the content is modeled after Chapter 7 of David Deutsch's highly under-appreciated book, "The Fabric of Reality". Following Deutsch (who himself followed John Worrall) the essay takes the form of a dialog between me (Ron) and a Unicornian, one who believes in Unicorns.

A disclaimer: this piece is a poor paraphrase of Deutsch's argument. If it weren't protected by copyright, I would just cut-and-paste the entirety of chapter 7 of TFOR here and leave it at that. I asked Deutsch once to put Chapter 7 on the web as a service to humanity but he declined. I cannot hope to reproduce the clarity and completeness of Deutch's argument (which is really Popper's argument), though I'll certainly give it my best shot.

I will lift one passage verbatim from Deutsch as fair-use, which is his statement of the thesis, so at least I won't be able to screw that up:

Science seeks better explanations. A scientific explanation accounts for our observations by postulating something about what reality is like and how it works. We deem an explanation to be better if it leaves fewer loose ends (such as entities whose properties are themselves unexplained), requires fewer and simpler postulates, is more general, meshes more easily with good explanations in other fields and so on. But why should a better explanation be what we always assume it to be in practice, namely the token of a truer theory? Why, for that matter, should a downright bad explanation (one that has none of the above attributes, say) necessarily be false? There is indeed no logically necessary connection between truth and explanatory power. A bad explanation ... may be true. Even the best and truest available theory may make a false prediction in particular cases, and those might be the very cases in which we rely on the theory. No valid form of reasoning can logically rule out such possibilities, or even prove them unlikely [emphasis added]. ... [Nonetheless] I believe that we can justify our expectation that [good scientific theories make accurate predictions].

So here's my humble attempt at channeling Deutsch. But seriously, get a copy of the book and read the original. It's available in paperback.

The setting is, as with Deutsch and Worrall, the top of the Eiffel tower. The characters are Ron and a Unicornian. Ron has just eaten some bad mussels and come down with a very serious case of food poisoning.

---

Ron: I'm not feeling so good. I think I'd better go to the hospital to be checked out.

Unicornian: You know, the line at the elevator is awfully long. Why don't we see if we can find a unicorn instead? Their horns have curative powers you know.

R: If it's all the same to you, I think I'll take my chances with the elevator line.

U: You don't believe in unicorns, do you.

R: That's right, I am an a-unicornist.

U: I feel sorry for you. You are going to risk dying from salmonella just because you stubbornly cling to your faith.

R: I don't really think of a-unicornism as faith. My non-belief in unicorns follows from my belief in science, which is to say, in my belief that experiment and rational argument are the best paths to Truth.

U: But science can't disprove he existence of unicorns.

R: That's right, it can't. Science can't disprove the existence of anything. What science can do is to show, in a philosophically justifiable way, that certain things are extremely unlikely.

U: I don't see how that's possible. After all, any argument you could possibly make has to start with some unprovable and untestable assumptions. Even the belief that experiment is a reliable guide to Truth (with a capital T) is unprovable and untestable. So I don't see how one argument that rests on unprovable and untestable assumptions can be any more philosophically justifiable than any other argument that rests on unprovable and untestable assumptions. You have your faith, I have mine, and there's no way to determine who is right and who is wrong (except that I know that you are wrong).

R: No, that's not true. But explaining why is rather subtle and complicated.

U: The elevator line has hardly moved at all since we started this little chat, so it would seem that as long as you cling to your misguided beliefs about hospitals and modern medicine we have time.

R: OK, let's see how far we get. Have you read David Deutsch's book, "The Fabric of Reality"?

U: Nope, never even heard of it.

R: If you really want to understand this stuff you really ought to read it, especially chapter 7. But I'll do my best to explain it. I'm in a fairly dire situation here. Let us assume for the sake of argument that whatever it is that can save my life, be it unicorns or antibiotics, is only available down there at ground level. After all, if I'm not mistaken, unicorns have many magical properties, but they can't fly.

U: No, they can't fly, but they can ride elevators. There might be one up here. We really ought to look.

R: But then we'd risk losing our place in line.

U: Hm, good point. Maybe I should go looking and you could stay here?

R: If you don't mind I'd rather you stay. If I suddenly collapse I'd just as soon have the company.

U: It's your funeral.

R: Not yet, I hope. But there is another possible way of getting to the ground. We could jump over the side.

U: That seems like a bad plan.

R: Indeed it does. But *why* does it seem like a bad plan?

U: Duh, because you'd fall to your death.

R: And how do you know that? Maybe a unicorn would save me.

U: Dude, unicorns don't work that way.

R: I thought unicorns were magic.

U: They are, but their magic is limited. Their horns have curative powers (which is the reason I think you should seek one out) and they can, of course, become invisible (which is what makes them challenging to locate). But they can't fly, and they don't make particularly good backstops for a hard fall because of their horn and all.

R: And how do you know all that?

U: Because the Unicorn Bible says so.

R: And how do you know that what the Unicorn Bible says is true?

U: Because the Unicorn Bible is the Revealed Word of Uni, the omniscient and infallible unicorn who created the universe.

R: I see. Maybe if I jump over the side he'll break my fall?

U: Alas, no, that is not possible. Because while Uni is omniscient and infallible, he is alas not omnipotent.

R: How inconvenient. Though that does have the beneficial side-effect that you don't have to worry about the problem of theodicy.

U: Indeed not. All evil is the work of men, not unicorns.

R: OK, but even if Uni won't save me, something might. You can't prove that if I jumped over the side that I would fall to my death.

U: Sure I can. There are no exceptions to the law of gravity.

R: How do you know?

U: Because none has ever been observed.

R: But I've never seen a unicorn. You've never seen a unicorn. No one alive has ever seen a unicorn...

U: Duh, because they're invisible. I explained that already.

R: ...and yet you believe in unicorns. So just because something hasn't been observed doesn't prove it doesn't exist. How can we be sure that there isn't an invisible construction elevator just over there that would lower me gently to the ground?

U: Because that's ridiculous.

R: Is it? It seems no more ridiculous to me than the idea of invisible one-horned horses roaming the earth. Why should your intuitions about what is and is not ridiculous be so much more reliable than mine?

U: Because my intuitions are informed by the Revealed Word of Uni and yours are not.

R: That's not true. I've read the Unicorn Bible. I just happen not to believe that it is the Revealed Word of Uni.

U: "The fool sayeth in his heart: there are no unicorns."

R: Are you familiar with the concept of a circular argument?

U: Fair enough. But it seems to me that we have gotten exactly nowhere. It still seems to me that you base your beliefs on a set of unprovable untestable assumptions.

U: Not so. No one has (as far as I know) ever conducted the experiment of jumping off the Eiffel tower, so you have no basis for believing that you would fall to your death. Sure, you may have seen lots of other things fall under the force of gravity, but to extrapolate from those experiments to this situation requires a leap of faith.

R: No, it doesn't.

U: Why not?

R: Because of the explanatory power of scientific theories.

U: What does that have to do with anything?

R: Scientific theories have value for two reasons. First, they allow you to make predictions about the world which can serve as useful guides for action. (For example, a fairly complex array of scientific theories, ranging from gravity to the biology, is currently guiding me to wait for the elevator so I can go to the hospital rather than leaping over the side.) Second, and more important, they provide explanations of how the universe *really* works (for some value of "really"). The quality of a scientific theory is judged not merely by how well its predictions match experimental observation, but also by the quality of the explanatory mechanism that it provides. For example, the problem with your theory of using unicorn horn to treat food poisoning is not that it is wrong, it is that it provides no explanatory power. It merely stipulates that unicorn horns are an antidote against poison, it doesn't say anything about the mechanism by which these curative powers arise.

U: Sure it does. Unicorns are magic.

R: But magic is not an explanation. Magic -- real magic, not the "fake" magic practiced by professional magicians -- by definition cannot be explained. That's what makes it magic. As soon as you explain it, it ceases to be magic and becomes science. Calling something "magic" offers no more of an explanation than saying that you simply don't know.

U: OK, but you still have to assume with no proof that "explanatory power" is a reliable guide to Truth. So your beliefs are still, at root, based on faith.

R: No, I don't. This is the really cool thing about science. It just turns out that theories with better explanatory power invariably make better, more reliable, more accurate, more precise predictions about the world than theories with poorer explanatory power. Why this should be is indeed a great mystery, but it is an incontrovertible fact.

U: Nothing is incontrovertible.

R: To controvert this you would have to resort to quite the extremes of intellectual dishonesty (some would say you'd have to be delusional or even clinically insane, though I'm not sure I'd go quite that far). The evidence that theories with better explanatory power make better predictions (mind you this is a very different thing from saying that these theories are actually true) pervades every aspect of modern life. Our modern technological society is utterly dependent on being able to make all manner of accurate predictions about what will happen if things like cars and cell phones and bridges are built in certain ways and not others. My life currently depends on my deciding whether or not to jump down to the ground or wait for the elevator. Historically, people who base such decisions on scientific theories that both 1) agree with experimental evidence and 2) have explanatory power have vastly better track records (in terms of the objective material outcomes of their decisions) than people who base their decisions on anything else. And over the centuries people have tried lots and lots of different things, prayed to lots and lots of different gods, listened to countless prophets, tried to gleaned meaning from myriad signs and portents. Compared to science, nothing else even comes close.

U: OK, but this still isn't proof that scientific theories are actually true.

R: I never said that scientific theories are actually true, only that they are reliable guides to action. In fact, one of the fundamental tenets of science is that nearly all scientific theories are almost certain to be false. For example, there are two theories on which all of modern physics rests: general relativity and quantum mechanics. Both of these theories have enormous predictive and explanatory power. Neither theory has ever made a prediction that was not confirmed by experiment. And yet the two theories are fundamentally incompatible with each other so we actually know that at least one of them must be "wrong" in some sense. But (and this is very important) scientific theories, when they are wrong, are almost invariably wrong in a very particular way. The way in which they are wrong is rather tricky to describe. Isaac Asimov did an excellent job of describing how scientific theories are wrong in his essay, The Relativity of Wrong, which happily has been published on the Internet.

U: I'll be sure to read it at the earliest opportunity. But it still seems to me that none of this has anything to do with Truth (with a capital T). It may be that the methods of science could be effective even if they are completely wrong in a metaphysical sense.

R: Yes, that's possible, but it's unlikely. It's much more likely that the reason that scientific theories are as reliable as they are is that there really is a metaphysical reality "out there", and that our scientific theories really are a somewhat accurate description of that reality. Moreover, there is actually a sound scientific reason to believe that this is the case. You can actually treat the metaphysics of science as a scientific theory and test its predictions. For example, one of the predictions made by the theory that science is an accurate reflection of metaphysical reality is that we ourselves are a part of that metaphysical reality. This has consequences in terms of our ability to construct scientific theories. I won't get into the details here except to say that the theory of scientific metaphysical reality makes certain predictions about the structure of scientific theories, all of which turn out (so far) to be true. (For example, it predicts that scientific theories will be constructed by physical entities that emulate Turing machines.) So there really are some very powerful reasons to believe that science is a not-wholly-inaccurate description of metaphysical reality.

U: Holy cow.

R: Yes, it's rather a lot to take in, isn't it?

U: It certainly is.

R: And actually I've barely begun to scratch the surface. Scientific metaphysics is intimately related to neuroscience, computer science and information theory, all of which are fields of study that are only a few decades old. The entire enterprise of science only goes back a few hundred years. Who knows what we'll know a few decades or centuries from now?

U: OK, how do you account for all the Unicornians out there? All the early martys who died for their beliefs? People's personal experience with Uni and his minions? Do you really think we're all idiots?

R: "Idiot" is putting it a bit strongly. I certainly think you're (almost certainly) wrong in a metaphysical sense. But interestingly, there is a very satisfying (at least to me) scientific explanation of why so many people believe in unicorns. Unicorns may not be real, but the belief in unicorns is very real, and can have actual physical effects on the body by virtue of the placebo effect (which is a real, physiological, scientifically verified phenomenon). It is not even out of the question that if I could summon a sufficiently strong belief in unicorns that that belief alone could cure me or at least make me feel better.

U: So why don't you give yourself over to the power of Uni?

R: Because I can't. The placebo effect relies on believing in the efficacy of the placebo, and I know too much to ever be able to fool myself into thinking that unicorns are real in the absence of compelling evidence to the contrary. Belief is very powerful, but its power is predicated on a level of ignorance that I am no longer able to muster. I don't know how to "unlearn" my Scientific worldview.

U: I'll pray for you nonetheless.

R: I appreciate the gesture, and accept it in the spirit in which it is offered. Hm, it looks like the line is starting to move. And you know what? I'm actually feeling a little better. Maybe it was just a touch of gas, not food poisoning after all.

U: Or maybe Uni just worked a miracle in your life.

R: Maybe. We could put it to the test and order another round of mussels.

U: I would think you would be reluctant to put yourself at risk like that.

R: I meant for you, not for me. I've had quite enough of this kind of experiment for one day.

U: But that wouldn't be a valid experiment according to you because even according to you I can summon the power of Uni by virtue of my belief even though you can't.

R: Good point. But the placebo effect will only get you so far. If these mussels really are contaminated with salmonella I think even the strongest belief in Uni will be impotent. You really will need antibiotics. (Christian Scientists occasionally do this kind of experiment on themselves or their kids, usually with unhappy results.) So while it would not be conclusive, I do think it would yield an interesting data point. Shall I place an order?

"I never said that scientific theories are actually true, only that they are reliable guides to action. In fact, one of the fundamental tenets of science is that nearly all scientific theories are almost certain to be false."

and

"U: ... it still seems to me that none of this has anything to do with Truth (with a capital T). It may be that the methods of science could be effective even if they are completely wrong in a metaphysical sense.

R: Yes, that's possible, but it's unlikely. It's much more likely that the reason that scientific theories are as reliable as they are is that there really is a metaphysical reality "out there", and that our scientific theories really are a somewhat accurate description of that reality."

Deutsch goes into all this in exacting detail (not so much in chapter 7 but later).

Yeah, sorry, I should have been more specific (and the "necessarily true" part of my comment wasn't the phrase I should have used). It's the, "possible but unlikely" description that confused me. How do you even begin to form a probability estimate, of a correspondence between effective theories and truth?

At the end of the day, there's still the (possible) Matrix universe out there, where the Matrix programmers make anything at all they want, happen. Can't we all have just woken up yesterday, with implanted memories? Can't physics be completely different tomorrow? Of course it's all possible. I don't see how you go about showing that it's actually "unlikely". It would seem that you would need some distribution over the space of ways-the-universe-could-be. I don't know how you make progress on that topic.

> At the end of the day, there's still the (possible) Matrix universe out there

Along with the (possible) invisible unicorns and the (possible) invisible construction elevator. All of these can be (tentatively of course) rejected for the same reason: compared to existing scientific theories, these alternate theories are more complicated but have less explanatory power. There is no *proof* that simpler theories with greater explanatory power are better (in the sense that they also have greater predictive power), but there is overwhelming *evidence* that they are. And there is even a theory to explain *why* they are that I've alluded to, having to do with how physics gives rise to information. See TFOR for details.

> Can't we all have just woken up yesterday, with implanted memories?

Why yesterday? Why not last week? Why not a millisecond ago? Your theory has this arbitrary unexplained discontinuity in it that has no explanatory power. That gives it the same status from a scientific point of view as the Matrix, invisible unicorns, and the invisible elevator.

> Can't physics be completely different tomorrow?

That depends no what you mean by "completely different." In July 1994 comet Shoemaker-Levy collided with Jupiter, an event that according to some of the rival physical theories of the day should have been only slightly more probable than finding a unicorn. Those theories have since been revised. Was physics "completely different" after that event? Or an even better example: prior to 1905, light was thought to travel through an invisible "ether". Was physics "completely different" after Einstein?

> I don't see how you go about showing that it's actually "unlikely". It would seem that you would need some distribution over the space of ways-the-universe-could-be.

No, you don't, because we are not trying to divine the nature of metaphysical reality ab initio. We have data. It's quite possible that the universe we live in is a priori very improbable, but that objection is answered by the the overwhelming evidence that we do in fact live in this universe. The most parsimonious (and hence scientifically most likely) explanation of why we *appear* to live in this universe is that we actually do live in this universe.

(Just to be clear: I'm a scientist/atheist myself. Just playing devil's advocate here.)

The thing I'm trying to address is your Unicornian's statement that: "you still have to assume with no proof that "explanatory power" is a reliable guide to Truth. So your beliefs are still, at root, based on faith."When you tell me that science is self-justified (that there are scientific reasons for believing that the process of science gets closest to truth), that doesn't seem like an argument much different in kind from religious people telling me that faith is self-justified. That you believe faith leads you closer to truth, because you have faith that it does.

To put it another way: there are always an infinite number of theories compatible with any finite collection of data. Generally, scientists pick the "simplest" one. And, in practice, that seems useful, avoiding overfitting the data when it comes to future observations (in this universe, anyway).

But still. When a Creationist tells me that the earth is only 6,000 years old, and the fossils and geologic evidence were put there 6K years ago by God ... that "fits" the data too. It's much more complex of a theory, and it offers no additional explanatory power. So it's a poor scientific theory of the data.

But what is the connection between explanatory power and truth? What do you say to someone who rejects science as a process that leads to Truth?

You have to distinguish between two different kinds of truth. There's objective truth, and there is metaphysical truth. Explanatory power is (empirically) related to predictive power, and predictive power is related to objective truth *by definition*. If I predict that it is going to rain tomorrow and it does indeed rain tomorrow then by definition my prediction was true.

Science is *empirically* a better guide to objective truth. Using science I can make much more accurate objective predictions than I can using faith. (I can even make accurate predictions about the effects of faith!)

Science has nothing to say about metaphysical truth, except to observe that the most parsimonious explanation of the reason science is so good at getting at the objective truth is that it bears some relation to the metaphysical truth.

But I think it's actually important to retain some humility about this. There's an argument to be made that science is *not* a reliable guide to metaphysical truth, what I have called the elephant in the atheist living room: the laws of physics are time and space symmetric, but my perspective on the world has a privileged location: *I* am *here* *now*. Symmetric laws of physics cannot account for this manifest asymmetry in *my* experience.

> What do you say to someone who rejects science as a process that leads to Truth?

That depends. There are a lot of different degrees of rejecting science, just as there are a lot of different degrees of drug use. Lumping all drug users together is a huge mistake, and lumping all religious people together is likewise a huge mistake. Social drinkers ought not be conflated with heroin addicts.

Ah! Objective truth vs. metaphysical truth. Yes, that makes sense. In the past, I've used the argument that: even if we are just brains in a jar somewhere, and all our perceptions are just illusion, science is about predicting exactly which illusions we'll be seeing tomorrow. There still remains something to predict, which we generally label "objective reality". That doesn't mean it is True, but there's a kind of truth there.

And yes, I like the "parsimonious explanation". Your comment is a good answer to my confusion, so thanks.

As for TFOR: Ordered yesterday, shipped today. And it's all your fault. :-)

OK, I got TFOR and read chapter 7 (as well as a couple of the earlier ones).

I enjoyed his takedown of inductionism, which was novel for me.

Similarly, I have long thought that science, at root, was about making predictions. He made a strong case that it is about making explanations, not (just) predictions. I found it really enlightening to consider his hypothetical example of an oracle that would generate the correct prediction for any possible experiment, and ask whether science would therefore be done.

The problem, of course, is that you can't use such an oracle to build a spaceship or a bridge, or solve much of any other problem. There's something missing: "understanding". He struggles a bit to define just what that might be, but he's convinced me that science is not just about predictions.

Christian: Ron may offer his own advice. Just for the record, his recommended The Fabric of Reality also covers quantum mechanics (for the layman!) in the early chapters.

I myself would recommend the Quantum Physics sequence from the Overcoming Bias blog. (It covers much of the same territory as Ron's TFOR, and seems compatible with the ideas there.) There's a subset which is "just the science".

I don't know of a simple layman's introduction to General Relativity. (Special Relativity is much easier and more accessible, but General is a bigger project.) Probably is one somewhere, though.

Just FWIW, I'm not sure what you mean by a "comparison" between the two. QM talks about what happens to very small things (photons, etc.) on very short time horizons. GR talks about very big things (planets, galaxies) over very long times. It's known that they are incompatible, so for sure there's some future theory which modifies one or the other. But they're applied in such different domains, that it's hard to even imagine a real experiment that could resolve the conflict. Roughly speaking, you just use the right theory for the right domain, and the conflict never really comes up.

It's sort of like "comparing" an apple and a coffee mug. They're mostly just ... different. It wouldn't really make sense to write a book "comparing" QM and GR.

The best accessible work on general relativity that I know of is Taylor and Wheeler's "Spacetime Physics." (http://www.amazon.com/Spacetime-Physics-Edwin-F-Taylor/dp/0716723271) Or you can go straight to the source and read Einstein's own popular account "Relativity: The Special and the General Theory" (http://www.amazon.com/Relativity-Special-General-Penguin-Classics/dp/0143039822). It's not quite as good as Taylor and Wheeler, but it's a lot cheaper :-)

QM is tougher to recommend because there is so much confusion out there. The entire field is still haunted by the ghosts of Bohr and Feynman, both of whom got it badly wrong in some very important ways. Even Deutsch doesn't get QM right. The best author I've read is Mermin, who has a very lucid account of the Bell inequalities (and relativity for that matter) in his book "Boojums all the way Through." But the only people who really get QM right are IMO Cerf and Adami, and the only popular account of their work as far as I know is the one I wrote: http://www.flownet.com/ron/QM.pdf

I don't know of any popular works that directly tackle the incompatibility of QM and GR, though I'm sure someone has written one. Apparently, the claim is somewhat controversial. Wikipedia says:

"[C]ontrary to the popular claim that quantum mechanics and general relativity are fundamentally incompatible, one can demonstrate that the structure of general relativity essentially follows inevitably from the quantum mechanics of interacting theoretical spin-2 massless particles [2][3][4][5][6] (called gravitons)."

But trying to move towards more primary sources yields things like this:

"Treating GR as an ordinary field theory encountered in the SM is practically useless,because the quantum theory obtained is not renormalizable.Hilbert-Einstein action describes at quantum level a theory of SELFINTERRACTING gravitons,quanta of gravitational field i.e.particles with spin two.Other attempts have been made of finding something else instead of the HE action.The linearized theory of gravity (developed by Einstein in 1916) is basically "good" when it comes to quantum behavior (i analyzed this theory using standard BRST antiparanthesis-antifield forrmalism (developed frankly by Batalin&Vilkovisky,but it's usually called Lagrangian BRST (cf.the Hamiltonian approach found independently by Becchi,Rouet,Storra and Tyiutin))),but it has the disadvantage of working with the gauge-fixed Pauli-Fierz action which describes the FREE (i.e.NONSELFINTERRACTING) GRAVITONS.In fact,these gravitons interract with ghost fields.Other attempt was the so-called Weyl gravity (i.e. gravity based not on the Riemann curvature tensor (of the curved manifold called spacetime),but on the conformal Weyl tensor,un ugly (still 4-th order) tensor).This theory is excellent at quantum level (i.e.renormalizable) but the classical (nonrelativistic) limit of the unquantized action gives you 4-th order LODE of motion (cf.2-nd order LODE of motion in the Newtonian limit of Einstein GR).A step forward was made by Elie Cartan who developed the so called "Einstein-Cartan GR" which used other fields (called vierbeins,usually seen as vielbeins) for describing the gravitational field.This theory is good because it allows coupling with spinor and scalar matter fields in a theory called SUPERGRAVITY.If i'm not mistaking,these theories of Sugra (apud Supergravity),though allow an unifying theory of all 4 fundamental interractions,are,at quantum evel,still nonrenormalizable.I mean,if they were (it doesn't matter how many supemultiplets of particles it envolved),why would ST and LQG be alive today??And then,ST,the final (??) frontiere.It is said to give a satisfactory behavior of gravitational interraction at the quantum level (this time there are no point-particles like in SM anf SUGRA,but strings,10-dimensional objects)."

I translate this as: people a lot smarter than me have been working on this for a few decades now and they still haven't figured it out.

> Just for the record, his recommended The Fabric of Reality also covers quantum mechanics (for the layman!) in the early chapters.

Yes, but unfortunately Deutsch makes a crucial mistake that causes him to run somewhat off the rails. I won't tell you what it is because it makes too good of an exercise to figure it out on your own, but I'll give you a hint: it's in the middle of chapter 2.

OK, I read chapter 2 of TFOR. Let me see if I can guess the "crucial mistake" that you mention.

Is it: Deutsch invents a concept of "trillions of shadow photons" in order to explain QM, which is really a needless complication. A much simpler model is that each individual photon is non-local, distributed throughout space. Deutsch seems to want photons to be point particles -- which would be a conflict with Heisenberg Uncertainty as well, although he doesn't mention that -- and then has to recover the non-local behavior by having these "trillions of shadow photons" follow along everywhere, with every real photon.

Instead, just realize that a single photon is a waveform, distributed throughout all space, but with a probability density that roughly localizes it somewhere around a point. This explains Heisenberg stuff much better as well.

You're on the right track, but you haven't quite gotten it yet. Again, the explanation is fairly long, so I'll make it a regular post. But here's another hint: Cerf and Adami's quantum information theory can be summed up in one pithy slogan: entanglement and measurement are the same physical phenomenon. Therefore, any explanation of QM that talks about measurement without talking about entanglement (which is what the theory of shadow photons does) will necessarily be incoherent. There is a very specific point where Deutsch's argument sweeps an inconvenient experimental fact under the rug, and that is where he runs off the rails. There's a particular paragraph you can point to and say, "This is clearly wrong."