Sunday, April 21, 2019

Why and how I understood QM as a teenager

First, because Ehab has reminded me, I must start with promoting my PhD adviser Tom Banks' December 2018 book on quantum mechanics. I have learned a lot from Tom, and if I didn't, our views on foundations of QM were aligned. The book discusses linear algebra and probability calculus as the background – Tom immediately presents the amplitudes and the main rules of the game as a Pythagorean-flavored probability calculus; "unhappening" is an essential new quantum feature; Feynman lectures and two-dimensional Hilbert spaces, the Feynmanian attitude (without continuous Schrödinger waves) to "start to teach QM" I have repeatedly defended; quantization of harmonic oscillator and the fields; more details on the QM linear algebra, eigenvalues, symmetries; the hydrogen atom and derivation of basic "atomic physics"; spin; scattering; particles in magnetic fields; measurement with Tom's favorite focus on collective coordinates; approximations for molecules; quantum statistical physics; perturbation theory frameworks; adiabatic and Aharonov-Bohm/Berry phases; Feynman path integral (!); quantum computation (!); seven appendices on interpretations of QM plus 6 math topics: Dirac delta, Noether, group theory, Laguerre polynomials, Dirac notation, some solutions to problems. I think there's no controversial Banksy visionary stuff in the book and if there's some of it, you will survive.
Now, switching to the dark side: Another book against quantum mechanics has been published – this time from a well-established, chronic critic of physics. Numerous non-physicists wrote ludicrous, positive reviews of that stuff for numerous outlets, including outlets that should be scientific in character. The book may be summarized by one sentence:

The only problem with quantum mechanics [...] is that it is wrong.

It doesn't look like a terribly accurate judgement of the most accurately verified theory in science. The contrast between the quality, trustworthiness, and genre of this anti-QM book and Tom's book above couldn't be sharper. Readers and their hormonal systems must be ready for hundreds of pages of comparably extraordinary statements. For example:

The risk, [the author] warns, is the surrender of the centuries-old project of realism...

So here you have it. "Realism" (which is called "classical physics" by physicists) "must" be upheld because it is a "centuries-old project", we are told.

In contrast to that, scientists are used to the fact that old theories are falsified and abandoned – events of this kind are really the defining events of all of science. All this worshiping of centuries-old projects is particularly amusing if you realize that the same author has previously claimed that research projects that are older than 5 years and don't produce a clear victory must be abandoned. The inconsistency is just staggering. There are tens of thousands of fans of this stuff who just don't seem to care.

In a recent lecture, the new self-declared lead warrior against quantum mechanics (how do you suddenly become the king in promoting some fundamentally wrong statements about science? Will the other billions of people who are wrong – including Maudlin, Bricmont, Becker, 't Hooft, ... – accept you as their new leader? How does this promotion work?), he clarified the issue "who should talk and who should listen" to this stuff (5:20):

If you are colleagues or professionals, then this talk is not for you. And if you have your own approach to quantum mechanics, then you shouldn't stand up and say "my approach is blah blah blah" because I am here now. You can be here later.

The audience laughed after every sentence – because those statements look like a parody showing an arrogant person who has no respect for the fine processes by which science converges to the truth. The problem is that both sentences were really meant seriously. People like this speaker are systematically spreading misconceptions that are immediately seen to be wrong by professionals – and they get away with it. The speakers are well aware that everyone who buys their talk is buying it due to his insufficient knowledge or intelligence. But they seem to enjoy it when their business works like that. The more people are deceived in this way, the better for the speakers.

And the main argument for these statements is that these misguided talking heads have been capable of conquering a podium. If you can conquer a podium as well, maybe you may become similarly influential. The ability to conquer a podium is what should decide about which science spreads, the speaker tells us. And just to be sure, this speaker is not getting the podiums because of his muscles but because of his whining.

This way of thinking and acting totally contradicts the scientific approach. In science, the author of a new important insight is primarily addressing the explanations to the greatest experts because they're the most likely ones to get it and appreciate it. When I wrote my first Matrix theory papers, I wanted Matrix theory experts to read it – but I also wanted Ed Witten to read it (because he was likely to be smarter than the average Matrix theory proponent – and Witten surprisingly hadn't written any Matrix theory paper by that time), and that wish came true as shown by an e-mail I got from Witten a few days later. I wasn't trying to impress some random people who have no clue about advanced physics. Neither does any real scientist because there's no scientific point in persuading random non-experts. And the real scientist derives the authority from evidence, not from some random temporary control over a podium – a podium where this particular speaker doesn't belong.

The people in the audience, can't you see that not just the content but the very approach to all these matters and to the argumentation is completely anti-scientific and wrong? Why do you allow such speakers to give additional talks there?

At any rate, I still can't believe how insanely difficult it is for most people – even for people who have been accepted to a famous college and who have overtly studied physics for half a century – to understand the basic rules of quantum mechanics that may be summarized to a few paragraphs and whose unavoidability may be almost fully proven by an additional bunch of paragraphs.

Do I remember how and why I accepted quantum mechanics – in the proper, Copenhagen-style sense – in the first place? I have – and many of us have – some personal history of learning that is completely atypical and cannot be copied – and shouldn't be copied – at all. But I still think it's interesting.

OK, when I was 4, my model of the world was a 3D space plus time where each point of space carries a bit. Either the "matter" (some universal matter that behaves as continuum) is there, or it's not there. So all the interactions were pretty much collisions of solids at the fundamental level. I've learned lots of mathematics that was actually relevant for understanding classical mechanics. But at some level, the "binary solid" model of the world – classical mechanics with some model how the matter looks internally – was relevant for my views up to the age of 12 or so.

Only at that age, I started to be persuaded that there also existed fields and waves. First, as a young piano player, I was finally sufficiently certain that C+E, a chord, is surely nothing like a D in any sense (despite the fact that the frequency of D is the average of C and E), so each frequency has to have its own "tally" (OK, "amplitude"). Even more clearly, I could catch – not just Radio Free Europe – but many other radio stations. The volume in our apartment had to be full of information about the sounds associated with all those radio stations...

OK, so I accepted there were electromagnetic waves at all the frequencies that could co-exist in a given volume. Some sort of classical field theory was my default model of the Universe between the ages of 12 and 16 or so. I've never felt the urge to "reduce" the electromagnetic waves to any kind of aether or stuff like that. The amount of information in the apartment – which knows about all the audio streams of the radio station – was just high enough. That was the lesson and it was derived from direct empirical observation. I never found anything "more economical" in trying to reduce the new required description – audio streams at all frequencies – to a previous model, like the "binary solid" model of classical mechanics. I avoided all kinds of "aether paradigm" mainly because it was less elegant. Functions directly associated with the space looked more likely in the vacuum.

Fine. So as a high school boy, I was reading tons of especially Einstein's texts, was enthusiastic about his plans for a unified field theory, and I was gradually getting skeptical about his chances to advance physics in the last 20 years of his life. The amount of learning I did in a few years was substantial... and the ordering was sometimes highly illogical. So I understood how to deal with general tensors in GR and with the Christoffel symbol etc., how to define the curvature tensor, but only a year later, I learned the relationship between the dot product and the cosine of an angle – at school! ;-) So there have been things I only learned at school.

There have been other examples of this bizarre chronology. They become unavoidable if you try to rediscover or learn many things too fast – and no one is preparing the journey for you. At some level, you can do GR without knowing the link between the cosine and the dot product. If you think about it, you don't really need to know anything about the cosine to write and understand the equations of GR themselves at all. Well, you will be a bit limited if you try to apply GR to particular problems. And make no mistake about it, the number of GR calculations that I did in the high school was small. But even some calculations may be done without the cosine formulae. And I would rediscover those once they were needed.

But all this illogical ordering had some great virtues, too. It made me think about the actual relationship between the theories. What is needed for what, what is easier, what is harder, and I could see that other people could promote too contrived things before analyzing some simple and fundamental things – because I have accidentally done the same. Since that time, I would constantly subconsciously ask questions like: Shouldn't we try all these things before we do others? Or shouldn't we also try something else because we could have learned our "current theory" earlier just by accident? The lesson from the illogical ordering of my own learning has also made me sure – already as a teenager – that e.g. the people who favor QFT over string theory because QFT was written down first are utterly irrational.

More generally, such bizarre things that are illogical and (in this case pedagogically) uneconomic often turn out to be great lessons in the long run. If someone is learning things according to a curriculum that is too polished, he or she is likely not to develop his or her critical thinking. You really train your critical thinking when you collide with critical problems – and you need to deal with them. A similar explanation clarifies why the kids trained in the carefully designed indoctrination schools with safe spaces are learning virtually nothing.

Fine. Around the age of 16-17, I learned many mathematical things that were surprising for me. Taylor expansions – the infinitely long polynomials were enough to reproduce totally non-polynomial functions. The fact that the multiplication of tables – matrices – may be natural. That even non-smooth functions may be written as Fourier series (before that, I thought that only nice analytic functions were acceptable in nice mathematics and physics while unsmooth ones etc. didn't belong to fundamental mathematics and physics – the Fourier series showed an unexpectedly blurry line in between the two worlds). Orthogonal polynomials. I derived the formula for the 3rd and 4th degree polynomial equations and understood why my algorithms couldn't work for the 5th and higher degree equations.

Those had some implications for physics – but I was getting close to atomic physics etc. The hydrogen atom was clearly an object linked to quantum mechanics that attracted my attention most. It has a simple spectrum, surprising yet easy to describe behavior in emission and absorption, and there seems to exist an equation – Schrödinger's equation – that explains it right. So while I wanted to build the world out of classical field theory and complete a plan due to Einstein (and I was happy to have invented some Skyrmion-like topological solitons that could have explained the quantization of charges in an Einstein-friendly way), I found myself stealing an increasing fraction of the mathematical tricks that are used in the quantum mechanical solution of the hydrogen atom.

All the details work so well that the equation really has to be the relevant mathematical equation, I concluded at some time. The remaining problem was to understand the proper interpretation. I already had some access to wise sources – later also college instructors – who managed to communicate the view of the "technically active physicists" that the people doing the "interpretations" of QM haven't gotten anywhere.

(Another late surprise was awaiting me years later when the most savvy "public QM" seminar organizers in Prague, such as Bedřich Velický who is well-known to most of the particle physics Nobel prize winners in the West, turned out to be a Bohmian or something like that... and seriously wrong about QM. By that time, I had semi-assumed a model that the Western scientists were generally better than the Czech-confined ones and Velický's take on QM conflicts with that. Despite – or because? – his Western contacts and success and his different appearance, I think that Velický was an example of a pop scientist, after all.)

But for months, I do remember on insisting on some "realist picture". I thought that there had to be some detailed mechanisms that made the wave function collapse. This is a belief that all the anti-quantum zealots (mostly adults) still have. They (incorrectly) assume that the measurement is some process with many detailed parts that should be deconstructed and there's a lot of extra stuff to say about what's happening. When you believe in something like that, you expect some vindication around every corner. So because quantum field theory was what I was going to learn (I started with Bjorken-Drell treatment of QED – which I think was written in a lousy, obscure way; and then I got – somewhat randomly – Pierre Ramond's textbook of QFT in the library), I actually believed that one of the "added benefits" of quantum field theory "had to be" its perfect explanation what the collapse looks like – still assuming that the wave function was a real wave of a sort. It took weeks to just the get the basic information that the "added value" of QFT is something entirely different – many particles, antiparticles, relativistic invariance in QM... and the basic conceptual framework stays the same as in non-relativistic QM models.

We're getting to the point that should be clarified in this blog post according to the title.

Let me jump over the critical period now. When I came to the college in Fall 1992, I randomly got Roland Omnes' essay about the completeness of a "consistent histories" interpretation of QM – which was accidentally published weeks earlier – and at that time, I was already pretty much certain that QM worked and all the complaints and proposed revisions were rubbish. Also, since that time, I understood that while "consistent histories" were a nice modern language, all this stuff was just a modernized presentation of the original, Copenhagen "interpretation". Omnes has been rather clear about all these points, too. But I wasn't uncritically devouring his article. The article was already confirming some conclusions that I had made before.

Why did I make them? Why did I abandon the idea that the wave function should be a real wave or encoded in some real degrees of freedom?

I think there have been several contributing reasons. First, I got reasonably certain that the wave function had to evolve "almost precisely" according to the wave equation, i.e. it should be diluted into nice superpositions and respect the linearity principle, when you're not observing. So one simply shouldn't add some brutal deformations of the equations to keep the wave function more "localized" – those would be like a bull in a china shop and would conflict with observations. If you try to add some extra forces that "really prevent the wave function from spreading too much", these new forces will also damage the predictions in many basic situations. We just directly empirically know that such forces are non-existent – or at least weak enough to be negligible for the basic outcomes of normal experiments.

Second, I wrote lots of formulae for various transitions between bases of the Hilbert space in quantum field theories (I liked to write explicit "wave functionals" for states in the QFT Hilbert spaces etc.) – at some point, supersymmetric quantum field theories were added and I loved to play with the Grassmannian variables and the Berezin integral. But because I have seen the huge number of bases – and very useful bases – in these Hilbert spaces of quantum field theories and some other quantum mechanical theories, I became sure that it was utterly misguided to look for any "privileged basis" of the Hilbert space etc.

There couldn't be any universal rule that tells you what are the possible states of a measured system after the measurement. The relevant post-measurement states really depended on what you measured and because there were many ways to measure, none of them could have been "better than others". Even the "useful bases" were incredibly dependent on the situation – and the set of "useful bases" just couldn't possibly be pre-determined by any natural rules. It would be utterly unnatural to treat some bases as better than others: the "democracy between the bases" is really an example of a symmetry, in fact the \(U(\infty)\) symmetry on the Hilbert space, an example of naturalness, beauty, minimality, or Occam's razor. For this special treatment, some extra information would have to be added on top of Schrödinger's equation. The extra structure contradicts Occam's razor and would almost certainly not be unique. On top of that, there seems to be quite a direct empirical evidence that all these bases are equally good.

So quantum mechanics was a framework to calculate probabilities of measurements. This "interpretation" of the wave function is completely necessary for Schrödinger's equation to be applicable to real-world physical situations. The real question was whether there could be something more fundamental than Schrödinger's equation that would produce all these Hilbert spaces, nonzero commutators etc. as some effective approximate theory. My conclusion was a resounding No. The operator algebras had their own beautiful rules. They couldn't have been reconciled with something else, something "more classical". After trying a few possibilities – including all the major well-known "alternative interpretations" – how to "extend" the quantum objects to make them ultimately classical, I understood that "quantum" and "classical" were really the only two types of theories we could create, they are "competing with each other", and the evidence overwhelmingly favors quantum mechanics.

Well, yes, at some moment, I semi-imposed a deadline of a "few months" on myself to find a "better interpretation", or stop playing with these doubts. That was sensible, I think, because if some "better but unequivalent" rules to treat the wave function existed, one could find them and write them in a few paragraphs. So a few months should be enough – and if it is not, it's strong evidence that it doesn't exist and I could end up as the interpreters who are "lost in philosophical chatter".

There was some point – when I was 17 and maybe 18 – when I became subjectively certain that the voices saying "that the reform efforts about quantum mechanics were a waste of time" are precisely true. Some sociological observations have helped to push me in that direction, too. I just saw some texts by "interpreters" – and later, some "interpreters" in the college since late 1992 – who were doing things that were obviously low-brow. These people hadn't understood the beauty, naturalness, and effectiveness of the quantum mechanical apparatus including the amazingly diverse and complementary choices of operators and bases. They didn't understand something that – as I could clearly see – I did understand. Why the quantum apparatus is really more elegant, powerful, simpler in some respects, necessary to get certain observed qualitative traits of the Universe. So I was even more sure that they were wrong than I was sure that I was right.

At the end, the critical difference in the thinking was mostly due to the numerous calculations with particular quantum mechanical Hilbert spaces, theories, and operators. I think that the more actual, particular tasks you solve that depend on the apparatus of quantum mechanics (and some diversity of the tasks helps, too), the more seriously you take it. You may see increasingly clearly that the classical physics apparatus is just a medieval tool that is totally incapable of solving these modern physics problems. It's like riding horses to the Moon. You just can't do it. But to see that you can't do it, you actually have to make some realistic preliminary steps to get to the Moon. If you only sit on a horse, or beneath a horse while you're cleaning the horsešit, you may easily fool yourself into thinking that this wonderful horse could take you to the Moon. But it couldn't! Sitting beneath the beautiful stinky horse, you haven't even started to deal with the lunar tasks, so you have zero credentials to say anything about the relationship between the horse and the Moon! Buzz Aldrin has a better idea whether he could have gotten there on a horseback than a talking head sitting beneath a horse!

To summarize, I believe that the fundamental difference between those who understand quantum mechanics on one side and the "realists" who don't on the other side is that the first group has done many more calculations where quantum mechanics is applied to understand modern physics problems and their various aspects. The more stuff of this kind you do, the more you realize that the classical physics framework has nothing to do with all the clever tricks and "miracles" that are absolutely needed for quantum mechanics to succeed. There's really a direct contradiction. Classical physics is what would prevent you from successfully exploiting the characteristic properties of quantum mechanics.

"Realists" are simply people who haven't done much modern physics – or who haven't done any modern physics at all. They prefer to talk about nonsensical philosophical prejudices, insist on the horsešit's ability to get them to the Moon. They are not doing science. If you do modern physics intensely enough, you will understand that the most insensitive proposed "bull in a china shop" reforms are catastrophically devastating and even if you tried to construct some "harmless" modifications or new ways of talking about quantum mechanics, they wouldn't contribute any added value. To say the least, they would add lots of arbitrary, ugly, untested structures to the theory.

The disputes between quantum mechanics and its critics aren't scholarly conversation between two possible hypotheses anymore. They're social interactions between a class of scientists on one side and the children who were left behind on the other side. Sadly, contemporary schools are increasingly producing whole generations of children who skip the classes on Fridays and who are left behind.