Reverse-Engineering Quantum Mechanics, I.

Generic Feynman Diagrams

‘The Coyote Problem’

I think I can safely say that nobody understands quantum mechanics.
— Richard Feynman, The Character of Physical Law (1967, p.129).

Six decades later, Feynman’s claim arguably still stands (and remember, he won the 1965 Nobel Prize in Physics for his work on quantum electrodynamics, including inventing those squiggly diagrams that everyone uses now; so, if he doesn’t understand it…). Despite a century of unprecedented empirical success, the interpretation of quantum mechanics still remains very contested. In general, it seems undertaking any attempt to try to actually make sense of it is “considered barely respectable at all, if not actively disparaged” (Carroll 2019, p.4). Multiple interpretations coexist—Copenhagen, Many Worlds, Bohmian, Objective Collapse, Quantum Bayesian and so forth, at least a dozen or so—each with committed proponents and unresolved difficulties. No consensus view has emerged. Quantum theory has taken on an almost mystical reputation, much of which is, frankly, arrant nonsense (Bricmont 2017).

This situation was noted by another physics Nobel Laureate, Steven Weinberg (2017), who wrote: “It is a bad sign that those physicists today who are most comfortable with quantum mechanics do not agree with one another about what it all means.”

The disagreement among practitioners even after a century since its creation suggests that the standard approaches to thinking about quantum mechanics might possibly be obscuring a potentially simpler, perhaps more direct understanding. Feynman offered a methodological hint for such a situation. Recounting an incident from his career, he described looking at some experimental data that he felt did not prove what was claimed, but that he, “like a dope”, deferred to the reports about it written by experts and never checked up on the strength of the original data for himself. Reflecting on this later, he remarked that had he been a “good” physicist, he would have ignored what the experts said and worked it out for himself (Feynman 1986, pp.254–5). The implication is clear: when the experts disagree, return to the very beginning and work it out from scratch, line by line, to ensure you understand what is going on, which is what he did ever after.1

This advice is ancient indeed. The Roman Stoic philosopher Seneca famously wrote:

Shall I not follow in the footsteps of my predecessors? I shall indeed use the old road, but if I find one that makes a shorter cut and is smoother to travel, I shall open the new road. Those who have made these discoveries before us are not our masters, but our guides. Truth lies open for all; it has not yet been monopolized. And there is plenty of it left even for posterity to discover.
— Seneca, Moral Letters to Lucillius, Letter XXXIII.

The history of science offers a strong precedent for such a return to first principles acknowledging, but not blindly following, the prior discoveries of our predecessors.

In 1905, the prevailing wisdom held that light was “obviously” a wave, ever since Thomas Young’s experiment a century earlier had established this, effectively demonstrating the “waves” contention of Christian Huygens, and counter-indicating Newton’s contention of light particles. However, in order to explain the photoelectric effect, Einstein went against this “obvious” view and proposed that light also possessed particle-like properties. This suggestion successfully accounted for the experimental data and eventually won him the Nobel Prize.2

Louis de Broglie, working soon after the award of Einstein’s prize, no doubt took notice of this result and went on to propose the converse (in his 1924 doctoral thesis, no less! Pretty brave!): that matter particles also possessed wave-like properties. This was very soon verified in the electron scattering experiments of Davisson and Germer in 1927, for which de Broglie, too, won the Nobel Prize, in 1929 “for his discovery of the wave nature of electrons” (pretty rare for doctoral work to so quickly, or even ever, get the top prize possible in physics).

This contention of wave-particle duality led Erwin Schrödinger to search for an equation that might be able to describe particle behaviour in terms of a wave equation. He did not derive his equation from axioms; rather, he followed an analogy from classical physics as a heuristic guide to infer an equation. He succeeded, as we know, in 1926, and another Nobel was awarded (in 1933 together with Paul Dirac) for the “discovery of new productive forms of quantum theory”. (At about the same time, Werner Heisenberg independently invented a way to do quantum mechanics based on matrix mathematics, which turned out to give the same results as Schrödinger’s wave equation. We shall have more to say about this in a subsequent post.)

So, taking Feynman’s hint seriously, if we really want to understand how quantum mechanics came about, we therefore need to go right back to the beginning, to its precursor, where classical physics began; that is, Newtonian mechanics. From this foundation, classical mechanics ascended through a well-defined ladder of increasing mathematical abstraction, the exact details of which are not necessary for the overall argument:

Newtonian \rightarrow Lagrangian \rightarrow Hamiltonian \rightarrow Hamilton-Jacobi

In tabular form, this progression looks like the left column of Table 1, starting at the bottom.

Each rung step up on the LHS increases calculational power but also increases mathematical abstraction further away from the familiar ontological concreteness of ordinary Newtonian mechanics. Newton gives us particles and forces in 3D space. Lagrange gives us the minimised-action principle in an abstract space (technically, the Lagrangian is a function in the tangent bundle of the configuration space, cf. Arnold 1989, chap.4, although the individual trajectories are in the configuration space itself). Hamilton gives us algebraic structures in an even more abstract phase space. And Hamilton-Jacobi gives us the action function S as a field in configuration space again (the same one). The other approaches all involve multiple differential equations, but the Hamilton-Jacobi formulation has a single master scalar function S that generates and “encodes” all the information needed to model the dynamics of the system in a single partial differential equation. It is very powerful, very efficient, and very far removed from our experienced 3D “reality” (whatever that is!).

Table 1: The ‘Dual Ladder’ of Abstraction
Classical Mechanics Quantum Mechanics
Hamilton-Jacobi Equation
Field S(\vec{q}, t) on configuration space.		 Schrödinger Equation
Field \psi(\vec{q}, t) on configuration space.
Hamiltonian Formalism
Phase space (q, p).
Poisson brackets \{A, B\}.		 \downarrow
Lagrangian Formalism
Action S = \int L \, dt.
Principle: \delta S = 0 selects a single path.		 \downarrow
Newtonian Mechanics
Particles in 3D space.
Forces: F = ma.		 ?

The classical side ascends from concrete particles in 3D space to an abstract field S in an abstract configuration space. The quantum side has the Schrödinger equation at the top as the direct analogue of the Hamilton-Jacobi equation, but the lower rungs—representing the steps ‘down’ to some sort of as-yet unknown ontology at bottom—are missing. This is The Coyote Problem.

Schrödinger noted a structural similarity between the Hamilton-Jacobi equation and the so-named eikonal equation of geometrical optics. Using the Einstein-de Broglie particle-wave hypothesis and the well-known optical-mechanical analogy (Arianrhod 2025), he sought for, and found, a wave equation such that the Hamilton-Jacobi equation emerges as the geometrical optics limit as wavelength \rightarrow 0, using the ansatz \psi \sim e^{iS/\hbar}. Here the classical action S is postulated as being the “phase” of the quantum function \psi. The heuristic analogy served its purpose and yielded the correct mathematics. But herein lies the problem. Having “crossed over” from the classical LHS to the quantum RHS, we find ourselves three layers of abstraction away from the bottom rung. The N-particle wavefunction \psi(\vec{x}_1, \vec{x}_2, \dots, \vec{x}_N) lives not in our familiar 3D space but in a 3N-dimensional configuration space. We have found the sophisticated mathematical machinery that produces correct results, but we have lost sight of the underlying reality.

We have therefore ended up somewhat like Wile E. Coyote from the Warner Bros. Road Runner cartoons: having run off the cliff edge in single-minded pursuit of our quarry, we suddenly find ourselves suspended “with both feet planted firmly in mid-air”.3 That analogy applies here very well. The mathematics works. The predictions are accurate. But we are apparently standing on thin air. I like to call this The Coyote Problem.

There is a school of thinking in quantum mechanics that tells us not to worry about what it actually means (compare Carroll’s observation above) and that one should instead just “shut up and calculate” (Mermin 1989, p.9). In cartoon physics, of course, the hapless characters standing impossibly in mid-air only ever fall once they realise they are doing the impossible. I can’t help but think that the “shut up and calculate” school of thought is really just saying, in effect, “whatever you do, don’t look down!”

It is here that I want to mention one of the texts that actually started me down this path a few years ago. Travis Norsen (2022, p.77) argued:

Let’s leave room in our thinking for the possibility—just the mere possibility!—that Schrödinger’s equation captures something fundamental and true about physical reality, but that it has a status like the status of the Hamilton-Jacobi equation in classical mechanics. Let’s leave room, that is, for the possibility that the quantum mechanical wave function \psi might not describe, in a literal and direct way, something weird and exotic and ‘out of this world’, but may instead indirectly describe something more mundane, just as the orchestrating field S in H-J theory can be understood as indirectly describing forces that particles exert on one another.

When I read this back then, it was a smack between the eyes moment. It is one of the most sensible things I have ever heard said about quantum mechanics in the 40-odd years I have been familiar with it. I even emailed him to say so. With Feynman’s hint ringing in my ears, it is what led me to retrace the increasing levels of abstraction on the LH classical side of the ladder and to go back to suitable sources to see how Schrödinger actually “crossed over” to the quantum RHS. We are pretty much never taught this back-story as students; the Schrödinger equation is usually just written down as the correct equation for non-relativistic quantum mechanics, and our job from then on is to just “shut up and calculate!”

Norsen’s paper also describes an alternative world where the physicists there have somehow discovered the Hamilton-Jacobi equation of classical mechanics, but without any idea of the “mundane” ontology that sits beneath it (i.e., the lowest rung of the LHS of our ladder above). Similarly to us with regard to \psi, they argue endlessly and fruitlessly about the nature of the mysterious “orchestrating function” S. Our own view of S is of course founded upon the knowledge that it is merely an abstract mathematical tool with no objective existence, simply encoding the way that Newtonian physical laws describe the motions of particles. Theirs is not. For them it is a strange and mysterious function in a higher-dimensional space that somehow almost magically “orchestrates” the motions of classical particles in real space through some unknown and possibly even “spooky” means. This has more than a small degree of resonance with our own world! Hence his contention quoted above. Pursuing this line of thought a bit further, I can’t help but think there may well be other planets where the physicists of those worlds are as familiar with the underlying ontology for the quantum function \psi as we are for the classical “orchestrating” function S.

The 100-year history of quantum mechanics has ascribed a number of weird and bizarre roles to the function \psi, which fact, in part, accounts for much of the dizzying proliferation of interpretations of it, as lamented by Weinberg above. If Norsen’s contention is correct then the wavefunction \psi is best not imagined as some weirdly bizarre object, occupying, as noted above, an almost mystical place not only in some approaches to physics but also in popular culture; but rather is simply a mathematical tool—no more weird than the “orchestrating field” of the Hamilton-Jacobi equation—that describes the dynamics of something potentially much more mundane. This underlying reality might even be as mundane as simply particles moving and interacting in three-dimensional space! But I’ll leave the question open to see where the mathematics leads us, rather than assuming any particular endpoint or underlying ontology. I intend to begin to follow the road laid down by our predecessors, per Seneca, and to branch off from it if necessary, by starting from scratch, per Feynman, and avoiding being overly swayed or influenced by the opinions of experts.

The research question then becomes: How does one “come down again” on the quantum side of the ladder? One of my wonderful physics lecturers, the late Peter Lloyd, who taught me quantum mechanics at Honours level, would always remind us to not simply “shut up and calculate” according to the recipes we learned, but to instead ask ourselves the deeper question: “what is going on?” It seems then that to begin to answer this question we must try to reverse-engineer the theory, moving back down through the layers of abstraction on the RHS to seek out what the ultimate bottom rung might actually look like.

What follows then in subsequent posts is an exploration of what Norsen’s suggestion might imply if taken seriously; not an argument for or against any particular interpretation, but merely a genuinely fair-minded attempt to figure out where the mathematics leads us, to try to work out “what is going on?” I’m pretty sure that Peter Lloyd would have been very glad his research-stance advice is still being followed all these years later, and that Sean Carroll would agree that it’s more than well worth the effort to at least try.

Notes

  1. Following this advice early in my career, I also did precisely this for my doctoral work on Einstein’s unified field theory. This is described in the Preface of the thesis, which can be found on ResearchGate. ↩︎

  2. In the very early 1920s, relativity was still considered a bit of a “too hot” topic, so the Nobel committee awarded the 1921 Physics prize (a year later in 1922) to Einstein for services to theoretical physics and “especially” for his explanation of the photoelectric effect.↩︎

  3. Donald Michael (1985) made this observation about Futures Studies, but it is quite apposite here.↩︎

References

Arianrhod, Robyn. 2025. “100 Years before Quantum Mechanics, One Scientist Glimpsed a Link between Light and Matter.” The Conversation, September 23. https://doi.org/10.64628/AA.739tf6y4x.

Arnold, Vladimir I. 1989. Mathematical Methods of Classical Mechanics. 2nd ed. Translated by K. Vogtmann and A. Weinstein. Graduate Texts in Mathematics 60. Springer.

Bricmont, Jean. 2017. Quantum Sense and Nonsense. Springer International Publishing. https://doi.org/10.1007/978-3-319-65271-9.

Carroll, Sean M. 2019. Something Deeply Hidden: Quantum Worlds and the Emergence of Spacetime. Dutton / Penguin Random House.

Feynman, Richard P. 1967. The Character of Physical Law. Reprint edition. MIT Press.

Feynman, Richard P. 1986. “Surely You’re Joking Mr. Feynman!”: Adventures of a Curious Character. As told to Ralph Leighton. Edited by Edward Hutchings. Unwin / Counterpoint.

Mermin, N. David. 1989. “What’s Wrong with This Pillow?” Physics Today 42 (4): 9–11. https://doi.org/10.1063/1.2810963.

Michael, Donald N. 1985. “With Both Feet Planted Firmly in Mid-Air: Reflections on Thinking about the Future.” Futures 17 (2): 94–103. https://doi.org/10.1016/0016-3287(85)90001-1.

Norsen, Travis. 2022. “Quantum Ontology: Out of This World?” In Quantum Mechanics and Fundamentality: Naturalizing Quantum Theory between Scientific Realism and Ontological Indeterminacy, edited by Valia Allori. Springer International Publishing. https://doi.org/10.1007/978-3-030-99642-0_5.

Weinberg, Steven. 2017. “The Trouble with Quantum Mechanics.” The New York Review of Books, January 19. https://www.nybooks.com/articles/2017/01/19/trouble-with-quantum-mechanics/.

Image credit: Generic Feynman Diagrams. Source: DuckDuckGo search

Leave a comment