Coming Back Down Towards the ‘Ground’
The main contention of the previous post can be summed up succinctly as: the Schrödinger equation is to quantum mechanics what the Hamilton-Jacobi equation is to classical mechanics. This is because it was – in a sense – ‘derived’ (really inferred) from it, via the optical-mechanical analogy between idealised particle paths and idealised geometrical light rays, first pointed out by William Rowan Hamilton in the early 1830s (and see, e.g., Masoliver and Ros 2010 for a detailed mathematical exposition). The main postulate of Schrödinger’s wave mechanics was that the action $S$ from Hamilton-Jacobi mechanics becomes the phase of the complex wavefunction $\psi\sim e^{iS/\hbar}$. This meant that we therefore found ourselves three levels of abstraction away from, and ‘floating’ above (so to speak), the ‘ground’ that classical mechanics was founded upon, namely Newtonian mechanics in 3D space. This post now begins the process of thinking about how to come down again to seek a more solid footing, if indeed there is even one to find…
When Travis Norsen (2022, p77) made his observation (in the previous post) regarding the Schrödinger equation’s analogous role to the Hamilton-Jacobi equation, he very quickly (in fact, the very next sentence) caveated that observation:
To be clear, I am not suggesting that there is a perfect parallel between QM and H-J theory.
Duly noted. But it would be very interesting to see just how far this parallel can be pushed, and what insights we might be able to draw along the way as we do the pushing…
The ‘dual ladder of abstraction’ table from the previous post showed the progression up the LH classical side and the crossover to the RH quantum side at level 4, yielding the Schrödinger equation as the quantum analogue of the Hamilton-Jacobi equation. Now let us ask about the levels below this on the RHS.
As it happens, there are very clear analogues on the RH quantum side for the formalisms found on the LH classical side, for each of levels 3 and 2. The step back down to level 1 is simultaneously both very obvious and rather ambiguous. But that is the ultimate goal, after all, and it has taken a century to not yet find it (or at least for no consensus to emerge on what it is), so one should expect that last step to be at least a little bit tricky 😁.
It is very tempting to begin writing out great swathes of equations and definitions, but—fun though that may be, at least for me—that would probably be counterproductive to reporting on the findings of the investigation in general terms. But don’t, even for a moment, think that I haven’t done this! The following is a photo of my whiteboard from around October 2023, showing the steps up to Level 4 in more detail, as well as some musings about how to proceed. (Note the arrow leading from the RHS linking to the Lagrangian on the LHS: the note there was about Feynman’s derivation of the Schrödinger equation from path integrals.)
Here I will just focus on the broader conceptual view rather than get deep into the weeds of the technical detail (which might come later, if this ever gets properly written up for a journal). Suffice it to say that definitions, equations and algebraic proofs can be found in any standard book on classical mechanics (e.g., Goldstein 1980; Arnold 1989), and easily compared with books on quantum mechanics (e.g., Auletta et al. 2009; Merzbacher 1998; Messiah 1999; Shankar 1994), if they don’t already have sections showing the relationships.
So, I will just fill in the RHS of the Table with the counterparts to the LHS for levels 3 and 2, offering some brief explanatory remarks, using as little mathematics as I can stand!
Level 3 – Hamiltonian
The direct quantum analogue of classical Hamiltonian mechanics is Heisenberg’s matrix (or operator) mechanics. Essentially, the methods of classical Hamiltonian mechanics are carried right over to quantum mechanics through a well-developed set of correspondences, which any number of books will expound.
The main object is the Hamiltonian $H$ which is a function of position, momentum, and time. It is a scalar function (i.e., a value with no direction) which represents the total energy of the system, usually equal to $T + V$, where $T$ is the kinetic energy and $V$ the potential energy, expressed in terms of coordinates and momenta. The dynamics take place in phase space, an ordered combination of position $q$ and momentum $p$, and allows for considerable simplification of analysis, reducing the finding of constants of the motion or symmetries to algebraic relations.
The use of phase space introduces a number of advantages that make calculations easier to perform through the use of what are termed Poisson Brackets. Without going into too much mathematical detail, Poisson brackets have certain mathematical properties that are incredibly useful. These end up turning into the famed commutators of matrix mechanics. The Poisson relation between position $q_i$ and momentum $p_j$ is
$$\{q_i,p_j\} = \delta_{ij}$$
where $\delta_{ij}$ is the 3D Kronecker delta (essentially a $3\times 3$ matrix with $1$s on the diagonal and $0$s off it), which expresses the fact that they do not ‘commute’ in the algebra of Poisson brackets (i.e., $q_ip_j – q_jp_i \neq 0$), which is to say that the ordering matters.1 This is exactly the relationship that the position and momentum operators (as they are in matrix mechanics) have, with the quantum of action $h$ (Planck’s constant) and (“h-bar”) $\hbar=\frac{h}{2\pi}$, with the imaginary unit $i$ (from complex numbers) thrown in:
$$ [\hat{q}_i , \hat{p}_j] = i\hbar\delta_{ij}$$
which is what gives rise to the famous Heisenberg Uncertainty Principle. This Principle states that it is not possible to simultaneously know both the position and momentum below a certain threshold value:
$$ \Delta q \Delta p \gtrsim \hbar .$$
I need to mention here, in passing, that $\hbar$ is an exceedingly small number, on the order of $10^{-34}$ joule-seconds, so quantum effects are not really a part of our normal everyday experience.
Thus, Heisenberg’s matrix/operator mechanics is the direct quantum analogue of Hamilton’s classical formalism, and is equivalent to Schrödinger’s wave mechanics in the sense that it makes the same numerical predictions.
This equivalence should not be surprising, really, since Hamiltonian mechanics is another way of doing classical mechanics, as is the Hamilton-Jacobi formulation. That is the essence of the LHS of the ladder of abstraction—the three levels above Newtonian mechanics are all re-formulations of Newtonian classical mechanics, albeit with distinct features that may make certain problems easier to calculate in one formulation than another. But they produce equivalent results.
These two approaches, Schrödinger’s wave mechanics and Heisenberg’s matrix/operator mechanics, both developed in the middle of the 1920s within a few months of each other, were the two main ways of doing quantum mechanics for over two decades. But, given that there is also a level 2 on the LH classical side of the ladder of abstraction, namely the Lagrangian formulation, it should not come as any surprise whatsoever that there would also be a quantum analogue to that approach.
Level 2 – Lagrangian
And there is. It is the path integral formulation developed by Richard Feynman in his PhD thesis in 1942 (Feynman 2005), and only later reported in the broader physics literature (Feynman 1948) after his work on the Manhattan Project during World War 2 (Feynman 1986). The back story to its development is fascinating. He used his Nobel prize lecture to describe how it came about (Feynman 1965),2 and it is an amazing insight into the thinking of one of the most brilliant and original physicists of the 20th Century (Bethe 1988).
However, the importance of the Lagrangian had been recognised earlier by Dirac (1933; reprinted in Feynman 2005). Having noted the correspondence between quantum mechanics and classical Hamiltonian theory in the Abstract of that paper, he then wrote the following:
it would seem desirable to take up the question of what corresponds in the quantum theory to the Lagrangian method of the classical theory. A little consideration shows, however, that one cannot expect to be able to take over the classical Lagrangian equations in any very direct way. […] We must therefore seek our quantum Lagrangian theory in an indirect way. We must try to take over the ideas of the classical Lagrangian theory, not the equations of the classical Lagrangian theory.
As a graduate student in the early 1940s, Feynman was thinking along the lines of trying to use the action $S$, when he was informed of Dirac’s paper. As related in his Nobel lecture, taking a comment by Dirac in that paper that two quantities were “analogous”, he sought to see what would happen if they were made equal. He soon found that he had to use a constant of proportionality to make it work, but that very soon after that, out popped none other than the Schrödinger equation! A very exciting discovery!
The classical Lagrangian approach uses Hamilton’s Principle of Least Action. In contrast to the Hamiltonian which is the sum of the kinetic and potential energies, $H=T+V$, the Lagrangian $L$ is instead the difference between them, $L=T-V$. The technique uses ‘generalised coordinates’ $q_i$, which live in a configuration space, $Q$, and their generalised velocities
$$ \dot{q}_i = \frac{dq_i}{dt}\,.$$
Configuration space has $n$ dimensions, where $n$ is the number of generalised coordinates, or the degrees of freedom, which can be up to $3N$, where $N$ is the number of particles (but can be less for constrained dynamics). The ‘state space’ (technically, the tangent bundle $TQ$ of $Q$) used to model the state of the dynamics has $2n$ dimensions, since it also utilises the $n$ generalised velocities $\dot{q}_i$ together with the $n$ coordinates $q_i$ to describe both positions and motions in the dynamical system.
The Lagrangian L is a function of the generalised coordinates $q$, generalised velocities $\dot{q}$ and time $t$: $L=L(q,\dot{q},t)$. The action $S$ for an individual trajectory $q(t)$ is found from the integral between an initial position $q_i$ at time $t_i$ and final position $q_f$ at time $t_f$ by way of:
$$ S[q(t)]=\int_{t_i}^{t_f} L(q, \dot{q}, t) \, dt\,.$$
(The square brackets denote that $S$ is a functional: a function of a function.) The Principle of Least Action looks for the unique (classical) trajectory (in configuration space, $Q$) arising from the dynamics described by $L$ such that the action is minimised (strictly speaking, extremised or made ‘stationary’, but this is almost always a minimum), so that the ‘variation’ (a formal mathematical procedure) in $S$ is zero:
$$ \delta S = 0.$$
That is, of all the possible (classical) paths that go from $(q_i, t_i)$ to $(q_f,t_f)$, only one makes the action a minimum, and this is precisely the path that (classical) Nature chooses. Carrying out this variation yields the so-named Euler-Lagrange equations that need to be satisfied along every point in the trajectory in order for the vanishing variation condition to hold.
The value of the action integral depends on two distant endpoints in time (and space), making the action a global, or non-local quantity. Thus, in order for the global condition to be satisfied (stationary action), the particles need to obey the Euler-Lagrange equations at each point along the trajectory (i.e., locally). These equations are the Lagrangian formulation’s equivalent of ordinary Newtonian equations in 3D, but the use of generalised coordinates significantly simplifies the analysis. Let’s denote the (minimised) action of the classical path $q_{\text{cl}}(t)$ as $S_{\text{cl}}$.
The quantum case is somewhat more complicated mathematically, if not conceptually. In a nutshell, instead of a single classical trajectory which minimises the action, it turns out that every possible trajectory contributes to the function that describes the dynamics.
To see how this works, recall that the main postulate of wave mechanics was that the action $S$ becomes the phase of the complex wavefunction $\psi\sim e^{iS/\hbar}$. In Feynman’s path integral approach (also called ‘sum over histories’ for this reason), as noted, every possible path in configuration space $Q$ contributes to the total ‘amplitude’ $K(q_f, t_f; q_i, t_i)$ (called the propagator) that describes the dynamics of a transition from $q_i$ at $t_i$ to $q_f$ at $t_f$. This is usually denoted
$$ K(q_f, t_f; q_i, t_i) = \int_{\cal{P}} e^{iS[q(t)]/\hbar}\,\mathcal{D}[q(t)]\, ,$$
where the total space of all possible paths in $Q$ from $q_i$ to $q_f$ is denoted by $\cal{P}$. Paths that are ‘far away’ from the stationary point of $S$ (i.e., where $\delta S=0$) will have phases that will oscillate rapidly and will interfere destructively so that their summed contributions effectively cancel out. For paths ‘close’ to the classical path where $\delta S=0$, the phases will be close to stationary and the contributions to the amplitude will thereby add constructively (or coherently). In the classical limit where $\hbar\to 0$, $q(t)\to q_{\text{cl}}(t)$, and the amplitude $K\sim e^{iS_{\text{cl}}/\hbar}$ which recovers the classical path of conventional Lagrangian theory.
It can be shown (Feynman and Hibbs 2010), that the propagator $K$ obeys the Schrödinger equation (recall that Feynman found it during his initial blackboard exploration), which confirms that it is equivalent to that formulation. And we know from earlier that the Schrödinger and Heisenberg formulations are also equivalent. So, we now have three forms of quantum mechanics to choose from: Schrödinger, Heisenberg, and Feynman.
I have to admit to a certain fondness and preference for the path integral approach compared to the other two because it allows me to visualise the transition in configuration space from the quantum sum over histories case to the classical single-trajectory limit much more easily. My physics investigation nose tells me (sniff sniff) that there is something important lurking there. But I’ll have more to say about that in a subsequent post.
Regrouping
If you were only familiar with quantum mechanics as a topic in isolation and not the basis it has in classical mechanics, then this might come as a surprise: three different ways to do quantum mechanics! Wow! But if you know about the classical background, as I have tried to show it, then this is not only not a surprise, but completely obvious, once the connection between the classical LHS and quantum RHS is made plain.
Let us now consider the point we have reached in this investigation. We now have the following identified correspondences between the classical LHS and the quantum RHS of the ladder of abstraction. Hamilton-Jacobi corresponds to Schrödinger (wave mechanics); Hamiltonian corresponds to Heisenberg (matrix/operator mechanics); and Lagrangian corresponds to Feynman (path integrals). So, all three re-formulations of Newtonian mechanics on the classical side have their counterparts on the quantum side. So far, so easy.
Let’s re-do Table 1 from the previous post with the new RHS cells filled in, to see what we have found.
| Classical Mechanics | Quantum Mechanics |
|---|---|
| Hamilton-Jacobi Equation Field $S(\vec{q}, t)$ on configuration space. |
Schrödinger (Wave) Equation Field $\psi(\vec{q}, t)\sim e^{iS/\hbar}$ on configuration space. |
| Hamiltonian Formalism Phase space $(q, p)$. Poisson brackets $\{A, B\}$. |
Heisenberg (Operator Mechanics) Operators $\hat{A}(t)$ on Hilbert space. Commutators $\frac{1}{i\hbar}[\hat{A}, \hat{B}]$. |
| Lagrangian Formalism Action $S = \int L \, dt$. Principle: $\delta S = 0$ selects a single path. |
Feynman (Path Integral) Sum over histories $\int_{\cal{P}} e^{iS/\hbar} \mathcal{D}[q]$. Principle: $\sum e^{iS/\hbar}$ sums all paths. Classical path emerges as stationary phase limit. |
| Newtonian Mechanics Particles in 3D space. Forces: $F = ma$. |
? |
The Next Step Down…?
This now leaves the research question as initially posed: what is the correspondence on the quantum RHS to Newtonian mechanics on the LHS? (You can see this musing going on in Figure 1 above.)
It turns out that this question, depending on a variety of considerations, leads to a variety of answers, which are (as noted above) simultaneously totally obvious and utterly ambiguous. Obviously! This is quantum mechanics, after all, right? Which answer is chosen depends a great deal on what considerations are given importance. And there is the rub! One of the ways to tackle this theoretical question is to now ask about what the experimental evidence tells us we can reasonably believe about what quantum mechanics is telling us about how the Universe behaves. And that will bring us to Bell’s Theorem (Bell 2004), and its many interesting implications, in the post after the next. The next post will try to look at what the ‘natural’ step ‘down’ might look like.
Next: Part III – An Option for the ‘Ground’ State
Notes
- I’m afraid I need to make a physics joke here. There is an old saying that one should mind one’s ps and qs, which apparently derives from the days of manually typesetting print. The letters, or ‘sorts’, are of course reversed so that when they are inked and pressed to paper, the letters come out correctly. Naturally, it is easy to confuse a p with a q, hence the expression. In quantum mechanics, of course, you also have to mind them. When I was a graduate student (many years ago) there was a saying that did the rounds of email signatures in the early days of the Internet. It was this: “Quantum mechanics is God’s way of getting you to mind your $p_i$s and $q_j$s.” And here I must cite Sean Carroll (2019, p.69), who said: “I think we can all agree that physics jokes are the funniest jokes there are.” ↩︎
- I used to have a passage from near the end of that lecture posted above my work area to remind me of the importance of not being afraid to be an outlier in research or to do things against the prevailing fashion. Having read Feynman’s book (1986) as a second-year undergrad—I even got the physics workshop techs to build a Feynman inverse sprinkler (pp.63-5) to test how it actually behaved—it’s probably why I pursued Einstein’s unified field theory as a research topic despite the fact that “everybody knew” it doesn’t work. People saying that just tripped my “Feynman hint”-ometer (see previous post), as described in the Foreword to my doctoral thesis (Voros 2002).↩︎
References
Arnold, Vladimir I. 1989. Mathematical Methods of Classical Mechanics. 2nd ed. Translated by K. Vogtmann and A. Weinstein. Graduate Texts in Mathematics 60. Springer.
Auletta, Gennaro, Mauro Fortunato, and Giorgio Parisi. 2009. Quantum Mechanics. Cambridge University Press. https://doi.org/10.1017/CBO9780511813955.
Bell, John S. 2004. “On the Einstein-Podolsky-Rosen Paradox.” Chap. 2 in Speakable and Unspeakable in Quantum Mechanics: Collected Papers on Quantum Philosophy, 2nd ed., with introduction by Alain Aspect. Cambridge University Press.
Bethe, Hans A. 1988. “Richard Phillips Feynman (1918–1988).” Nature 332 (6165): 588. https://doi.org/10.1038/332588a0.
Carroll, Sean M. 2019. Something Deeply Hidden: Quantum Worlds and the Emergence of Spacetime. Dutton / Penguin Random House.
Dirac, P. A. M. 1933. “The Lagrangian in quantum mechanics.” Physikalische Zeitschrift der Sowjetunion 3 (1): 64–72.
Feynman, R. P. 1948. “Space-Time Approach to Non-Relativistic Quantum Mechanics.” Reviews of Modern Physics 20 (2): 367–87. https://doi.org/10.1103/RevModPhys.20.367.
Feynman, Richard P. 1965. “The Development of the Space-Time View of Quantum Electrodynamics.” Nobel Lecture. December 11. https://www.nobelprize.org/prizes/physics/1965/feynman/lecture/.
Feynman, Richard P. 1986. “Surely You’re Joking Mr. Feynman!”: Adventures of a Curious Character. As told to Ralph Leighton. Edited by Edward Hutchings. Unwin / Counterpoint.
Feynman, Richard P. 2005. Feynman’s Thesis: A New Approach to Quantum Theory. Edited by Laurie M. Brown. With P. A. M. Dirac. World Scientific. Originally published as The Principle of Least Action in Quantum Mechanics (Princeton University, 1942).
Feynman, Richard P., and A. R. Hibbs. (1965) 2010. Quantum Mechanics and Path Integrals. Edited by Daniel F. Styer. McGraw-Hill. Emended edn. Dover Publications.
Goldstein, Herbert. 1980. Classical Mechanics. 2nd ed. Addison-Wesley Series in Physics. Addison-Wesley Pub. Co.
Masoliver, Jaume, and Ana Ros. 2010. “From Classical to Quantum Mechanics Through Optics.” European Journal of Physics 31 (1): 171–92. https://doi.org/10.1088/0143-0807/31/1/016.
Merzbacher, Eugen. 1998. Quantum Mechanics. 3rd ed. Wiley.
Messiah, Albert. 1999. Quantum Mechanics. Dover Books on Physics. Dover Publications. Originally published as Mécanique Quantique (Previously published in English: North Holland Publ; Interscience Publ; 1961-1962). Two volumes bound as one.
Norsen, Travis. 2022. “Quantum Ontology: Out of This World?” In Quantum Mechanics and Fundamentality: Naturalizing Quantum Theory Between Scientific Realism and Ontological Indeterminacy, edited by Valia Allori. Springer International Publishing. https://doi.org/10.1007/978-3-030-99642-0_5.
Shankar, R. 1994. Principles of Quantum Mechanics. 2nd ed. Springer US. https://doi.org/10.1007/978-1-4757-0576-8.
Voros, Joseph. 2002. “On the Electrodynamics of Einstein’s Non-Symmetric Unified Field Theory.” New edition. Reprinted with a Foreword, Afterword and further revisions. PhD thesis, Monash University, Dept of Physics, orig. subm. 1996. Copy on ResearchGate. https://www.researchgate.net/profile/Joseph-Voros/publications.