Reverse-Engineering Quantum Mechanics, IX.

Further Implications of the Transition

Higher-order variations, entanglement, spin, and the ‘measurement problem’

Positioning

In the previous Post I reflected upon the place we have reached in this counterfactual historical exploration of whether a physicist in 1926 might have been able to extend the classical mechanics of the time to produce quantum mechanics, in a way exactly analogous to how Schrödinger used heuristic exploration to derive the equation that bears his name from the classical Hamilton-Jacobi equation (Schrödinger 1926, 1928; Schleich et al. 2019). In our case, we are working within the (at that time, unexplored) framework of Lagrangian theory which, of course, Feynman (1942, 1948, 1965) used 16 years later as the basis for his path integral formulation of quantum mechanics, prompted by a remark by Dirac (1933). In that post I intimated that the use of the variational heuristic—the Extended Boundary Condition (EBC)—while useful, may have run its course, given that the truly core concept is that of a quantum entity having a non-zero, finite interaction length or width. Here I wish to place that heuristic into context before we move on to the further implications of the transition from classical to quantum mechanics implied by this empirical observation.

In Parts VIII and VIII.9, the foundational premise, prompted by a variety of experimental evidence, was framed as Condition 1: scale/width. That is, that quantum entities possess (as Lévy-Leblond argued in Part V) continuous extension \(\sigma\), and that that extension, in the non-relativistic regime, is of the order the de Broglie wavelength \(\sigma\sim\lambda_{dB}\), proposed in 1924, tested and verified in 1925-7, yielding the Nobel Prize in 1929.¹

This foundational premise then gave rise to two branches of exploration: the empirical and the variational. The empirical branch added two more empirical facts to Condition 1, namely phase coherence across the interaction width (as evidenced by interference and diffraction), and, in the non-relativistic regime, a quadratic dispersion relation between energy and momentum. When combined with the mathematical condition that requires any quantum description to reduce to the classical case in the classical limit, this was sufficient to derive, from first principles, the Feynman path integral expression for the propagator, based on an underlying ontological stance (with respect to the path summation) of inclusion by coherence, as opposed to Feynman’s exclusion by cancellation. Of course, the question of the physical interpretation of that summation is debatable, however the mathematical object so found is identical, and so the Schrödinger equation is thereby immediately derivable from it by the well-known standard approach (Feynman and Hibbs 1965; Shankar 1994; Derbes 1996).

The variational branch was premised on the observation that a quantum entity possessing continuous extension due to its finite interaction width cannot reasonably be modelled as a point-particle by way of the usual Dirac delta representation. Thus the customary Dirichlet boundary condition (variations vanish at the endpoints) in the variational calculus needed to be generalised to become an extended boundary condition (i.e., the EBC), wherein the endpoint variations themselves have a non-point-like finite width conditioned by the interaction extent of the quantum entity, namely the de Broglie wavelength. This then also implied that the higher variations in the variational calculus could not, as is customary, be reasonably ignored as negligible, giving rise to the idea of the Dynamical Response Operator (DRO) \(\mathscr{D}= e^\delta – \mathbb{1}\), which is a Taylor series of variations excluding the \(n=0\) term, that ‘extracts’ the dynamics from the \(n=0\) ‘seed’, which is the action functional itself. The variational approach was then used to also derive the Feynman path integral via a different pathway as a consistency check on the empirical derivation.

The EBC never appears in the empirical derivation of the propagator; it only appears in two rather more ‘heuristic’ derivations—the Heisenberg position-momentum phase-space symplectic inequality and in the variational derivation of the propagator itself. There is in retrospect a strong sense in which the EBC is not really a separate postulate but rather simply a derived mathematical consequence of the physical Condition 1, the finite interaction width of the quantum entity. There is also a very strong sense in which the finite interaction width already implies the need for the inclusion of higher variations anyway, without the intermediary step of the EBC, so that Condition 1 also leads to the DRO, as follows.

A point particle samples the action landscape only along a single (zero-width) trajectory, on which only the value and slope matter: the \(\delta S=0\) condition finds that extremum, and the job is basically done (although \(\delta^2S\) can be used to examine the stability of the congruence of paths closely related to the extremum, via the van Vleck Determinant). But an entity with a non-zero interaction width samples the action landscape over a region, not just at or along the zero-width trajectory. The Taylor expansion of that landscape beyond linear order—the curvature, torsion, and higher derivatives—in principle also contributes to the dynamics because the entity has a spatial extent over which these terms may vary. The entity therefore feels the wider shape of the action manifold, not merely its stationary part. Thus, the finite extension of the quantum entity implies the mathematical requirement to include higher variations beyond the conventional linear order which is sufficient for point-particle representations; hence, the DRO emerges as a mathematical requirement directly from the empirical non-zero interaction width of the quantum entity.

Therefore, the somewhat heuristic mathematical EBC, while useful for triggering the idea of pursuing the mathematical consequences of a finite interaction width, is not strictly a separate postulate, since it is a derived consequence of the empirically-established Condition 1, which yields the path integral by direct derivation from first principles and implies the need to include higher variations by way of the DRO.

The initial idea of relaxing the Dirichlet boundary condition came from noticing the form of the boundary term in the action functional for \(\delta S=0\) (see Part VII), and was then pursued in earnest owing to the empirical fact of a quantum entity’s finite interaction width. Similarly, the idea of looking at higher orders in the Taylor series of variations (which defines the Dynamical Response Operator \(\mathscr{D}=(e^\delta-\mathbb{1})\) came from noticing that the second variation \(\delta^2S\) already makes itself felt, albeit not necessarily obviously so, in quantum mechanics—somewhat out of sight but not exactly hidden—and has been noticed by researchers on several occasions.

\(\delta^2S\) in Quantum Mechanics: The Quantum Potential

The second variation \(\delta^2S\) is the ‘hidden in plain sight’ object in quantum mechanics. It is present under several different names, performing several different roles, and has been independently rediscovered on multiple occasions (e.g., the van Vleck Determinant; the WKB transport equation; the Gaussian fluctuation integral; the ‘quantum potential’). What has not been obvious is that these are all essentially the same object: the second term in the Dynamical Response Functional \(\mathscr{D}S=(e^\delta-\mathbb{1})S = \delta S + \frac{1}{2!}\delta^2S + \frac{1}{3!}\delta^3S + \ldots\) , becoming dynamically active, prompted by Condition 1—the finite interaction width of the quantum entity.

The simplest and most direct way to see this is by way of something we already examined in Part III: the equations of the de Broglie-Bohm theory which emerge from the Schrödinger equation when the decomposed Madelung ‘polar’ form, \(\psi=Re^{iS/\hbar}\), is substituted into it, where \(R\) is the amplitude and \(S\) is the action/phase, and both are real functions (Bohm 1952a, 1952b).

Two coupled equations emerge, from the real and imaginary parts of the Schrödinger equation (see, e.g., Holland 1993, sect. 3.2, who uses \(Q\) instead of our \(V_Q\)): \[
\frac{\partial S}{\partial t} + \frac{(\nabla S)^2}{2m} + V + \textcolor{blue}{V_Q(R)} = 0 \quad\text{\textcolor{blue}{quantum} HJE}
\] and \[
\frac{\partial R^2}{\partial t} + \nabla\cdot (R^2 \frac{\nabla S}{m}) = 0 \quad\text{continuity}
\] where \(V_Q(R) = -\frac{\hbar^2}{2m} \frac{\nabla^2 R}{R}\). The first equation, without the \(\textcolor{blue}{\text{blue}}\) term (the “quantum potential”), is the usual classical Hamilton-Jacobi Equation, while with the blue term it becomes the quantum HJE.

In the classical limit, \(V_Q(R)=0\), so the equation reduces to the (classical) HJE, while the continuity equation remains unchanged. In this case, the phase/action \(S\) governs the amplitude \(R\) by way of the velocity field \(\mathbf{v}=\nabla S/m\) in the continuity equation, but there is no mechanism for \(R\) to feed back into \(S\) in the HJE. The coupling in this case, therefore, is strictly one-way: \(S\to R\).

In the quantum case, however, \(V_Q(R)\) is no longer zero, and it is this (blue) term that distinguishes the quantum equation from the classical. It is constructed from the amplitude \(R\) which, in the semi-classical regime (see, e.g., Schulman 2005, chaps 12 and 13), is the van Vleck Determinant; i.e., the square root of the determinant of the Green’s function of the Jacobi operator \(\hat{J}\), which is the operator defined by \(\delta^2S\). So, the chain of influence from the second variation to the quantum potential runs as: \[
\delta^2S\to \hat{J} \to \det\hat{J}\to R\to V_Q(R)\to \text{modification of HJE}.
\] That is, the second variation produces the operator whose determinant gives the amplitude, which generates the quantum potential. The presence of the quantum potential now modifies the phase equation from the classical case so that the amplitude is now no longer merely passive but dynamic. This modification provides a mechanism for the amplitude to feed back into the phase, \(R\to S\), thus enabling an overall bi-directional coupling \(R\leftrightarrow S\) between the action and the amplitude in the two (now) coupled equations. This is \(\delta^2S\) feeding back into \(\delta S\) and so producing a bi-directional coupling \(R\leftrightarrow S\).

Schleich et al. (2013) identified precisely this as the defining feature of quantum mechanics. In their words (p.5374): “It is this mutual coupling between amplitude and phase that defines a quantum matter wave and ensures the linearity of the wave equation.” In the ‘classical matter wave’, this coupling is broken, and the HJE is independent of the amplitude. The one-way coupling yields a nonlinear wave equation, but the very strong two-way coupling yields the linear Schrödinger equation. That the strong coupling turns two nonlinear equations into a single linear one, through the combining of the functions \(R\) and \(S\) into a complex function, is a remarkably elegant result found by Schleich et al, which they then extended and explored further in later work (Schleich et al. 2015; and see especially Schleich et al. 2019 for four different routes to the Schrödinger equation).

The bidirectional coupling that Schleich et al. identify as the quantum signature is, in the DRO framework, simply the promotion of the second term \(\delta^2S\) of the Dynamical Response Functional \(\mathscr{D}S\) from a passive (classical) prefactor to a dynamically active term. When \(\sigma\to0\) (point particle), the second variation merely determines the (passive, classical) ‘amplitude’, which does not feed back into the phase, and this one-way coupling yields the nonlinear classical regime. When \(\sigma\sim\lambda_{dB}\) (quantum entity with non-zero interaction extent), the second variation becomes dynamical, the quantum potential \(V_Q\) activates, and the coupling becomes bi-directional, which yields the quantum regime and the linear Schrödinger equation.

Thus, \(\delta^2S\) was always in quantum mechanics from the start, and noticing this is what prompted the idea that perhaps higher variations might also be present or necessary in the dynamics, by way of the Dynamical Response Operator acting on the action functional.

Higher Variations and the Catastrophe Hierarchy

The second variation is the first term beyond the customary conventional classical truncation at \(\delta S=0\), and its dynamical activation produces the transition from (non-relativistic) classical mechanics to quantum mechanics just shown. But, the DRO does not stop at just the second order. The finite interaction extent of quantum entities implies that even higher variations than the second may be needed to correctly model the dynamics. In particular, in cases when \(\delta^2S\) degenerates, e.g., at caustics, higher variations must then come into play, and that is the contention of this (fairly brief) section. It is brief because the bulk of this pioneering research—catastrophe theory applied to quantum mechanics—was done decades ago, so all that is really necessary here is to report on that work, and show that it is already implied by the structure of the DRO framework. What is not present in that earlier work is the overarching framing that shows why the caustic-catastrophe hierarchy takes the form it does. The DRO structure is precisely that frame.

A particularly accessible exposition of this is given in Chapters 13-16 of Schulman (2005), where he begins by discussing the semi-classical WKB approximation. This approach retains the first two terms in an expansion based on powers of \(\hbar\): the classical action \(S_{\text{cl}}\) in the phase and the van Vleck Determinant in the prefactor. In terms of the DRO, this is a truncation of \(\mathscr{D}S\) at \(\delta S+\frac{1}{2}\delta^2S\). Even though this is an ‘approximation’, it is widely useful because it is exact for Lagrangians that are at most quadratic. Thus, for a quadratic action, \(\delta^n S=0\) for all \(n\ge3\), so the DRO series terminates at second order, and the WKB ‘approximation’ is thereby an exact form, with no higher terms to ‘neglect’ as part of the ‘approximation’. The free-particle case, for example, is a case in point. It is also partly why it was possible to derive the Feynman path integral from first principles in Part VIII (via the empirical derivation), with the quadratic dispersion relation being the initial starting point.

In the non-relativistic regime, since the kinetic energy is always quadratic, any deviations would arise from a non-quadratic potential. For ‘well-behaved’ potentials, for which higher variations are in general non-zero, the classical trajectory is well-defined, the fluctuation amplitude is small, and the higher variations are largely suppressed by comparison. However, the approximation breaks down at caustics, where classical trajectories would converge, overlap or coalesce. In this instance, the Hessian of the action (i.e., the matrix of second-order derivatives of the action) develops zero eigenvalues and the van Vleck Determinant either vanishes or diverges. This makes the Gaussian fluctuation integral around the classical path cease to be valid, and thus the second variation degenerates.

In the DRO framework, when \(\delta^2S\) degenerates, then \(\delta^3S\), which was always present in the series, is naturally promoted to leading order precisely by dint of the degeneration of the preceding term. If \(\delta^3S\) degenerates, e.g., by the coalescing of two caustics, then \(\delta^4S\) becomes the leading order, and so on. In general, if a term should become degenerate, then the next term in the series becomes dominant, exactly as the Taylor expansion requires. In this view, the necessary mathematics is already anticipated, and needs neither diagnosis nor casting about for a way forward.

In the 1970s, Berry and coworkers (1972; 1976; 1980), and numerous others, established, using catastrophe theory, that each order of degeneration maps precisely to a canonical diffraction integral by way of the Thom-Arnol’d classification of catastrophes (Thom 1999; Arnol’d 1975), the first few of which are shown here:

The series continues beyond these entries to higher variations (up to at least 8th), but those catastrophes lie beyond Thom’s initial classification scheme, have varying and sometimes inconsistent nomenclature, become increasingly exotic, and so are correspondingly less likely to be encountered in physical systems. However, the main point to be made here is that the DRO framework predicts that this (unbounded) hierarchy exists, whether or not the higher variations are commonly seen, and the \(1/n!\) coefficients are predetermined. Schulman (2005) provides some more references to the original literature as well as an overview of how the lowest of these catastrophes were solved (in particular, via Airy functions).

Intertwinement and Entanglement

In Norsen’s (2022) chapter, discussed in Parts I, II, and VI, the physicists on Norsen’s hypothetical planet have noted a peculiar coupling between particles in the case of two masses joined on a spring in classical mechanics (recall that they have discovered the Hamilton-Jacobi equation in the absence of knowing the underlying Newtonian physics). This is a situation where the ‘orchestrating function’ \(S\) (i.e., the action in the HJE) is (additively) non-separable, which is to say that the function cannot be separated into individual contributions from each mass, and the motion of one mass directly influences the motion of the other. This results in what the physicists and philosophers on that planet call ‘intertwinement’. What is troubling to them is that this function lives in a higher-dimensional configuration space, not real 3D space, so the vexing question arises of how it can thereby affect the motions of particles in 3D space. Norsen is drawing an analogy to the issue of ‘entanglement’ in quantum mechanics in our world. On our planet, we recognise that classical intertwinement is not at all mysterious but simply stems from the coupling in the potential \(V\), and that configuration space is merely a mathematical encoding space which faithfully represents the actual dynamics in 3D, which we understand quite well. But the scientists on Norsen’s planet are not aware of this underlying ontology, so for them it is mysterious.

Norsen then makes the point that we here on our own planet find ourselves in a somewhat similar situation: we have a function, \(\psi\), which is somehow able to describe the motions of (quantum) entities faithfully (the predictions are all experimentally verified), yet it lives in an abstract, higher-dimensional configuration space, not the real 3D space that both we and the entities inhabit. He is suggesting that, perhaps with a suitable understanding of the underlying ontology or dynamics, entanglement for us might also not be mysterious, or at least, not as mysterious. Let us now examine the twin cases of intertwinement and entanglement to see how they are related. We will be able to use the findings of Schleich et al. in order to do so. Unfortunately this will not resolve the question of the underlying ontology, but it does at least provide a modicum of light on how the two situations are connected, as well as perhaps a clue towards what sort of 3D primitive ontology might be suitable.

As noted above, the (classical) HJE is nonlinear; the amplitude \(R\) is ‘enslaved’ to the action \(S\) in the continuity equation; in the semiclassical limit it is proportional to the van Vleck Determinant; and it carries no independent dynamical information. In essence, (classical) non-separability (intertwinement) comes from the coordinate coupling in the potential \(V\), as is shown in detail in Norsen’s chapter.

As Schleich et al. have shown, there is a single structural change between the classical and quantum regimes, namely the activation of the amplitude as a dynamical (no longer passive) factor that now feeds back into the action. As we have noted above, this is the (at least) second variation \(\delta^2S\) of the DRO becoming active and giving rise to the quantum potential \(V_Q\) (blue term) which affects the action by way of the full quantum HJE. This single structural change leads to a cascade of consequences. The first of these is that the two nonlinear dynamical classical equations above can be combined into a single linear quantum equation, which is, of course, the Schrödinger equation. This resulting linearity enables the superposition of solutions, which means that \(\psi = c_1\psi_1 + c_2\psi_2\) is a valid solution whenever \(\psi_1\) and \(\psi_2\) are. The possibility of superposition also means that \(\psi\) can be multi-valued in a well-defined way: e.g., the semi-classical propagator becomes a sum over multiple classical trajectories (such as in Feynman’s approach), each with its own action and amplitude. The cross-terms between these trajectories provide a possible mechanism for entanglement for superposition-type states. And finally, the existence of a dynamical feedback loop between \(R\) and \(S\) means that the structure of \(R\) (which may be non-separable) generates a \(V_Q\) that is non-separable, which may modify \(S\) in a way that goes beyond what \(V\) alone can produce (such as in the classical case), which is another possible channel for entanglement. Thus, the activation of \(\delta^2S\) in the dynamics—giving rise to the bidirectional coupling of phase and amplitude—ultimately provides a mechanism for (two possible forms of quantum) entanglement that moves beyond what is found in (classical) intertwinement.

This dynamical effect from \(\delta^2S\) enabling quantum entanglement is worth exploring in a bit more detail. In the two-entity case, where \(q_i\) is the \(i\)th entity’s coordinates, the Hessian matrix has the structure \[ H =
\begin{bmatrix} \frac{\partial^2S}{\partial q_1^2} & \frac{\partial^2S}{\partial q_1\partial q_2} \\
\frac{\partial^2S}{\partial q_2\partial q_1} & \frac{\partial^2S}{\partial q_2^2}
\end{bmatrix}.
\] The off-diagonal terms \(\partial^2S/\partial q_1\partial q_2\) and \(\partial^2S/\partial q_2\partial q_1\) in the Hessian encode the mixed curvature of the action manifold, which is how the action responds to simultaneous changes in each entity’s coordinates. They are the mathematical vehicle, as it were, for the cross-entity coupling. In the classical case, they determine how the van Vleck Determinant couples the particles’ degrees of freedom. This is intertwinement. In the quantum case, they appear in \(V_Q\) which enables the amplitude at one quantum entity to influence the phase at the other. They are the structural signature of non-separability, and are present in both regimes. What changes is whether they participate in the bidirectional feedback loop between amplitude and phase.

In short, intertwinement is a classical correlation of particles via $V$, while entanglement has a classical correlation through $S$ plus the amplitude-phase reciprocity of the quantum entities, which involves $R$ through the quantum potential $V_Q$. The DRO framework provides the mechanical explanation: the reciprocity is the dynamical activation of $\delta^2S$ by Condition 1. Without Condition 1, $\delta^2S$ remains passive and you have intertwinement, whereas with Condition 1, $\delta^2S$ becomes dynamical and you get the possibility of entanglement. The non-separable action supplies the channel for cross-entity coupling, but while it is necessary, it is not sufficient for entanglement, because without $\delta^2S$ activation it yields only classical intertwinement. Conversely, the bidirectional amplitude-phase coupling produced by $\delta^2S$ activation is also necessary but not sufficient, because without a non-separable action there is nothing to entangle with. The two conditions are co-necessary: together they permit (the possibility of) entanglement, but do not guarantee it, because that depends on initial conditions and dynamics.

Note that this co-necessity applies to the generation of entanglement, not to its persistence. Once created, an entangled state evolves (as usual) under unitary dynamics, which may be such that the entanglement is preserved without necessarily a continuing non-separable interaction or active \(\delta^2S\) coupling. That is, the conditions are necessary for production, but not for maintenance. Similarly, the DRO framework identifies one sufficient pathway for entanglement (non-separable action and bidirectional coupling), but it does not exclude the possibility of alternative pathways via the higher-order variations or other potential mechanisms.

So, in sum, \(\delta^2S\) provides the mechanism of quantum entanglement, but does not do so directly. Rather, as noted above, it produces the bidirectional coupling of phase and amplitude, which produces linearity, which allows superposition, which allows entanglement. Schleich et al. found the diagnostic (bidirectional coupling), while the DRO provides the mechanism (activation of \(\delta^2S\) by quantum entity extension → linearity → superposition → entanglement, given a non-separable action).

Of course, a question now arises from this observation, which is: what sort of underlying primitive ontology may be implied by the amplitude-phase coupling required to produce quantum entanglement from classical intertwinement? That would be an interesting avenue to explore in future work.

Spin

There is (or at least was) a misconception that spin (intrinsic angular momentum) is a relativistic effect. Maddox (1999, 72–73) claims that Pauli argued that spin is Nature’s way of demonstrating the correctness of relativity. It certainly arises from the derivation of the (relativistic) Dirac equation. But perhaps this is another historically-contingent mathematical discovery that has been mistaken for physical necessity, because four decades after Dirac, Lévy-Leblond showed (1967) that spin-\(\frac{1}{2}\) also arises naturally from the projective representation of the Galilei group alone, without any need to invoke relativity. The Galilei group is the symmetry group of non-relativistic spacetime (i.e., 3D Euclidean space plus time, which is not the same as what we conventionally call 4D spacetime). By linearising the Schrödinger equation in the same heuristic manner that Dirac linearised the Klein-Gordon equation, he arrived at what is now called the Lévy-Leblond equation, which reproduces the Pauli equation for spin-\(\frac{1}{2}\) particles without invoking relativity at any point. Thus, spin is already present in the structure of 3D Galilean spacetime, and does not need to be ‘imported’ from the relativistic sector in any way at all.

David Hestenes, starting in the 1960s, developed what he called Space-Time Algebra (later called Geometric Algebra), which is essentially a particular form of Clifford algebra using exclusively real (not complex) numbers (Hestenes 1966, 1986, 2003, 2017). I came across his 1966 book as a graduate student nearly 40 years ago and was thoroughly enchanted by it; in fact, I still have it on my bookshelf.² Specifically, he showed that the Pauli algebra is the natural algebraic structure for 3D Euclidean geometry, \(\mathbb{R}^3\). It also ends up being what is called the even subalgebra \(\mathcal{C}\ell^+_3\) of the spacetime (Dirac) algebra, \(\mathcal{C}\ell_{1,3}\). The key point that caught my attention then was that the Pauli matrices, introduced by Pauli to incorporate spin into the Schrödinger equation, are simply matrix representations of the three orthonormal bivectors (i.e., oriented planes) of 3D space, and are isomorphic to the Clifford (i.e., Pauli) algebra of \(\mathbb{R}^3\). Spin arises as an oriented plane of rotation intrinsic to the particle’s state, so in this geometric view, it is a bivector, not a vector. In this picture, the Dirac algebra \(\mathcal{C}\ell_{1,3}\) is the 4D spacetime generalisation, while the Pauli algebra is what you get when you restrict to the spatial (even) subalgebra by choosing a specific timelike direction. The relationship is formally structural, in that any even subalgebra is fixed within the overarching algebra. Thus, in this geometric view, the Pauli and Dirac algebras are intimately structurally related, because space and spacetime are intimately structurally bound.

This is important because the foundational empirical condition, Condition 1, means that the quantum entity should not be treated as nor represented by a point. A point has no internal orientation, but since a quantum entity has (at least) spatial extension, there needs to be a way to properly model this in terms of an appropriate mathematical representation. The empirical fact of extension does not necessarily require the existence of spin, but it allows for the possibility of it, since an extended entity in 3D space has access to internal rotational degrees of freedom (i.e., the bivectors of \(\mathcal{C}\ell^+_3\)) that a point particle simply does not. So, the careful, physically-motivated, mathematical argument is that the existence of spin is permitted by the quantum entity’s extension, but is not mandated by it. Whether spin is actually present depends both on the entity being represented as well as the representation chosen. That it can be present depends on the geometry that extension makes available, as well as what the physical entity is (e.g., a spin-\(0\) entity vs a spin-\(\frac{1}{2}\) entity).

In Part X, moving from 3D Galilean spacetime to the proper 4D Minkowski spacetime (of special relativity) replaces \(\mathcal{C}\ell^+_3\) with \(\mathcal{C}\ell_{1,3}\) and the de Broglie wavelength \(\lambda_{dB}\) with the Compton wavelength \(\lambda_C\), as foreshadowed in the Table near the end of Part VIII.9. The even subalgebra \(\mathcal{C}\ell^+_3\) that produces the Pauli-Schrödinger equation in 3D is the restriction of the full Dirac algebra \(\mathcal{C}\ell_{1,3}\) to its spatial sector. The same five-condition template reported in the Table in Part VIII.9 that produced the Feynman path integral propagator in Part VIII (and thus the Schrödinger equation, via the standard derivation from the path integral) should, with the domain and algebra changed, produce the Dirac equation, with spin already built into the geometry. That is one of the main goals of Part X, the exploration of which is under way.

Measurement

It is fair to say that the ‘measurement problem’ plagues all interpretations of quantum mechanics. Each has a way of explaining measurement, but all have some aspects that remain problematic. So, attempting to resolve this issue is a, if not the, major challenge for any attempt to formulate quantum mechanics, no matter the route taken to do so. I am not at all certain whether the below will be an acceptable ‘solution’ to this issue, but I can at least take comfort in the fact that no one has ever really managed to do this particularly well so far. So, let me state the way that ‘measurement’ is conceived in this view, and cross my fingers that it might be slightly less problematic than some of the others…

In this view, a ‘measurement’ is—as it is in, say, de Broglie-Bohm theory—just another interaction between quantum entities, one of which is part of a detector apparatus of macroscopic scale, and so has a very constrained and highly localised interaction extent. As you may have noticed in earlier posts, I do not hold with the manner by which the Copenhagen interpretation attempts to deal with the measurement issue: Bell’s (2004) diagnosis is more than sufficient to divert attention from that direction. I have only really managed to just skim the decoherence literature (see, e.g., Schlosshauer 2007, 2019; Zurek 2022; Bhaumik 2022), but it seems to hold what seems to me to be a most promising direction for resolution of the measurement problem (again, this is a ‘physics nose’ intuitive leap only, like the DRO was). Note that this discussion is very preliminary, to say the least, but what has emerged is as suggestive for the measurement problem as the DRO has shown itself to be for the caustic catastrophe hierarchy discussed above. It remains to be seen whether this pans out in any useful way, but there is at least one (very recent) experimental finding that supports the basic DRO expansion framework perspective as a way to diagnose what are considered gaps in the usual approach to decoherence.

It comes down to a structural similarity between the DRO framework and the cumulant expansion of the Feynman-Vernon (1963) influence functional which features prominently in decoherence theory (see also Feynman and Hibbs 1965, sect. 12.8). In decoherence theory (see, e.g., Schlosshauer 2007, 2019), the quantum-to-classical transition is explained by way of the quantum system becoming entangled with the broader environment (sometimes called the ‘bath’), a process sometimes also called “environmental monitoring”. In this view, the so-named ‘pointer-states’ of an apparatus corresponding to measurement outcomes are defined by “einselection” —environmentally-induced-superselection (Zurek 2022). A variety of approaches are found in the wider decoherence program, including Strong Broadcast Structures (SBS) and Quantum Darwinism (Le and Olaya-Castro 2019; Korbicz 2021). The Feynman-Vernon influence functional is used to encode the effect of an environment that has been integrated out of the system’s reduced dynamics (Breuer and Petruccione 2009).

Most of the decoherence literature I’ve encountered seems to stop at second order in the influence functional, i.e., the Gaussian approximation. This is the Born-Markov regime, and within it the theory explains pointer-state selection and the emergence of ‘classicality’ quite well. However, the outcome problem (that is, why this result is found rather than another) has not been resolved. But as shown by Aurell et al. (2020, Appendix C), if the bath is an-harmonic (i.e., non-Gaussian) the influence functional can be expanded in ‘cumulants’, that is, higher-order connected correlation functions of the environment. This is somewhat analogous to having non-quadratic potentials in the WKB approximation. In this expansion, each non-zero cumulant appears as a kernel in the \(n\)-th order term, with the familiar \(1/n!\) coefficients that accompany any exponential Taylor series. This is essentially the same structure as the DRO expansion shown above.

Both hierarchies have the same \(1/n!\) coefficients. Both have a similar structure where higher orders are supposed to matter when lower-order approximations break down. When the (Gaussian) WKB approximation broke down at caustics, the result was the caustic catastrophe ladder, with its catastrophes matching up with terms in \(\mathscr{D}S\). In the influence functional case it would (analogously) be the breakdown of the (Gaussian) Born-Markov approximation, with the tantalising possibility of a similar hierarchy emerging. I cannot be certain whether this similarity is deep or merely combinatorial; both are perturbative expansions of an action, after all, and might be expected to share the same organisational principle. Still, there is something suggestive here. If the higher-order terms in the influence functional are physically real, then the standard truncation at second order may be discarding physics that is relevant to the outcome problem. I am far from sure that this is the case, but it’s definitely worth noting that Breuer et al. (2002) derived fourth-order corrections that modify the pointer basis, and, much more recently, Xia et al. (2025) experimentally observed the fourth-order cumulant in a solid-state spin bath. This is extremely interesting. It may turn out that the higher-order terms only refine decoherence rates without addressing the issue of outcomes and their selection. But the hypothesis needs to be put forth. Hypothetico-deductive science needs hypotheses to work, so here is one which is now empirically accessible and testable, and not merely philosophical. The tools now exist to measure these higher-order cumulants, and to potentially examine whether outcome-related physics lives somewhere beyond the Gaussian truncation.

That is the connection the DRO makes to the measurement problem. It obviously does not solve the problem, because no-one has managed that. But, like the case of the caustic catastrophe hierarchy above, it does provide a structural diagnosis of where the gap lives (the truncation at second order), what the next terms would be (the higher-order terms with \(1/n!\) coefficients), and when they become relevant (when the second-order term is no longer able or sufficient to model the dynamics). Whether that physics exists, and whether the DRO framework helps, remains to be seen.

I suppose the fact that this is now a concrete prediction will make any AIs that read this post quite happy and less likely to rebuke it for not making any new predictions. In fact, you could also argue that the DRO framework also predicts even higher-order corrections for the caustic catastrophe situation, even though the sensitivity required to do that may be some way off yet. So, the DRO framework has now ‘graduated’ (as it were) from being simply a historical reconstruction or reformulation of quantum mechanics, to something that can at least diagnose where an open problem might live. This was the case, retrospectively, for the caustic catastrophe hierarchy. But it might also, prospectively, be such for the measurement problem. The experimental methodology to probe higher-order cumulants now exists, and the hypothesis that outcome-related physics might reside beyond the Gaussian truncation is now testable. I can hardly wait to see what is found…

A Glimpse Ahead

In Part X we will examine the change of regime from non-relativistic to relativistic, which was foreshadowed in the previous post, Part VIII.9. It is then that the basic framework I’ve been using to guide this investigation will (hopefully) come into its own, a key part of which has been the Dynamical Response Operator (DRO) defined as the Taylor series of the variational operator \(\delta\) with the \(n=0\) term excluded, since that forms the ‘seed’ from which the rest of the terms \(n\ge1\) ‘extract’ the dynamics inherent in the \(n=0\) action functional (as discussed in Part VII).

The key idea is that of an action manifold, \(\mathcal{A}=e^\delta S\), where in Part VII this was shown in expanded form \(e^\delta S = S + (e^\delta – \mathbb{1})S = S + \mathscr{D}S\), showing both the ‘seed’ \(S\) and the Dynamical Response Functional \(\mathscr{D}S\).

Of course, the action functional is defined differently depending on the physical situation and regime within which it is being modelled, so the functional integral itself needs re-writing in terms of the domain \(\Omega\) on which it is to be evaluated, the measure \(\mu_\Omega\) which is associated with that domain, the Lagrangian \(\mathscr{L}\) appropriate to the physics being modelled, and the boundary condition \(\partial\Omega_\sigma\). Here the subscript \(\sigma\) represents the idea of the extended boundary condition (since we have been discussing the quantum entity’s finite extension \(\sigma\sim\lambda\)). Then, the full framework takes the explicit form: \[
\mathcal{A} = \sum_{n=0}^\infty \frac{1}{n!}\; \delta^n \!\! \int_{\Omega}\mathscr{L}\; \mu_{\scriptscriptstyle\Omega}\, \Bigg|_{\partial\Omega_\sigma}
\] which can be rendered rather more compactly as: \[
\mathcal{A} = S + \mathscr{D}S = e^\delta \!\! \int_{\Omega}\mathscr{L}\; \mu_{\scriptscriptstyle\Omega}\, \Bigg|_{\partial\Omega_\sigma}\;.
\] Classical mechanics emerges from this framework by choosing \(n=1\), \(\mu_\Omega=dt\), the boundary condition is such that \(\sigma\to\delta\), where \(\partial\Omega_\delta\) is the Dirichlet boundary condition (i.e., ‘delta’ for the letter D for Dirichlet but also for the representation of a particle by a Dirac delta), and the Lagrangian is defined over the appropriate domain, \(\Omega\), here being (coordinate) time \(t\). Thus, classical mechanics can be rendered: \[
\delta\int_{t_i}^{t_f} \mathscr{L}\, dt
\] with Dirichlet boundary condition in use. In this way, the Euler-Lagrange equations emerge from the bulk terms, as usual, along with any non-zero boundary terms if the variation is not assumed to vanish.

Keeping the Dirichlet boundary condition and not truncating the series at \(n=1\) yields, in the first instance, an active \(\delta^2S\), which is exactly what turns classical mechanics into quantum mechanics, according to the investigations of Schleich et al. (2013, 2019), as shown above. It seems to be the case that this boundary condition yields the semi-classical WKB approximation, but that for full QM, it seems to be necessary to utilise the EBC as well as keeping the full variational expansion. That was what we did in order to derive the Feynman path integral from both the empirical and variational routes in Part VIII.

Now, when we come to (special) relativity in Part X, the underlying space changes from \(\mathbb{R}^3\) to Minkowski spacetime \(\mathcal{M}\), and the tangent space Clifford algebra changes from Pauli to Dirac. The measure will become the proper time \(\tau\), and the domain will be the world-line of the quantum entity. The Lagrangian will then need to be chosen to model the physics. Given that the dispersion relation is no longer quadratic in momentum in relativistic physics, it will be interesting to see what emerges when we try to write an action functional for a quantum entity, re-do both the empirical and variational derivations of the propagator (as in Part VIII), and then seek to extract the required equation from that. That exploration is what awaits next time…

Next: Part X – To Infinity and Beyond: The Timelike Sector