Reverse-Engineering Quantum Mechanics, VIII.9.

A Pause to Take Stock

A pause to catch our breath and revisit why we went down the current pathway and how we got to where we are.

Framing

Given the feedback from the AI that judged the original version of the Part VIII post so harshly, I’ve decided to largely forego the language of Lévy-Leblond (1999) and try to stick to the more conventional (albeit problematic, in Lévy-Leblond’s view) terminology. When quizzed, the AI informed me that unfamiliar terminology is often associated with crank or pseudo-scientific work in its training data, so that is likely why it’s negative reading was triggered. Also, that non-standard heuristics are often flagged as anomalous, as well. Hence, a post that is using (long-established but) not widely used terminology, while also using a heuristic exploratory mode of investigation, seems to have led to a couple of these triggers. It did warn me, though, that the nature of its training data is based on how physics journal articles are generally assessed, so that a human accustomed to reading those journals is very likely to do something very much like it. Hence, the stance from here on is to reduce the friction of the terminology so as to focus on the actual findings, which I still find to be pretty interesting. Thus, to avoid such mis-classification by automated assessment tools in future, I am reframing these exploratory insights using the standard terminology while seeking to preserve the underlying empirical logic.

Re-Framing

This affords an opportunity to revisit the overall logic of this exploration, so as to re-frame what may have been an incorrect impression arising from reporting ‘on-the-go’ instead of ‘at-the-end’. Perhaps I should have explicitly identified these posts as the blogging equivalent of what are generally called ‘working papers’ in conventional research institutions. When exploring a topic area for clues as to how it might be extended, one tends to use all sorts of heuristics and tries all sorts of new ideas and pathways just to see where they go, in the hope that they might open up a new insight or idea or pathway (this was what Schrödinger did). This is tentative heuristic exploration, not rigorous deductive derivation. In the systematic techniques of lateral thinking that Edward de Bono introduced six decades ago (see e.g., de Bono 1995 for a mature explication of them), the heuristic used in any particular case is considered almost entirely secondary to the end result it produces, which result is the very justification for having used it in the first place.

This is a somewhat reversed form of reasoning from the more usual approach to thinking, which is to judge the result on the strength of the premise(s) and/or the robustness of each step in the train of logic. de Bono (1994) countered the overuse of this tendency by coining the term “water logic”, which is concerned with open-minded exploration and unimpeded inductive ‘flow’ (he called it “movement”), as opposed to “rock logic” which is hard-edged and concerned with rigorous categorical analytical judgement. The two words that capture the difference between these two forms of logic are: water: ‘to’ (as in ‘flows to’, promoting movement to newer ideas); and rock: ‘is’ (as in ‘this is’, a static judgement, which makes the thinking stop). Both are necessary for thinking, of course, but de Bono’s main contention was that rock logic is not particularly useful for exploratory thinking that is attempting to find a way forward from an existing state, because it tends to judge each step in turn, which may shut down a potentially promising avenue because there is no obvious or clear justification for it. By contrast, water logic allows forward flow not just without judgement, which is far too passive, but instead with active movement, so as to see where it ultimately leads. Only once a destination state is reached by movement-thinking should the judgement-thinking turn on again, to test the idea for robustness. Note that this not to be considered a free-for-all of irrational sloppy thinking, which he repeatedly admonishes against in his books. Rather, it is the careful use of different logics in their correct place and properly suited to the thinking task at hand, be it divergent, emergent or convergent.

I have been reporting on this process pretty much as it has been happening (“flow”), rather than waiting until the program is complete and at its final destination (“is”). That means that the heuristics I’ve used that guided a certain exploratory pathway might have been afforded more attention or weight than may be justified by a later reappraisal once at the destination, including a much clearer sense of the conventional (rock) logic that would lead to such a destination, but which most likely would not have been reached using just that logic. In other words, having got to where I have so far with this process, it’s now clearer to me how that could have been framed in a better way. But such re-framing typically only comes in retrospect, once the end-state has been reached and alternative pathways or arguments have been explored and investigated, and unfruitful ones discarded. It is always much easier to trace the course of a river by starting at the source and just flowing downstream. But in order to find it, you tend to have to get there by exploring upstream from the mouth, and having to examine very similar-looking branches that may or may not be the main river or simply just a tributary. This typically leads to many blind alleys and dead ends, requiring assessment and discarding of most of them. Such is the process of exploration. I remember reading an interview with (now Nobel Laureate, Sir) Roger Penrose many years ago that remarked that it was a shame we only tend to publish successes and not dead ends in our research programs, because access to those dead ends might save a lot of people a lot of wasted time if they knew about them and to avoid them. I can’t recall the place the article was published, but that idea has stayed with me ever since my grad student days, as has Murray Gell-Mann’s famous statement that the most important tool in theoretical physics is the wastepaper basket (Schultz 1985).¹

Now, where does this leave this investigation? Well, having reached this way station, it seems pretty clear that I may possibly have overplayed the variational heuristic and not focused enough on the empirical requirements—indeed, the absolute necessities—that have been provided by the long well-established results of experiment. That is, I may have focused a bit too much on the mathematical gymnastics rather than on the empirical imperatives that drove those gymnastics in the first place. In trying to ensure the mathematics was sufficiently rigorous, I may have lost sight of highlighting the primacy of the physics that was guiding it. And that meant that some readers (and an AI or two) could easily mistake the investigation being reported as just so much mathematical sophistry, instead of it being the necessary result of paying proper attention to what the universe tells us about how we need to think about quantum entities.

So, let me now lay out how to think about the recent posts, from VII onwards, in this series.

As I said in Part VI, the de Broglie-Bohm theory is, in my view, a perfectly workable replacement and cure for the incoherence of the Copenhagen interpretation, so long as it treats the wavefunction \(\psi\) in a way analogous to the function \(S\) in Hamilton-Jacobi theory—which is to say, as an abstract representation or description of the motions of the particles, rather than as a literal wave that actually actively guides them. Now, the notion of ‘guiding’ is admittedly a somewhat loose one. One could also imagine that the function \(S\) ‘guides’ the particles in Hamilton-Jacobi theory, in some kind of way, because of the way it allows us to model how they behave, and so to understand how they will move, which could easily seem like it could be doing something very much like ‘guidance’.

At least, that is what the physicists on Norsen’s (2022) hypothetical planet thought about \(S\) as an ‘orchestrating field’. Would that stance be so terrible for us? Perhaps not, provided it is recognised as a convenient abuse of language (“par abus de langage”, as one of my mathematics lecturers used to say), and not as the actual reality. But the trouble with abuses of language, convenient or otherwise, is that they can then all-too-easily become embedded as mostly-implicit faulty ways of reasoning about the referents that the language is supposed to represent. (Think of ‘particles’ vs Lévy-Leblond’s ‘quantons’, for example…) So, as a tool for representing or describing how the particles move, \(S\) and \(\psi\) are both extremely useful, and the tendency to ascribe the property of active guidance to what is simply passive description may therefore be very easy to fall into. But, they do not literally guide the particles, and we should keep that in mind. To think that they do is, as I see it, an error of dynamical attribution. Particles are primary and active, the modelling function is secondary, passive and descriptive, for both Hamilton-Jacobi and de Broglie-Bohm alike. We know why the particles move the way they do in Hamilton-Jacobi theory (i.e., Newtonian mechanics); but we haven’t the foggiest clue for de Broglie-Bohm theory.

That essential epistemic asymmetry (discussed in Part VI) between classical mechanics (we know both the referents and the forward mappings from 3D to configuration space) and quantum mechanics (we know neither) means that even with a workable interpretation, it never hurts to try on new 3D ontologies now and then just to see whether they might provide a simpler conceptual grounding while also potentially giving rise to predictions that are consistent with well-established experimental results. That was the motivation for the posts following Part VI.

What the recent posts in this series have tried to do is the reverse of the usual direction that is almost universally presented in the conventional pedagogy of quantum mechanics. Instead of starting with the wave equation and deriving consequences from it, I have sought to start from the physical observations of quantum entities and tried to derive the necessary mathematical formalism needed to correctly model them.

The key idea at the bottom of all this was to see whether a physicist in 1926 (i.e., 100 years ago) could have arrived at a formulation of quantum mechanics from these empirical observations, using the Lagrangian formalism as their basic mathematical starting point. This is in contrast to Schrödinger, who used the optico-mechanical analogy in the Hamilton-Jacobi equation and the de Broglie hypothesis to infer his wave equation, and Heisenberg, who used the empirical results of observations to develop an operator mechanics that essentially maps to the Hamiltonian formalism. Feynman (1942; see also 1965) of course did use the Lagrangian formalism 16 years later, guided by a remark from Dirac (1933) that gave him the essential starting point.

But I wanted to see whether it might have been possible at about the same time as the other two forms of quantum mechanics were being developed to similarly argue from the empirical observations of around that time using the Lagrangian formalism, which would imply a change to the usual assumption in the variational calculus, namely to relax the Dirichlet boundary condition. That is, to instead use the Extended boundary condition, which is motivated by the empirical result of a quantum entity having a non-zero interaction width set by the de Broglie wavelength, postulated in 1924 and which was already being observed in the experiments of Davisson and Germer in 1926 before being formally reported in 1927. That is why I have always come back to the empirical results—“the mathematics always needs to be guided by the physics”. And, if you do so, it seems you can indeed cook up a form of quantum mechanics based on the Lagrangian formalism that could have been developed in 1926 instead of 1942. So, this whole exercise is really an attempt at a sort of counterfactual history of the third (Lagrangian) form of quantum mechanics, rather analogous to the way I mused in an earlier post about how the history of quantum physics could have been very different in the 20th Century had de Broglie’s theory been accepted in 1927. This is how a futurist thinks about the dynamics of counterfactual historical contingencies, as a way to prime the mind to think about different scenarios as potential future contingencies.

It is this reversal of the usual pedagogical sequence, combined with trying to cook up a conceptual simplification that uses fewer and different starting axioms than in conventional quantum mechanics, that has so confused and apparently vexed the algorithms of the AI instances that I have had take a look at it (and have done so several times now). They have almost always complained that it doesn’t predict anything new than does standard quantum mechanics, and in fact one even complained that I used quantum mechanics to derive quantum mechanics. This latter is a remarkably asinine reading of what was going on, given that I was actually using empirical observations from known experiments to guide how to change the formalism of classical (Lagrangian) theory to arrive at a suitable formalism for quantum theory. When I pushed back on the AI that made this rather idiotic claim, and pointed out that this is exactly analogous to what Schrödinger himself did, it relented and apologised (as reported in the previous post). But the endless complaints of “no new predictions”, in pretty much every instance of an AI I have used, is remarkable. That metric must be baked so hard into the training data that even an attempt at a conceptual simplification that might be useful in teaching is regarded as not meriting anything other than a rebuke for lack of any ‘new’ empirical implications. It is only when pushing back on that stance that the AIs reconsider and admit that reformulations that provide conceptual clarity and possible heuristic guidance for further exploration might be useful (maybe even valuable) after all.

I’d have thought that not ending up with new physics that contradicts well-established experimental results, but instead recovering the same physics from a quite different logical starting point, and using established mathematical tools to do so, was actually a feature, not a bug. This is an attempted derivation from empirically-based first principles with a simpified ontology, not a proposal for a new theory. I’m not trying to change the laws of physics; rather, I’m trying to find a way to understand their origin better, by starting from the physical constraints forced on us by experiment, and seeing by how much we would need to change well-established classical mechanics to do so. Turns out, not by that much, despite what Dirac thought; and that’s actually the real source of surprise and novelty here, which I thought might be interesting enough to report on. However, in a later post I will detail several circumstances where the nature of the framework being explored here might be able to indeed make some predictions which could in principle be used to compare it with those of conventional quantum mechanics, in case there are any deviations.

The Basic Framework

Now that I have had a chance to reflect on the overall logic of the investigation, I can present it as a set of five conditions, adding one to the four I used in Part VIII to derive the Feynman propagator from first principles, and altering the ordering slightly.

The real fundamental driver here is the empirical fact that quantum entities display a finite interaction width given by the de Broglie wavelength, \(\lambda_{dB}\), proposed in 1924, the confirmation of which won him the Nobel Prize in 1929. The logical consequence of this empirical fact is that the mathematical Dirichlet boundary condition for classical point-particles is thereby incompatible for entities that do not interact like point-particles. (Recall from Part VIII that there is no claim being made that quantum entities are physically extended, merely that they have a non-point-like interaction width.) This requires relaxing the usual assumption used in modelling zero-width point particles in the boundary condition in order to properly take into account the non-zero interaction width, which thereby generates the requirement to use an extended boundary condition. The finite interaction extent of the quantum entity then means that more variations than just the conventional first (i.e., \(\delta S=0\)) for zero-width particles need to be taken into account in the dynamics, which thereby generates more terms and leads to the variational series \(e^\delta S\), no longer truncated at the first variation. The consequences of that finite interaction extension, as well as two other empirical observations (phase coherence and quadratic dispersion), also allows the derivation of the path integral propagator from the Lagrangian formalism, from which immediately follows the well-known standard derivation of the Schrödinger equation. Thus, in this way, the mathematics of path integral quantum mechanics emerges from the mathematics of classical Lagrangian mechanics via relaxing the conventional point-particle assumption due to the observation that quantum entities have a finite interaction width, scaled by Planck’s constant, \(h\), which is the only numerical input required, as well as by also paying attention to other experimental observations. No magic, just observation, and the appropriate mathematics based on observation, founded upon the long-established classical formalism.

The only thing more surprising than this is the consternation (and misunderstanding) it has caused AIs when asked to analyse it!

The five basic conditions that underpin this observation-guided first-principles structural derivation are:

You can see that preparations are being made for the further transition to the relativistic regime for Part X, in three ways:

1. by taking account of the different interaction widths depending on the regime: de Broglie for non-relativistic, Compton for relativistic;
2. by recasting ‘quadratic dispersion’, which is non-relativistic (and was used in Part VIII to derive the Gaussian form of the propagator), to the more general concept of ‘dispersion’ which differs depending on the regime; thus, this will be different in the relativistic regime from the non-relativistic quadratic form; and
3. identifying the underlying space—the geometric ‘arena’ in which the physics takes place—as a formal condition, because in each space there is a dimensionality, metric signature, and an associated Clifford Algebra in its tangent space for describing the internal degrees of freedom of entities which have extension (i.e., which are not idealised points). For 3D space, this naturally yields the Pauli algebra; for 4D spacetime, this naturally yields the Dirac algebra.

What may be surprising is that ‘spin’ emerges even in the non-relativistic regime, by way of the Clifford Algebra \(\cal{C}\ell_{3}\) of Euclidean 3-space \(\mathbb{R}^3\), which latter I have been loosely calling ‘3D space’ up to this point. This algebra is isomorphic to the Pauli spin matrices, which are one representation of that algebra (e.g., Hestenes 1966; Lounesto 2001; Doran and Lasenby 2003). The next post, Part IX, will deal with and present the results mentioned at the end of Part VIII.

Next: Part IX – Further Implications of the Transition