How are theories selected?

Question

My motivations for asking this question are philosophical but I think this is a question that is best answered by the physics community.

There is a problem in the philosophy of science called the 'underdetermination' problem. There are two versions.

1. One is the idea that for any set of data, theory choice is underdetermined. There could be multiple theories which explain the data in the same way and we don't yet have the ability or the data required to choose between them on completely empirical grounds.

2. This is the idea that two theories could predict exactly the same thing in all situations but use different unobservable entities to explain it.

As far as I know (early on in the process at least) Lorentzian interpretation of SR vs Einstein's interpretation of SR could be compared in this way. Both made similar or the same predictions but many favoured SR in terms of its simplicity, not having to posit the existence of an unobservable ether. (Obviously later the empirical differences were evaluated and the ether theory had to be modified to account for its failed predictions and slowly lost favour but these empirical considerations aren't really the point of this post).

There are three main questions from this:

1. What are the main criteria that physicists approach when deciding between two theories that make the same predictions or which have yet to be decided up on experimentally?

2. Is it even possible to construct empirically identical theories which posit different unobservable entities to explain phenomena? Is there any reason to believe that this is impossible?

3. In practice, does this ever actually happen?

P.s. It seems as though the conflict between the Lorentzian and Einsteinian interpretations (at least early on) was an example of this but I may have some details about the history of these ideas wrong. As far as I know, the Lorentzian interpretation was able to be modified to fit with experimental failures but was later rejected on the grounds that it was much more complex (I assume this means it required many additional assumptions?) and therefore was rejected by the community at large.

It does seem however that both interpretations posit some unobservable thing. In the Lorentzian case, it seems as though he outright assumes the existence of an ether. In the Einsteinian case, there is some unobservable warping of space time. I say this warping is unobservable because you can only observe the effects of it, though this may also be wrong. So if anyone wants to clear up any misunderstanding I have about the important differences between the ether and Einstein's idea of spacetime warping, please do. Similarly, if I'm wrong to say that only the effects of spacetime warping can be observed, please correct me. Although, at this moment in time, I don't know what it would mean to directly observe the warping of spacetime since spacetime is the canvas on which we make observations??

The tag 'popular-science' is here because I'm not studied in theoretical physics very much. I have just completed A-level physics (somewhere in between high school and university by US standards) and wish to study theoretical physics at university which is why I find the problem so interesting and I used the Einstein vs Lorentzian argument as the example for this question because during my study I did a report about this argument and about simple derivations of Lorentz transformations etc... so it's an argument that I'm more familiar with than others.

I suppose a similar debate would have been during he original development of kinetic theory as kinetic theorists posited the existence of atoms which were unobservable at the time in order to explain certain phenomena which could have been explained with some other kind of unobservable (maybe, I don't know)?

Answer 1

To address,

What are the main criteria that physicists approach when deciding between two theories that make the same predictions or which have yet to be decided up on experimentally?

first and foremost it should be noted, without experimental evidence favouring one, the community will likely continue developments to some extent on all viable competing theories, naturally.

If there is no experimental support available, two significant factors are relied upon by theorists:

1. Aesthetic appeal: Does the theory require constraints to be imposed by hand or do they arise naturally? Does the theory possess notable symmetries or mathematical properties?
2. Theoretical consistency: Is the theory in agreement with other established theories, either exactly or in some limit? Is the theory self-consistent?

The latter is easier to demonstrate quantitatively, while the former relies to some extent on a subjective view of mathematical beauty or appeal, though constraints put in ad hoc by hand for example would be a potential red flag.

An example of a 'beautiful theory' is certainly general relativity, in the sense provided by Lovelock's theorem: for $d=4$, up to second derivatives in $g_{\mu\nu}$, the choice $\mathcal L = \sqrt{|g|}\mathcal R$ is the only option.

As for consistency, take your favourite tree level scattering amplitude from quantum electrodynamics, and in the classical limit, it should correspond to the cross section for the process evaluated using non-relativistic quantum mechanics.

Answer 2

The answers above are fairly comprehensive, so I will just add a point by way of analogy which I think gives an intuition for why physicists (and scientists in general) prefer simpler theories and tend to be more inclined to believe they are somehow fundamentally correct if they give the right predictions.

It is common in science to have to perform "curve fitting", where one has many sets of measurements (to give a concrete example, let's say we have 50 measurements of the force between two newly discovered particles and the distance between them), generally with some noise and measurement error, and must determine how well they agree with some model. In our example, we might have a model of force where $F(r) = \frac{a}{r^2}$. Here we have a single parameter, $a$, so we have to answer two questions - which value of $a$ makes the model best fit our measured data, and once we've found it, how well does it fit? Usually it won't fit perfectly even if the model was correct because the measurements are never perfect, but generally the better the fit the more confident we can be in the model. We may compare two models by asking which can be made to better fit data.

However we must be careful with this line of reasoning, because of a phenomena called "overfitting". Let's say someone else comes along and claims that the force is described by a 100th degree polynomial, i.e. $F(r) = a_{100}r^{100} + a_{99}r^{99} + a_{98}r^{98} + ... + a_2r^2 + a_1r^1 + a_0r^0$. As it turns out, given any set of $n$ pairs of measurements, we can perfectly fit that data with an $n$th degree polynomial, so that means our 100th degree polynomial can be perfectly fitted to our 50 measurements.

But obviously this doesn't mean the model is correct, in the case of our two new particles, because we could have always fit those data-points regardless of the underlying physics. This is known as "overfitting". This is also why most judgements of how well a model agrees with data don't just account for how closely it fits, but how many degrees of freedom the model had to fiddle with.

In general when discussing physical theories we may not have lots of easy to count numerical parameters. Whether one theory or the other has greater or fewer metaphorical parameters is non-obvious, and two theories where one seems more complicated may turn out to be mathematically, or even conceptually identical. Thus physicists are forced to rely on an intuitive sense of "elegance", a gut feeling that says a theory is "simple", "inevitable", that it is either right or wrong and can't be easily modified to fit observations that disagree slightly. I believe Feynman once summed up the feeling when he said something like "You can't put imperfections on a perfect thing, you have to come up with a new perfect thing instead". Perhaps one day we will figure out how to quantify this idea, but for now we're stuck with our guts.

Answer 3

To further answer

Is it even possible to construct empirically identical theories which posit different unobservable entities to explain phenomena?

Another example of the underdetermination problem is in the interpretation of quantum mechanics. Here you have several competing theories of how nature behaves on small scales, which are all purportedly consistent with the predictions of QM (at least so far...). These include the many-worlds, pilot wave, transactional, and Copenhagen interpretations. These theories offer dramatically different visions of the universe, but by construction they have the same experimental predictions.

So how does one choose an interpretation? As mentioned previously, aesthetic appeal, simplicity, consistency, and utility for predictions are often important. But since QM requires such a large departure from our everyday intuition, I think that someone will also choose an interpretation which offends their sensibilities the least. For example, if you really want determinism but don't mind non-locality (i.e. spooky action at a distance), choose Bohmian pilot-waves. But if you'd rather save locality at the expense of mild retrocausality, go with the transactional interpretation. If you don't like to think about it and would prefer to select a theory based on familiarity and ubiquity, choose Copenhagen.

Personally, I like how the transactional interpretation gives you the Born probability rule "for free", in the sense that it is not needed as a postulate (as it is for every other interpretation); rather, it is "justified". This enhances the transactional interpretation's simplicity and, in my mind, aesthetic appeal.

Answer 4

If you want a 'cop out' answer to your number 2, there's always 'chance', that could cause your empirical observations to 'look similar' to one another. Of course, repeated measurements would make such less likely by each measurement. But it does mean it's possible! (meaning there's no reason to believe it's impossible!).

Secondly, to take a bit of a philosophical / tongue-in-cheek approach to this. What you're asking is in sort to ask for a limit to our 'imagination', since wouldn't you agree that this is what's used when coming up with 'the possible reasons for the unobservable'.

I also don't find it too hard to believe in there being 2 equally flawed interpretations that both coincide with one another (in their observational effects).

lo and behold, some bad gravity 'interpretations' examples!

1. gravity is a small particle that when it collides with objects, it doesn't slow down (it passes through the object), and deposits 'negative kinetic energy' (for some reason?, further theorycrafting required).

2. gravity is a fully connected graph of invisible & untouchable & unbreakable rubber bands, with its 'relaxed' state at the 'origin point of the universe' (where the big bang happened :p ), everything is connected! (even photons, since we can see gravity bending light!). Also... the bands get weaker the more they are stretched... even to the point of pushing away after a certain length?

^ those are pretty bad theories, but... you can imagine that they would produce 'similar observable outcomes' (at least... to a certain point, and the cracks start to show... I guess... or maybe the patched rubber bands theory is the way it works :p).

---

In the end, I guess we can only really apply the occam's razor principle as a loose measure of believability for a theroy / idea.