What types of observations necessitate postulating more complex theories without violating Occam's Razor?

Question

This question is inspired by the related question How could both Occam's Razor and complex theories like Quantum Mechanics be correct at the same time?. The accepted answer reasonably argues:

The more complex and numerous the phenomena that are to be explained, the more complex the theories with ever more necessary entities that have to be proclaimed for explaining them.

...

The complexity of quantum mechanics and the like is simply due to the vast variety and distinctiveness of phenomena we encountered in experimental environments. They necessitate all these fields/entities to be included because without them, not all of these phenomena could be explained (or rather: described).

My question concerns the necessitate aspect. What types of observations necessitate postulating more complex theories? Specifically, the accepted answer mentions phenomena, not observations, but it is implicit that these phenomena are observed by observers. This raises the question: what types of observations justify an observer in postulating a more complex theory of reality than the one they previously held? For instance, consider people who have mystical experiences and claim encounters with what they perceive (i.e. observe) as actual mystical phenomena. As a result, they postulate a mystically enhanced theory of reality (related discussion). How should such cases be understood in the context of observations necessitating more complex theories?

---

See also: Is simplicity the most important criterion when choosing between theories?

Answer 1

Historically, newer theories have been required as a result of more representative, more accurate, more extensive and more reliable observations. Consider, for example, the observations that led to quantum theory, general relativity, the theory of evolution, plate tectonics, etc etc. There is a world of difference between a mainstream scientific theory of the sort I have listed above and the speculations of an individual about the implications of a personal experience unshared by others.

Answer 2

I think you're right to say that phenomena have implicit in them observers. A phenomena is something that is accessible to me through my faculties with or without the assistance of external tools.

All phenomena need to be explained within the theories we posit. The idea is something like this: we observe phenomena X1, we posit theory Y that explains X1, we test theory Y by generating observations of other phenomena X2 and X3. If either X2 or X3 would be inconsistent with Y, then we reject Y and pose Y' which is a modification that accounts for X2 and X3.

As NotThatGuy suggested, this process is something like:

1. Observation: I see Sarah walking long distances
2. Theory: Sarah does not own a car
3. Observation: I see Sarah driving one day when she had to go a long distance
4. Theory: Sarah must have rented a car
5. Observation: I see Sarah driving every day
6. Theory: Sarah must have bought a car

The thing to note is that the theories remain the simplest explanation of the set of observations. After 5., we could also assume that she rents a car everyday, but that wouldn't make economic sense and it seems improbable. So, we pick the simplest, most probable theory that accounts for the observations we made.

With mystical phenomena, the general understanding is that the possibility of our perceptual systems failing is greater than the possibility of a mystical reality. It is not about the observations per se.

I could grant that there is a person John who thinks he saw a giant flying demon. The thing is, historically when people have seen such demons without capturing it using non-perceptual monitoring systems, it has usually been cases of perceptual failure. We know that there are a number of mental illnesses that may cause hallucinations, or even general stress may cause it for some people. Therefore, the theory that John was hallucinating seems much more probable and is a simpler explanation.

Answer 3

Any given explanation automatically becomes more complex as you make more observations, at least / especially when those observations are unexpected or can't be predicted from the explanation, because you need to add things to your explanation to make sense of why you got those specific observations (or you can leave it unresolved, which makes for a more limited explanation).

So this could result in a new explanation overtaking an existing one in terms of simplicity, if those observations are expected or can be predicted under the new explanation.

• We had some observations that were unexpected under Newton's law of gravity, so we accepted a new explanation of general relativity which better accounts for those observations.
• We had some observations of electron movement that were unexpected under existing laws, so we accepted a new explanation of quantum mechanics which better accounts for those observations.

But this could also apply to beliefs in one's life more broadly:

• If you believe that Sarah doesn't own a car, it might be somewhat unexpected to see her driving a car day after day. Now you might posit that she is renting or borrowing a car, which could explain those observations. But this adds complexity, because there has to be some reason why she would rent or borrow. She might also tell you "this is my car", which would also not be particularly expected, and she could be lying, but one would need to explain why she would.

  At some point, it might be simpler to just conclude that she owns a car.

• As for the supernatural, let's say you accept some religion as an explanation for some things. This religion might posit that some all-powerful deity created the Earth 300 years ago, that the deity has the greatest love for sandwiches, cares about the well-being of sandwiches and really wants a relationship with every single sandwich, and every sandwich has some transcendent spirit.

  Now one might say it's unexpected if (a) we make scientific observations that seem to indicate that the Earth is far greater than 300 years old, (b) we observe many sandwiches being cut up and consumed with no intervention from this deity, (c) we observe sandwiches being constructed by forces in the natural world, (d) we see the parts they consist of and how these interact with each other, and (e) we observe sandwiches from every angle and at every level and we don't see anything that could be explained by a transcendent spirit, and things we thought to be a result of the transcendent spirit we observe to be strongly linked to things in the natural world.

  There may be various ways to try to make sense of those observations under the sandwich religion, but those would increase the complexity of the explanation. (If one changes the core claims of the religion, this may increase or decrease complexity, depending on whether this makes the explanation more contrived, or if one has to make sense of how these changes are compatible with one's initial reasons for accepting the religion as true.)

  These observations, and the increase in complexity, may lead one to rejecting the explanation that is the sandwich religion, and instead accept other explanations for the data.

  One might also imagine that the sandwich religion is the only hypothesis we have for some observations, and that we may have good enough reasons to reject the religion outside of this, leaving us with simply not having an explanation for some given hypothesis.

---

This is putting aside whether you had good reason to accept initial explanation in the first place, and whether it was the simplest or best explanation given the evidence you had, which is also an important consideration.

Answer 4

You should try to answer your own question by asking the question: which theory leaves more unexplained?

Every theory has something left unexplained, since one can always ask a further why. The question is: at which level do you consider a further why to not be appropriate? When someone creates a “mystical theory of reality” it’s usually without any details fleshed out. Hence, the way it actually works is a mere postulation and doesn’t tell you anything more than what we already know about the physical nature of these mystical experiences.

If a “theory” doesn’t tell us anything about how it does things nor does it predict any new things consistently, it is a sign that there is no reason to believe it. A theory is only “necessary” to explain something when it is obvious, in front of your sight, has compelling evidence for it usually through predictive power and other things, and simplifies the data in a neat way (which is why mathematical laws are so useful).

An explanation in a theory that is hard to vary in its details is also a sign of a good theory. For example, “goblins did it” can be used to explain anything. It fits all the data. A law such as Newton’s second law can only fit a certain set of observations. If suddenly observations broke that mathematical pattern, the law would be falsified.

The analogy with computer programs is very useful. Simply ask yourself this question: if all observations could be represented as a simulation from a computer program, would the currently proposed theory reduce the lines of code needed to generate the number of observations? If the answer is yes, then combined with additional evidence, it’s probably necessary. If not, you can probably throw it out or atleast not worry about it.

Answer 5

Relativity is more complex than Newtonian physics - research the experimental motivations for inventing the theory of relativity for a perfect example. From what I understand, Maxwell's equations were really important for it.

Answer 6

Any observation that is not easily dismissable as a trick, a con, an illusion or a delusion warrants closer investigation, not immediately a change to scientific paradigm.

Mystical reports have been analyzed often in the past without any useful result. And after so many fruitless but expensive investigation showing how easily various people are deceived one way or another, more investigations of yet always same kind of dodgy report is not considered an activity worth of science or philosophy.

This is not merely a rejection of superstitions, but also the embrace of all that science and technology have to offer. As an individual, becoming an expert in all sciences is already near impossible in this day and age, with libraries filled with writing about wonders that are confirmed and repeatable and part of the technology surrounding us. The technology empowering smartphones has so much research behind it that a single person would struggle to design or understand each aspect of the production of a smartphone, even if they studied their whole life. And that's just technology, on top of that there are social sciences, economics, history, the arts...

It's not reason that keeps some people clinging to superstitions about mystical experiences, it is desperation for something more meaningful than materialism. This however can be satisfied with deistic faith having no visible trace in this universe.

It's emotionally understandable that some people would like to experience real magic beyond that, true miracles, clairvoyance and signs from the other side, most humans have been there as children wishing the world was magic. But science and philosophy have moved on from that, and so do most people when growing up, around the time of high school.

The reliable approach to reality as a non-scientist is that if there were magic around to be witnessed, we would know it already and would use it on a daily basis like people use smartphones or cars. Just draw a glass of water from the tap, speak the magic word and enjoy some wine. Don't use the elevator, levitate between floors. You'd learn it just after walking before primary school. Don't get health insurance, don't wear a seatbelt, just rely on the local magical resurrection and amputation healing service. That's what a world would look like in which there was anything to "reports of the mystical" beyond tricks, deception, illusion or delusion.