Evidence that epistemological anarchism by Feyerabend is wrong

Question

Feyerabend was critical of the scientific method and claimed in his book "Against method" that "anything goes". If I understood correctly he meant that there is no single scientific method,but he does not specify common components of every scientific method either,thus concluding that "anything goes". Are the following observations evidence that Feyerabend is wrong?

• The replication crisis demonstrated that a huge amount of finding in the social as well as natural sciences cannot be replicated. Isn't the solution to the replication crisis more stringent statistical methodology?

• Also Feyerabend never specified common elements of scientific methodologies such as observation, forming of hypothesis or data analysis which enabled the majority of scientific discoveries. There are common elements to all scientific methods, contrary to Feyerabend.

• One might find suitable theories through mathematical manipulation of existing theories. However these theories will be tested against the physical world, thus although one might argue to have found truth before testing, we cannot know if these theories really hold up to reality before testing them.

• Non-mathematical theorizing can discover "trivial" workings of reality. However this seems only to work for low hanging fruits and usually does involve at least observation of the physical world plus a lot of undocumented reasoning.

Answer 1

The most widespread opinion of Feyerabend is that his basic thesis is correct, that there IS no single method that applies to all of science, but that his inference to anarchism is an over extrapolation. There are instead multiple useful methods, each of which is generally useful across a range of different sciences. And this collection of methods have a familial resemblance that involves a respect for the pursuit of truth, fallibilism, and peer evaluation/critique.

However, the thesis of this question is incorrect. The replication crisis really has nothing to do with Feyerabend, and rigorous statistics will have no effect on it. One of the major features of the replication crisis is that experiment-experiment results tend to have mean solutions that fall well outside the expected range one would predict from the prior published test results. This was the surprising conclusion from multiple revisitations to historic drug effectiveness tests. More careful analysis of the prior test results would not have affected that the test-test variability is not well predicted by the experimental variability within a particular test series.

The basic cause of the replication crisis, is that science simply does not do enough replications. And repeated testing, in different labs, with slightly different protocols and experiment teams, is the best way for science to sort valid from spurious inferences. Revisiting the same question multiple times, demonstrably does not lead to the same conclusion each time. And one will not discover this, unless one actually does multiple replications of interesting results. An interesting corollary is that this principle applies to surprising UNinteresting results too. Failed experiments have not been the focus of the replication program, but the same effect of experimental variability likewise applies to them, where there are likely a large number of false negatives that were just due to bad luck in experimental variability.

There are a variety of science practices that lead to the replicability crisis.

• Almost all academic original research is from PhD programs. And every PhD awarded has to be for original work, which means that NONE of the primary output of academic research will be replications.
• Most journals will only publish "interesting" work -- IE only statistically significant, original, and with positive outcomes. IE, if a study has a negative or indeterminate outcome, or is a replication, it will generally be rejected for publication. This drastically skews the published literature toward innovative research rather than completing a database on a subject, an over-focus on statistical significance, when even real effects are very often not significant in every test, and a massive discouragement of ever doing replications.
• The best funded studies, with larger sample sizes, etc, are from commercial organizations. And they will generally only report results when they are commercially useful -- IE successful applications of a commercial product. This drastically skews the published literature on these commercial products.

One of the concerns of critics of science has been about "P-hacking" where researchers multiply reanalyze their data until they finally find an effect that meets a 95% P criteria. The focus on statistical rigor, and bayesian analysis, is intended to address this supposed problem. However, this "problem" is just a sociological reaction to the publication policy of journals. Until journals publish indeterminate and replication studies too, variations of "p-hacking" will continue to skew the science database.

And setting a more stringent criteria for publishing, which is the point of the Bayesian vs frequentist approach, will not do anything prevent the lucky statistical accident effect which is so much a part of the replication crisis. What it WILL do, is drastically increase the number of false negatives, where a real effect exists, but is never explored because the publication criteria for the first time it was observed prevented publishing the interesting hints that an indeterminate but suggestive initial investigation provided.

Answer 2

There are more than a dozen different explanations for the replication crisis. Different explanations require different responses. For example, publish or perish incentive structures wouldn't be addressed by encouraging psychologists to be more statistically rigorous. I list some other explanations on slide 6 of this presentation: https://drive.google.com/file/d/11LpcCZlPKg-kzK_WhCv6rsy7PXLI5-4v/view?usp=drivesdk

Computer simulations and formal mathematical models are used in fields such as climate science, evolutionary game theory, cosmology, and computational chemistry. The relationship between these models and observation is complex and often indirect, in part because these fields often study phenomena that can't be observed directly (eg, because of time scales or abstraction). The definition of "observation" has to be stretched very thin to fit a generalization like "all sciences use observation." There are similar problems with talking about hypotheses or mathematical theories. Paleontology and archaeology often can't operate in a hypothesis testing way because they can't produce data on demand. The big claims made by these historical sciences might be qualitative explanations of particular events (what caused the KT extinction?) rather than mathematizable generalizations.

In general, it's extremely difficult to find substantive commonalities across all scientific fields.

Answer 3

Agreeing with @Dcleve's answer, I would like to add to that.

Feyerabend's thesis(1) of "anything goes" means, among other things, that there is no a priori reason that throwing a coin in order to derive natural laws or decide actions is on the average worse(2) that what is done using scientific methods (where applicable). In fact there is a non zero chance this random walk on possible actions can arrive at more profound and deeper outcomes.

Also there is no a priori reason that effective cures for diseases cannot be found through, for example alternative conceptions of reality, like myths and story telling, than "narratives" based on molecular biology.

Another reason is that cultural hegemony should not blind us from alternative but equally effective conceptions of physical reality and methodology.

It is a fact that a given outcome may be arrived at through different routes.

So for these and other reasons Feyerabend's thesis cannot simply be dismissed. Even if one adheres to the scientific method, it is still beneficial to keep in mind that anything goes, as it can have a positive effect on scientific narrow-mindedness which can be detrimental to the pursuit of knowledge.

References

1. Against Method
2. NFL theorem