Is it possible for a layperson to suitably evaluate scientific disputes?

Question

As a layperson, I try not to fall victim to the Dunning-Kruger effect. As an example, one area where I know that I am vulnerable is when biochemistry intersects with nutrition and disease.

Is it possible for me, as a layperson without specific scientific knowledge (in this case, biochemistry), to make any sound or valid judgment regarding a scientific dispute (in this case, over nutrition)? For instance, the China study says X, critics A say not-X, and critics B say Y. Isn't any attempt by a layperson to evaluate any of those claims going to be based on guesswork and confirmation bias?

Answer 1

In examining any claim, you always have at least two options: checking validity and checking the truth. In layman's terms, this means you can:

• Examine the logical structure of the argument. Does the conclusion follow from the premises?
• Examine the content of the argument. Are the premises in the argument true?

In your case, it seems you lack knowledge of the content, but anyone who understands logic should be able to—even with no understanding of the topic being discussed—critique the logical structure of an argument and check whether it is valid. Your examination would be entirely free from the influence of confirmation bias because logical coherence has nothing to do with content but of form. On the other hand, if you were to question the truth of the premises in the claim, you would be susceptible to (but not necessarily fall prey to) the confirmation bias, because the confirmation bias deals with the truth/falseness of information.

Answer 2

Even a seasoned research scientist will be qualified to directly evaluate experimental data only in a small subset of all areas. Fortunately, the peer review system allows scientists to make informed estimations regarding the research that falls outside of their area of expertise.

The best case scenario in science is that the experimental data is so overwhelming in support of one theory, that even those previously opposed cannot now justify opposition; the proper conclusion is totally obvious and consensus in the scientific community is ~100%. That doesn't mean that the theory is correct, but it does mean that anyone other than an expert possessing new experimental data or rare insight would probably be irrational or unable to doubt the theory supported.

Typically though, uncertainty centers around the questions that have not been firmly settled. In such a case, it is generally not appropriate even for an expert in the area to draw firm conclusions. The simplest heuristic in this case, is to assign a probability to each theory, where each probability is biased by the level of consensus in the scientific community, and keeping in mind that possibly none of the theories are correct. The more expertise you have in the area, the more you can use your own judgment to modulate those probabilities.

Often though, an issue is too new or fringe to have attracted the critique of the scientific community, or that critique has not reached the public. In this case, there are a few questions that should be asked by non-experts (keeping in mind that this list is non-comprehensive).

How much research is there in favor of the claim? does that research stand in opposition to or in the face of a large body of other research? Does it seem likely that, if the claim were true, it would have gone unknown until now?

Who is promoting the claim, through which forum, and for what potential motives? (If they're not scientists, or it's not published in a scientific journal, be suspicious. Also be suspicious of non scientists or scientist non-experts referencing obscure research in which they were not involved.)

Does the claim offer an easy solution to any of life's problems, and is it monetized by the people who back the claim.

Typically, the best sign that a claim is legit, is when the experimental evidence in favor continues to mount, originating from multiple independent laboratories, along with the approval of the scientific community.

Answer 3

This is actually a very interesting epistemological question.

In Nyāya (classical Indian) epistemology, there are recognized four types of epistemological warrants (Pramāṇa-s):

1. Perception
2. Inference
3. Analogy
4. Authority

Now, with this in mind, let us take a case where you have a number of scientific studies with conflicting results, and you wish to form an opinion as to which is most reliable. Which of the aforementioned warrants could be used to lend credence to one or another study?

Clearly "direct perception" is unlikely, and from a scientific perspective would be likely to be viewed as anecdotal unless the observation was controlled, etc.,-- in other words, unless you were functioning as a scientist, which you stipulated was not the case.

Inference and Analogy could be useful for following individual claims, but at the same time, inferences and analogies are going to necessarily be based upon a process of reasoning from things you already believe to be true, which puts you in danger (as you point out) of confirmation bias.

Thus, oddly enough, we're left with basing our judgments upon the authority of others, and going with the study that comes from the source we judge to be most trustworthy. Which leads, of course, to the question of how we evaluate the qualifications of the authorities themselves-- which leads us back to the same list of epistemological warrants.

To cut to the chase: if you follow a foundationalist epistemology, you have some kind of axioms or first principles that you need to build up from (but it is likely that doing so would require you to actually become a scientist in order to make a decision in this case); if, on the other hand, you follow a coherentist epistemology, you adjudicate each new claim against the things you already believe to be true, which puts you in danger of confirmation bias.

Rock, meet hard place.

Answer 4

Yes, it's possible for a layperson to evaluate scientific disputes. The process, according to anthropologist Jennifer Raff, requires reading the science on the topic:

What constitutes scientific authority? ... What constitutes enough proof? Obviously everyone has a different answer to that question. But to form a truly educated opinion on a scientific subject, you need to become familiar with current research in that field. And to do that, you have to read the “primary research literature” (often just called “the literature”). You might have tried to read scientific papers before and been frustrated by the dense, stilted writing and the unfamiliar jargon. I remember feeling this way! Reading and understanding research papers is a skill which every single doctor and scientist has had to learn during graduate school. You can learn it too, but like any skill it takes patience and practice.

Her article, How to read and understand a scientific paper: a guide for non-scientists, describes this process in detail.

Answer 5

We know from experience that some questions are simple, some complex. What is the capital of Nebraska? What is 2+2? But others are more problematic: What is the cause of stomach ulcers? How many people must there be to guarantee that at least two of them have the same number of hairs on their heads?

Sometimes checking with our own eyes, thinking on that logically, being the scientist ourselves (but still not infallible in both sensation/method and judgement/logic) can lead to an answer, but sometimes we just don't have the resources to check; at some point we have to rely on the experience and judgement of others. At that point your own less expert judgement will be not about the original data but about seeing and judging the second hand and condensed information. All you have there is the experience of the contents of the report (which is not the data but a distillation of it) and the experience of the report itself, the author (reliability of similar reports by that scientist), even the publication method (the respect of the journal).

You'll have doubts about everything at every step (whether you are the original scientist or the second or third hand reader. But your experience (at each level) can help on judging reliability.

Is global warming human caused? Does evolution happen...sorry, does natural selection cause evolution? Even, does the moons path in the sky act on principles the same as a baseball? I don't have direct experience with any of these (except the baseball), but I can work with what others have told me so that I can 'suitably evaluate' them.

As to modern scientific journalism (reporting of scientific results in the current media), I am much more skeptical though; there's lots of (reported) bias, corruption, sensationalizing, story making.

So executive summary: yes, you as a non-expert can evaluate expert claims. It depends a lot on context (how much you already know, how good the report is); it's not as good as being omniscient, but it's better than nothing.

Answer 6

As Joseph has kindly pointed out, it depends completely on your level of knowledge in biochemistry and how much research and digging you are willing to do.

Answer 7

In general, anyone who has studied Logic debate should be able to form reasonably sound arguments, and suss out weak arguments. I consider myself a laymen in a lot of things, but I feel with a couple hours of research and my knowledge of Logical debate I can participate in any argument and form strong arguments. That is not to say that they are undefeatable, as I cannot have researched everything in the field, and there may be evidence that I have not considered, but the arguments on their own are reasonably sound.