2

I am writing a publication in which I mention the fallacy of of first executing a method and then claiming the results are what was aimed for (regardless of the results).

This sounds very much like the story of the Texas sharpshooter fallacy

The name comes from a joke about a Texan who fires some gunshots at the side of a barn, then paints a target centered on the tightest cluster of hits and claims to be a sharpshooter.

but does not fit with the description

The Texas sharpshooter fallacy is an informal fallacy which is committed when differences in data are ignored, but similarities are overemphasized. From this reasoning, a false conclusion is inferred.

I find this weird, because the story, and thus the naming, does not fit with the description, but instead with the fallacy that I described.

By questions:

  1. Why was the Texas sharpshooter fallacy named this way if the story does not fit the meaning at all? Am I missing something or misinterpreting the intent of the story?
  2. Is there a name for the fallacy that I am describing?
Improve this question
0

1 Answer 1

Reset to default
1

The fallacy is about picking a "target" in retrospect after one already has the data (akin to drawing a target against the tightest cluster of bullet holes on the barn after one has already shot the gun to create them) and then calculating the probability that the data would all be so close to the target in the same way one would if the target had been predicted beforehand.

For example, if one wants to show that some food has a health benefit, one could take a sample of people who started eating that food and look at how they compared to a control group on a very large number of health variables. Even if the food has no causal effect on any of the variables, if one picks a large enough number of them to test, it may actually be fairly likely there will be some statistically significant difference between the control group and the test group on some variable, just by random chance (similar to the spurious correlations website which charts a large number of different variables and then only shows the ones whose graphs happen to "match" fairly well). And then if one picks the variable that has the largest difference between the test group and the control group--say, performance on a test of grip strength--and calculates in retrospect the "probability" that the two groups would differ so much on that variable under a null hypothesis, one may get a low probability and claim that this makes a case for rejecting the null hypothesis and saying the food was the cause of the difference.

I don't think the wiki's phrasing about "differences in data are ignored, but similarities are overemphasized" is very clear, but one could say in my example the "similarity" that's overemphasized is the way the members of the test group are similar to one another in having a statistically significant level of higher average grip strength, while the "differences" that are ignored are all the other variables where members of the test group aren't any more strongly correlated with one another on that variable than they are with members of the control group.

The wiki gets that particular phrasing from this list of fallacies which they cite, you can see the page for it here and the examples they give of focusing on similarities but ignoring differences, like a dating site that tries to claim two people are a great match by highlighting a few questions they answered similarly while ignoring all the other questions they didn't.

Note that when these types of examples are analogized to the Texas sharpshooter, the important thing is that the target is chosen in retrospect, it's not important to the analogy that the person drawing the target is also the one who "performed a method", i.e. shot the gun. If one sees a friend's car that has a bunch of bugs that have splattered on the windshield, and draws a target around the greatest cluster and then argues the bugs must be preferentially attracted to that part of the windshield, it would be the same fallacy. I don't think there's a name for the idea of a version of this fallacy where it's treated as important that the same person created the data by performing a method and then picked the target in light of the data, if that's what you're asking for.

Improve this answer
16
  • 1. So it is not about the (dis-)similarities of samples, but of the (dis-)similarities of variables? Seems like it in both of your examples. Then the "painting of the target" would be the claim that "yes, we wanted to look for grip strength", even though you did not have that in mind when you started testing.
    – Make42
    Commented Jun 15, 2020 at 15:43
  • 2. Your last paragraph reveals a misunderstanding, maybe I did not communicate well enough: I am not emphasizing the point who is developing or applying the method. My fallacy is the connection between method (shooting) and result (target), in contrast to having a connection between data (shooting) and a hypothesis (target). The point is that someone (anyone) develops a method (here clustering), then looks what one gets and during the evaluation says "what we get is what we wanted, so the method is great". Is this a Texas sharpshooter fallacy?
    – Make42
    Commented Jun 15, 2020 at 15:51
  • On 1), I'd say it's about the statistical relationship between the sample and the variables you test for in retrospect, like finding the test group are 'similar' in having better grip strength than the control group. On 2), I'm not clear what you are counting as a "method"--are you also talking about a method of generating data (akin to shooting the barn to produce bullet holes, or to setting up the test where one group eats a certain food while the control group doesn't), or just to a method of statistical analysis of preexisting data like clustering?
    – Hypnosifl
    Commented Jun 15, 2020 at 17:06
  • Also is it important to your question that the person actually claims the pattern they found was the one they were looking for all along? One could imagine a study like this where they just look at a bunch of variables and announce something like "eating this food was associated with better grip strength" without specifically claiming they were interested in grip strength from the beginning, but also not mentioning that they looked at a host of other variables when studying the data.
    – Hypnosifl
    Commented Jun 15, 2020 at 17:07
  • Regarding your question: I am talking about clustering (see the link in my comment). A clustering method assigns labels to previously unlabeled samples such that samples within the same cluster are more similar to each other and samples in different clusters are more dissimilar to each other. What that exactly entails depends on the definition of what constitutes a cluster and on the method. So clustering methods are a means to analyze data, but they also generates data, namely the labels. [...]
    – Make42
    Commented Jun 15, 2020 at 17:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .