25

A recent post on the Stack Overflow blog, Stack Overflow Isn’t Very Welcoming. It’s Time for That to Change., discusses the problems that members of marginalized groups may experience when participating on Stack Exchange. Most of the post focuses on how their background may lead them to perceive the same interactions differently than someone from a more privileged background, but one sentence and footnote appears to suggest actual racist/sexist bias in on-site interactions:

Those of us who have privilege, but care deeply about reducing bias should be uniquely positioned to help, but we struggle the hardest to recognize that we are (unintentionally) biased ourselves.²

² If you’re shaking your head thinking, “not me,” I’d encourage you to take these implicit bias tests, specifically the Race IAT and the Gender-Career IAT. If you’re like me, they’re going to hurt.

The post suggests that individuals take the Project Implicit "Implicit Association Test" to assess their own subconscious biases. I'm not sure whether it's suitable for that purpose or not.

Are Implicit Association Tests effective at determining racist/sexist bias at the single-sample level?

0

1 Answer 1

14

No, the IAT(or any other predictive test) does not reliably determine behavior on a specific person. A predictive test is not accurate at the individual level unless the criteria being measured matches perfectly with the criteria being predicted, and as shown below this is not the case for IAT's.

According to this meta-analysis of 122 research papers representing 14,900 subjects, implicit association tests can predict unconscious bias regarding intergroup behavior which would not have been predicted by conscious self-reporting of bias. However, all other forms of bias examined in the meta-analysis were predicted better through explicit self-reporting than through IAT's. Furthermore, a combination of implicit tests and self-reporting of bias results in better predictions than predictions made with an IAT alone for all types of bias.

Regardless of the type of bias being measured, overall the IAT's had a predictive validity of 0.274, and the explicit tests had a predictive validity of 0.361. This is far short of the '1' needed to perfectly predict individual behavior. A value of 0.35 is considered a decent value for predictive validity, so IAT's can be considered a below-average estimation of bias.


The Study: The meta-analysis aggregated 122 research papers that examined the predictive validity of both IAT's, such as the one mentioned in the OP, and explicit tests, such as self-reporting of a subject's conscious thoughts when presented with some situation. The overall results of the meta-analysis can be seen in the following graph:

Graph showing integroup unconscious bias but no unconscious bias in other areas

Essentially, for Black-White(and any two other groups) interracial behavior, such as whether a subject is more likely to associate weapons with black Americans or white Americans, IAT's offered a better prediction of behavior than explicit self-reporting of the subject's conscious beliefs. From the paper:

For 32 samples with criterion measures involving Black–White interracial behavior, predictive validity of IAT measures significantly exceeded that of self-report measures.

However, IAT's fail to predict bias in other areas, particularly in the fields of consumer products, political preferences, and clinical phenomena (phobias, anxieties, etc.):

In several other topic domains — especially consumer preferences, political preferences, and clinical phenomena — it was strongly evident that self-report measures predicted criterion variance not predicted by IAT measures.

Furthermore, when combined the predictive validity of both implicit and explicit tests increased when the IAT-explicit correlation (IEC) was examined:

The finding that IEC magnitude was positively associated with predictive validity for both ICCs and ECCs was expected from reasoning that, when IAT and self-report measures agree, the constructs that they measure will likely reinforce each other in determining behavior... Confirming this expectation, IEC magnitude positively predicted 29.9% of the variability of ICC effect sizes and 34.3% of the variability of ECCs (see Tables 4 and 5).

In short, unconscious interracial bias can be better predicted by IAT's than by explicit self-reporting, though other types of unconscious bias are better predicted by self reporting. Regardless of the type of bias, if the results of explicit and implicit tests match then bias can be reliably predicted. However, the predictive value for both is (far) less than 1, so a perfect prediction is not possible. Thus, an IAT cannot guarantee the bias at an individual level.

5
  • Thank you for the answer and putting its use in context. Would it be possible for you to help me understand what this means in terms of the confidence of its results for an individual/at the single-sample level?
    – Jeremy
    Commented Apr 27, 2018 at 20:05
  • 2
    @user1114: Added some clarification. Basically, no predictive test is perfect, and they cannot really say anything about a single subject. It would be like saying you getting a 1600 on the SAT will guarantee a 3.0 GPA in college, which is obviously not true. However, saying groups who get 1600 on the SAT tend to have 3.0 GPAs could be true.
    – Giter
    Commented Apr 27, 2018 at 20:24
  • 4
    Out of curiosity, I wonder how much these results are tied to the specific context where they are run: would we find the same effect on race reporting if we were to run the tests in Africa, for example? From a cursory look, the meta analysis does not really check for this, but the vast majority of the results they studied seem to be Western.
    – Sklivvz
    Commented Apr 27, 2018 at 22:04
  • 8
    I'm having trouble understanding how bias is apparently measured and predicted in the first place. You keep comparing these IATs to self-reporting, declaring one more accurate than the other in certain circumstances. Ok, that's fine, but I fail to see a baseline. How is bias measured outside of these to determine one is better than the other? I feel like there's a "begging the question" problem in here.
    – user11643
    Commented Apr 28, 2018 at 5:12
  • 1
    Since the quoted blog post specifically recommends the Race IAT for people who are "shaking [their] head thinking, “not me,”", and your cited study says that for "measures involving Black–White interracial behavior, predictive validity of IAT measures significantly exceeded that of self-report measures", it would seem that (that part of) the blog post is at least somewhat justified—more so than your TL;DR would suggest.
    – 1006a
    Commented May 11, 2018 at 16:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .