Are presence questionnaires valid in a between-subjects design?

Question

Usoh et al. (PDF) applied presence questionnaires (which are usually designed to measure one's feeling of presence in a virtual environment) in both a virtual and a real world environment in a between-subjects design and found no difference in people's sense of presence between the two environments.

While the authors only argue that presence questionnaires are not valid unless they pass a "reality check", i.e. participants self-report a higher feeling of presence in the real environment, in my opinion, results might have been very different if they had employed a within-subjects design, exposing participants to both environments and thereby arriving at a relative sense of presence score for each environment.

If this were true, it would imply that such questionnaires are essentially not applicable in between-subject designs, as people would only be able to reliably rate their sense of presence in a particular environment relative to another environment. Hence I've recently asked whether it is valid to treat repeated measures data as both within- and between-subject data and compare the analysis of both to see if there are differences.

But my underlying assumption was questioned by some of the answers, and I posted a follow-up question to ask if it ever makes sense for between- and within analyses of the same data to differ.

By way of a very contrived example, assume I get subjects to rate on a scale of 1 to 10 how much they like an offer of free beer. In one condition, people are offered one litre of beer for free, in the second condition, people are offered two litres of beer for free. If I ran this experiment with a between-subjects design, I would personally assume that there isn't any discernible difference between the two conditions, because - hey, free beer! But as a within-subjects design, I think I stand a reasonable chance of seeing an effect, because more free beer is better than less free beer.

One comment has asked for the question to be migrated to here so, to give the whole thing a more cog-sci spin, my intent here is to question my (admittedly intuitive) assumption that people can not relate their absolute sense of presence to a number from 1-7, but can relate their relative sense of presence compared to another environment. Is there any research other than Usoh's paper on presence in particular, or are there similar phenomenon in other measurement instruments?

Answer 1

The underlying question here is: Does the context of a questionnaire affect the answer?

I think you already answered the question yourself. Yes it does.

Let's make the comparison even more extreme:

1. Please rate on a scale of 1 (not at all) to 7 (very) how happy you would be if you won 1 million dollars because you were the 1 millionth customer of a super market.
2. Please rate on a scale of 1 (not at all) to 7 (very) how happy you would be to win 100 dollars because you were the 1 millionth customer of a super market.

Now imagine you would have been asked only the second question. Will the answer on the second question be affected by the fact that there was question 1? Very likely.

For an excellent introduction to research about such context effects and other survey-related phenomena check out "Self-reports: How the questions shape the answers." by Norbert Schwarz (1999).

Schwarz summarizes several ways in which pragmatic aspects of questions affect answers in questionnaires. There is an important distinction with regard to how people construct the meaning of such questions. On the one hand, there is the literal meaning of the question, its semantic content. On the other hand, there is also the pragmatic meaning of a question, which involves an assessment about the intention of the person (the researcher) who has asked the question. (What is implied? Also called the implicature). To infer this pragmatic meaning, people rely on several communication norms or "maxims" (Grice, 1975).

The maxim that is most relevant here is the maxim of relation:

a maxim of relation enjoins speakers to make their contribution relevant to the aims of the ongoing conversation. In research situations, this maxim licenses the use of contextual information in question interpretation and invites respondents to relate the question to the context of the ongoing exchange. (Schwarz, 1999, p. 94)

So in the context of question 1 (1 million dollars/presence in environment 1/2 liters of free beer), the pragmatic meaning of question 2 (100 dollars/presence in environment 2/1 liters of free beer) may be interpreted as comparative (compared to the first option, who much do you like it?) and the answer may be different that than when it is asked in isolation.

Another way in which the context may affect the answer is that the implied meaning of the scale anchors change (What means "very happy" vs. "not at all happy"?). An example of research that has investigated such issues is the research on shifting standards in stereotyping (e.g. Biernat & Manis, 1994). For example, if you judge woman X with regard to aggressiveness, she may be rated as "highly aggressive" in the context of other women (because the stereotype is that women are low in aggressiveness). However, if the question is asked in the context of men, she might be rated lower in terms of aggressiveness (also because of the stereotype that women are less aggressive). The point is that the inferred meaning of the scale anchor "very aggressive" changes based on the context the question is asked in.

References

Biernat, M., & Manis, M. (1994). Shifting standards and stereotype-based judgments. Journal of Personality and Social Psychology, 66, 5–20. doi:10.1037/0022-3514.66.1.5

Grice, H. P. ( 1975). Logic and conversation. In P.Cole & J. L.Morgan ( Eds.), Syntax and semantics: Vol. 3. Speech acts (pp. 41– 58). New York: Academic Press.

Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105. doi:10.1037/0003-066X.54.2.93