How many test participants to 'prove' an issue?

Question

Its often said that 5-10 usability testing participants is what you should aim for to identify the bulk of issues in a system. There's loads of research behind this. This is the right way to do usability testing, its about uncovering issues rather than proving anything.

However...it is often demanded from stakeholders that you provide proof that things are an issue. That your participants having trouble with something are not just a handful of stupid people out of their userbase of half a million.

I wonder, is there any way maths can be brought into usability testing analysis on top of the useful stuff of identfying issues and their likely causes to provide numbers for the liklihood of a person encountering an issue?

Yes. If I have 10 participants and 8 have the issue I can say its a 80% chance, but thats relying on the people I tested with. How can this be blown up to take into account the size of the overall user base? Would the direction to take be a separate calculation on my 80% to show the liklihood of my participants actually being representative users?

Answer 1

You asked if there is any statistics that can be used to essentially determine if the 10 people you are testing with are especially dumb. Essentially what you're asking is:

What is the likelihood that my sample deviates significantly from the population?

Looking at it simply you could assume all your users (let's say 500,000) are drawn from a normal distribution of intelligence. That would tell us that roughly 68% of the 500,000 (340,000) are within one standard deviation of the mean. Let's assume (somewhat arbitrarily) that anything below one standard deviation below the mean is considered dumb, that means any time you pick a person at random there's a 16% chance they are dumb (100-68 = 32, but half of those would be unusually intelligent). 16% is pretty high, until you start getting more and more people involved.

• 1 person = 16% chance they're dumb

• 2 people = 2.56% chance they're both dumb

• 3 people = 0.4% chance they're all dumb

• ...

• 5 people = 0.01%

• ...

• 10 people = 0.000001%

You get the idea. Each time you draw another sample the odds are less and less likely. (The values above are, 100 x 0.16^n, where n is the number of people and 0.16 is the decimal representation of 16 percent).

EDIT:

I figured I could add a little more to this. I thought I'd add a little more detail to the latter part of your question. You said if 8 our of 10 participants have an issue, how can this be blown up to take into account the whole user base. We can't simply say it's 8 dumb users so 100 * 0.16^8, because there isn't just 8 dumb users, there's also 2 smart users. There's a few ways to handle this, but I find this the most intuitive:

If we have 10 users, and we assume the distribution we discussed above (dumb is more than 1 standard divination below the mean) then we'd expect there to be between 1 and 2 dumb users per 10 people (16% = 1 or 2 out of 10). What's the chance we get 8 out of 10 as dumb users? Well we could first say what the chance we get 8 dumb people, followed by 2 smart people - 100*(0.16^8 * 0.84^2) or 0.00003% chance. But that's not really correct, because here we're saying that we specifically must get the dumb people as the first 8, and then the smart people as the last 2. What if we got 1 smart person, 8 dumb people and then 1 smart person again? Well that would be 100 * (0.84 * 0.16^8 * 0.84) which again gives us 0.00003%, but it's a different allotment of the possibilities so these percentages add up. So now the question is, how many arragements of 2 and 8 are there? There are 45. There's a few different ways to work out the 45, but I like the slow way:

1. ['s', 's', 'd', 'd', 'd', 'd', 'd', 'd', 'd', 'd']

2. ['s', 'd', 's', 'd', 'd', 'd', 'd', 'd', 'd', 'd']

3. ['s', 'd', 'd', 's', 'd', 'd', 'd', 'd', 'd', 'd']

4. ['s', 'd', 'd', 'd', 's', 'd', 'd', 'd', 'd', 'd']

5. ['s', 'd', 'd', 'd', 'd', 's', 'd', 'd', 'd', 'd']

6. ['s', 'd', 'd', 'd', 'd', 'd', 's', 'd', 'd', 'd']

7. ['s', 'd', 'd', 'd', 'd', 'd', 'd', 's', 'd', 'd']

8. ['s', 'd', 'd', 'd', 'd', 'd', 'd', 'd', 's', 'd']

9. ['s', 'd', 'd', 'd', 'd', 'd', 'd', 'd', 'd', 's']

10. ['d', 's', 's', 'd', 'd', 'd', 'd', 'd', 'd', 'd']

11. ['d', 's', 'd', 's', 'd', 'd', 'd', 'd', 'd', 'd']

12. ['d', 's', 'd', 'd', 's', 'd', 'd', 'd', 'd', 'd']

13. ['d', 's', 'd', 'd', 'd', 's', 'd', 'd', 'd', 'd']

14. ['d', 's', 'd', 'd', 'd', 'd', 's', 'd', 'd', 'd']

15. ['d', 's', 'd', 'd', 'd', 'd', 'd', 's', 'd', 'd']

16. ['d', 's', 'd', 'd', 'd', 'd', 'd', 'd', 's', 'd']

17. ['d', 's', 'd', 'd', 'd', 'd', 'd', 'd', 'd', 's']

18. ['d', 'd', 's', 's', 'd', 'd', 'd', 'd', 'd', 'd']

19. ['d', 'd', 's', 'd', 's', 'd', 'd', 'd', 'd', 'd']

20. ['d', 'd', 's', 'd', 'd', 's', 'd', 'd', 'd', 'd']

21. ['d', 'd', 's', 'd', 'd', 'd', 's', 'd', 'd', 'd']

22. ['d', 'd', 's', 'd', 'd', 'd', 'd', 's', 'd', 'd']

23. ['d', 'd', 's', 'd', 'd', 'd', 'd', 'd', 's', 'd']

24. ['d', 'd', 's', 'd', 'd', 'd', 'd', 'd', 'd', 's']

25. ['d', 'd', 'd', 's', 's', 'd', 'd', 'd', 'd', 'd']

26. ['d', 'd', 'd', 's', 'd', 's', 'd', 'd', 'd', 'd']

27. ['d', 'd', 'd', 's', 'd', 'd', 's', 'd', 'd', 'd']

28. ['d', 'd', 'd', 's', 'd', 'd', 'd', 's', 'd', 'd']

29. ['d', 'd', 'd', 's', 'd', 'd', 'd', 'd', 's', 'd']

30. ['d', 'd', 'd', 's', 'd', 'd', 'd', 'd', 'd', 's']

31. ['d', 'd', 'd', 'd', 's', 's', 'd', 'd', 'd', 'd']

32. ['d', 'd', 'd', 'd', 's', 'd', 's', 'd', 'd', 'd']

33. ['d', 'd', 'd', 'd', 's', 'd', 'd', 's', 'd', 'd']

34. ['d', 'd', 'd', 'd', 's', 'd', 'd', 'd', 's', 'd']

35. ['d', 'd', 'd', 'd', 's', 'd', 'd', 'd', 'd', 's']

36. ['d', 'd', 'd', 'd', 'd', 's', 's', 'd', 'd', 'd']

37. ['d', 'd', 'd', 'd', 'd', 's', 'd', 's', 'd', 'd']

38. ['d', 'd', 'd', 'd', 'd', 's', 'd', 'd', 's', 'd']

39. ['d', 'd', 'd', 'd', 'd', 's', 'd', 'd', 'd', 's']

40. ['d', 'd', 'd', 'd', 'd', 'd', 's', 's', 'd', 'd']

41. ['d', 'd', 'd', 'd', 'd', 'd', 's', 'd', 's', 'd']

42. ['d', 'd', 'd', 'd', 'd', 'd', 's', 'd', 'd', 's']

43. ['d', 'd', 'd', 'd', 'd', 'd', 'd', 's', 's', 'd']

44. ['d', 'd', 'd', 'd', 'd', 'd', 'd', 's', 'd', 's']

45. ['d', 'd', 'd', 'd', 'd', 'd', 'd', 'd', 's', 's']

(Where 'd' = dumb and 's' = smart)

Phew. Now that that's done, and we've already proven the order doesn't effect the percentage change of the specific outcome, we can say the the chance that your 8 out of 10 users getting the wrong answer because they are dumb is: 100 * (0.84^2 * 0.16^8 ) * 45 or 0.00136373801582592%. Now, how low is this number? Well... it's about as likely than getting a straight flush in poker.

Answer 2

In one study I discovered "5 out of 5 users couldn't complete the task" in the past, and my stakeholders simply didn't believe the data. So this is a something to consider.

While Nielsen advocates 5 people will discover your more serious problems, he also goes on to say that you should address those findings, then retest with a different set of 5 users. He explains that if you keep doing this, the number of new problems will deminish very quickly. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford

He later followed up on that article with this one: https://www.nngroup.com/articles/how-many-test-users/

Testing with 5 people lets you find almost as many usability problems as you'd find using many more test participants

In gerenal you need to think through your testing (mix quantitative with qualitative) and you need to spend as much time on planning as analysing results - the testing is the smallest part in time.

Remember, you need to present your results to your stakeholders, and they need to see how rigourous you have been in discovering those insights (the learned insights are your output to your stakeholders). If your sample size is small, we generally do NOT convert numbers into percentages and don't forget you can use statistical techiques like calculating the confidence interval, e.g. so they can see "value x plus or minus y couldn't complete this task."

Answer 3

From a statistical perspective, the larger your test group, the more accurate your result. With an increasing number of participants, your result in the limit value always approaches the real value. However, this is practically impossible to make. In my data sciene courses I often heard that 10000 people are a reliable number, but that should also be difficult to implement.

I would spontaneously remember to base your tests on the construction of neural AI networks. A possible approach would be, for example, to select 10 ** random ** people and divide them into two random groups. With the first 5 people you normally do your tests. You then determine the result of what you get from this first group in a hypothesis and specifically test this hypothesis with the second group. If the second group confirms your results, you have a good chance that other groups would do the same. If you repeat this procedure several times, you should be able to achieve very precise values ​​with a relatively small group of participants.

You can make the result even more precise if you change the group sizes over time. I personally achieved good results with the following values: Group 1 (lets call it batch group) 70-80% of the people. Group 2 (test group) 30-20% of the people. 10 runs. Cheers!

Answer 4

Most usability questions can be answered with small batch, qualitative testing, as mentioned above.

Some questions require large samples and quantitative proof to answer. A/B testing uses large samples of live traffic to answer narrow design questions. Many of the client-side A/B test platforms include a stats engine to determine "statistical significance". Optimizely, a leading A/B test platform, includes this handy calculator to figure out how many people you need for proof.

Another way to figure out what constitutes proof, is to calculate the margin of error.

Answer 5

Change the way you communicate with stakeholders

If you need evidence to resolve a deadlock between stakeholders, the battle is already lost. Good design is not about quantitative science, it's about getting stakeholders to communicate effectively.

First off, user tests are an informal procedure. They can be very informative to a designer or a design team, but they are not a scientific procedure and the results are strongly subjective. If everybody on the team shares the same view about what the application is supposed to be, and has the experience to interpret a test, they are a great tool. They inform the design process. However, they are not evidence, and if you have stakeholders with opposing views and different agendas, user tests are not the way to deal with that.

There are tests that are rigorous enough to provide genuine evidence. A/B testing, for instance. Using these to make design decisions is also a terrible idea. The reason is that you end up optimizing for some arbitrary goal (like conversion) because that goal can easily be expressed as a number, while all your other goals, like customer satisfaction, cognitive load, rate of return can less easy to measure.

The article Should designers trust their instinct or the data on the Google Ventures blog discusses this in greater detail. It includes the following image:

The checkout button on the right is great for conversion, but that doesn't make it good design, because design is about balancing different goals, not about blindly optimizing the ones that can easily be measured.

The challenge of designing for different stakeholders is not solved by deciding based on data or user testing. It's solved by taking every stakeholder's concerns seriously, mapping them out, investigating them honestly, and finding creative solutions that make everybody happy. In your case, if viewing a user test doesn't convince stakeholder X that Y is a serious issue, find out what problems they have with fixing the issue (time, money, adding complexity, lowering security etc). Maybe you can come with a solution that solves the issue without causing the problems X is worried about. Ideally, you solve it in a way that makes everybody happy.

Sometimes viewing a user test brings people on the same page, but if it doesn't, don't point to the user test as evidence.

Answer 6

This seems to me to be more of a statistics question than a UX one. If you have a binary classification (issue did or did not arise), then you have a binomial distribution. If you want a lower bound for the "true" probability of the issue arising, you can enter that lower bound into a binomial distribution to get the probability of getting what you saw. So, for instance, if you saw the issue 8 out of 10 times, you can enter 10 as the number of trials, 8 as the number of successes, and 0.1 as p. That will get you the probability of seeing those results if the issue really were present only 10% of the time. You can then say "If the true rate is less than 10%, the probability of seeing the results we got is less than [give number you got]". You could also find the confidence interval. Google or any statistics package (including Excel) should get you what you need.

And as an aside, "this is just stupid people" isn't much of an excuse. If you're doing UX, your job is to make an interface that your actual users can use, not one designed around an ideal user who always acts perfectly.