How valid is Lumosity's Brain Performance Index and what normative information is available?

Question

Background

There is a test called Lumosity's Brain Performance Index. A sample profile of scores might be:

Speed           680
Memory          707
Attention       277
Flexibility     304
Problem Solving 900
Overall BPI     573

I'm interested in how to interpret a profile of scores.

Questions

• What validity information is available for the BPI?
• What normative information is available for BPI? In other words, how could the sample profile of scores be interpreted both absolutely and relative to other groups of people?

Answer 1

General thoughts on brain training: Lumosity is a commercial tool that aims to improve brain functioning. In general, I am sceptical of the potential for "brain training programs" to improve cognitive functioning in a generalised way (e.g., see this Nature discussion). Practice is powerful, but tends to be domain specific. So if you want to become skilled in brain training tasks, then do brain training exercises, but if you want to become skilled in a particular domain of life, then study and get experience in that domain. While I can see the value in researching the topic, I can also see the potential for sellers of "brain training" programs to exploit people's concerns about their mental shortcomings or about fears of mental deterioration with ageing.

Normative data: From what I can gather from this review, Lumosity has lots of normative data based on existing users which you can use to compare your performance. However, I imagine this is proprietary information that they would be reluctant to publish in a useful form.

There was an unpublished report I found by Cruz et al based on a small sample of young adults. The following summarises Table 2:

Number of days        4.75 ± 3.18
Number of sessions    10.79 ± 9.31
Median (IQ R )        9.00 (2.25‐17.75)
Minimum‐Maximum       0 ‐ 34
Overall BPI           676.93 ± 323.94
Lumosity points     173.43 ± 158.89

Validity of the BPI: The website writes:

Your Overall BPI is your average BPI across each of the four cognitive areas: attention, memory, processing speed and cognitive control.

It also states

"We then evaluate your game scores and use a proprietary algorithm to derive your BPI"

Performance on almost all cognitive tasks are intercorrelated to some extent, and if you take the average, or preferably the first principal component, of a battery of cognitive tasks, it will tend to have a general factor, which will have some loading on g. Of course, if the tests are of a particular type, this might lead to different outcomes.

I could not find any validity data on the website whereby BPI scores are correlated with other measures. Thus, it is unclear what relationship BPI has with other ability measures, or IQ for example.

In general, practice effects on cognitive tests are seen as a source of error variance. What a test measures after people have taken the test many times may be very different. In particular, the degree to which the test correlates with a domain general quality of interest may be reduced. Instead, the test may start to reflect a domain specific adaptation. Also, if people differ in the amount of practice, this would compound the measure of a domain general ability.

Overall thoughts: In general, the target market for Lumosity's product seems to be consumers. In contrast, tests like the WAIS or the CANTAB have a target market which is researching and applied psychologists. In the consumer market, it is typical for companies to use "proprietary algorithms" and for there not to be a test manual with extensive validity data. While this might make commercial sense, it limits the scientific value of such instruments.

References

• Cruz, A., Canelas, A., Machado, F., Pedreira, J., Barroso, J., Beir\~ao, L., Freitas, L., Belo, L., Seabra, M., Portero, M. & others (). Brain Fitness--Clinical Trial to Measure the Effect of Cognitive Training in Brain’s Capabilities. PDF

Answer 2

Because it has been a few years since Jeromy's original answer, and because I just read a very apt article, I will venture an update on the state of the field with respect to the BPI's validity. Overall, despite more research into brain training and Lumosity, there is little to no peer-reviewed evidence supporting the Lumosity BPI's validity, nor evidence that an average consumer should expect practical cognitive improvements from Lumosity games.

Testing the BPI

A very recent test of the BPI's validity was published by Shute, Ventura and Fe (2015), who compared participants practicing Portal 2 with participants practicing Lumosity games; the Lumosity players served as active controls. Portal 2 measures included (1) total number of levels completed, (2) average number of portals shot, and (3) average time to complete levels; Lumosity was measured by BPI. They reported no evidence of transfer effects for measures of either problem solving or spatial abilities in the Lumosity condition, and partial support for transfer in the Portal 2 condition only on spatial abilities.

Hypotheses 2 and 3 test pretest to posttest gains on specific problem solving and spatial test scores within each condition. We computed paired t-tests for each measure in each condition. For hypothesis 2 (problem solving gains), we found no significant gains for the Portal 2 condition, from pretest to posttest, for any specific problem solving measure. We also did not find any significant pretest-to-posttest gains for the Lumosity condition across any of the problem solving measures. Results provide partial support for hypothesis 3 (spatial gains). Participants in the Portal 2 condition showed pretest to posttest improvement on the MRT, t(1, 39) = −1.80, p < .05; Cohen's d = .30 under a one-tailed test. Additionally, Portal 2 players' VSNA scores significantly differed from pretest to posttest, t(1, 40) = 2.42, p < .05; Cohen's d = .44. There was no significant improvement for Portal 2 players on the SOT test. For participants in the Lumosity condition, there were no significant pretest-to-posttest improvements on any of the three spatial tests.

Brain training

A rigorous study by Redick et al. (2013) suggested that the initial research on brain training generally, and thus the basis for Lumosity (mentioned by name in the paper), may have been compromised by design limitations and lack of theoretical grounding, and criticizes those studies for struggling to adequately explain often mixed results. They reported three relevant findings from their own validation study of dual n-back working memory training:

Our study yielded three main findings. First, subjects improved with practice on both the dual n-back and visual search tasks. Second, training group subjects showed no transfer to any of the ability measures, in keeping with the prediction outlined in Figure 4D. Third, dual n-back trained subjects reported subjective improvements in various aspects of cognition in the absence of any objective evidence for change.

Concluding remarks

The jury is still out on brain training more generally and on when, where and whether it may be useful (e.g., Deveau et al., 2014; Mayas, Parmentier, Andrés and Ballesteros, 2014). However, the Lumosity BPI appears to have little to no scientific validity as a cognitive training method for average consumers.

References

• Deveau, J., Jaeggi, S. M., Zordan, V., Phung, C., & Seitz, A. R. (2014). How to build better memory training games. Frontiers in Systems Neuroscience, 8.
• Mayas, J., Parmentier, F. B., Andrés, P., & Ballesteros, S. (2014). Plasticity of attentional functions in older adults after non-action video game training: a randomized controlled trial. PloS one, 9(3), e92269.
• Redick, T. S., Shipstead, Z., Harrison, T. L., Hicks, K. L., Fried, D. E., Hambrick, D. Z., ... & Engle, R. W. (2013). No evidence of intelligence improvement after working memory training: a randomized, placebo-controlled study. Journal of Experimental Psychology: General, 142(2), 359.
• Shute, V. J., Ventura, M., & Ke, F. (2015). The power of play: The effects of Portal 2 and Lumosity on cognitive and noncognitive skills. Computers & Education, 80, 58-67.