Cronbach alpha - descriptive statistics

Question

I researched thinking styles based on Sternberg's Investment Theory. I used a TSI questionnaire that measures preferences in thinking styles. The problem is that I got low scores of Cronbach's alpha for some types of thinking styles. Examples are shown below:

• Local type of thinking styles, which contains six items, with a reliability of $a = 0.545$
• Monarchic style ($7$ items), $a = 0.638$
• Oligarchic ($6$ items), $a = 0.636$
• Anarchic ($7$ items), $a = 0.647$
• Global ($6$ items), $a = 0.648$.

There are $13$ scales, and $6$ of them have a low Cronbach alpha coefficient. What should I do? How do I interpret this? Does this mean that my research results are not valid?

I used Pearson's $r$ to calculate the correlation between scales, and it shows a good correlation between scales. How can I interpret those results as well? I hope I was clear enough because my English is not that good. :)

Answer 1

Even though low alpha values like the ones that you describe could mean that the scale scores contain a lot of measurement error, an alternative (or additional) explanation could be that the items within a given scale are heterogeneous/multidimensional (i.e., that they measure specific factors and only to a certain extent a common factor). As Jeremy Miles has indicated, running a confirmatory factor analysis to evaluate the factor structure could shed more light on this issue of uni- vs. multidimensionality of the items within each scale.

Also, for alpha to be properly interpreted as "composite reliability", the items of a given scale not only have to follow a single-factor (unidimensional) model, but they also have to have equal loadings and uncorrelated errors (tau-equivalence assumption). Otherwise, alpha could potentially be misleading as a reliability index.

One thing that you could do if you were worried about lack of reliability is use an SEM with multiple indicators in your research. That way, you could correct for error when examining structural relationships (correlations or regressions between latent variables), and low observed variable reliabilities need not be a problem.

Answer 2

First, reliability estimates in the 0.6 - 0.7 range are not great, but they're not completely awful.

There are a lot of things that you could do. It's not as easy to say what you should do.

You got low reliability. That's not great for your research. Your may not be great measures. But they still might be. One thing you can do in your write up is say "Reliability was low, so perhaps results are less trustworthy". But there are many possible reasons for this. You could investigate these reasons.

Alpha is a function of variance - if you have low variance, you will have low reliability - that is, everyone giving very similar answers. Say you have a 5 point scale about how much you like cats. And you give it out at a cat fancier's meeting, everyone will answer with 4 and 5. Your reliability will be low. Combine that with data you collect at a dog fancier's meeting, the reliability will increase, because the variance will increase.

Perhaps it's particular items. You could examine individual items, using things like the item-total correlation. Maybe the scales improve if you remove an item that didn't work well. (Example: To an American, the word 'touchy' means sensitive, easily offended. To a British person, it might mean 'tactile' - likes touching people. If you keep the item 'touch' in a measure of sensitivity, the reliability might be low in a British sample.

You could do factor analysis to investigate the underlying structure of the scales.

Alpha is an estimate of reliability, but to interpret it as reliability makes the assumption that there is an underlying latent variable which is the cause of the responses to the items. You think there is a variable called happiness, and this causes many behaviors - happy people smile more, sleep more, laugh more. Or is happiness a function of things that happen to me: I get enough sleep, I'm paid enough, I have friends, I have a supportive family - perhaps these things cause happiness. If the latter is the case, alpha is not a good estimate of reliability. What is the relationship in your data?

How big was the sample? You could investigate the confidence intervals of alpha.

You could investigate whether there are subgroups for whom the TSI has higher alpha. Perhaps it's not relevant to some people, so they're not able to answer. If I ask people to complete my "CrossValidated Style Inventory" most people don't have a CrossValidated style, so they can't give sensible answers that make sense. But heavy users (and answerers) of CrossValidated could be measured on many questions (answer length, use of equations, answer frequency, critical style, etc).

Answer 3

Jeremy already has a good answer. I just wanted to highlight the last part of the question since I didn't see that specifically addressed. If you mean that the scales as a whole correlate well with each other, then this just means they have good convergent validity, or that they are closely related (either by how it measures a construct or other factors). This is somewhat independent of the reliability of a test. The classic image used for this distinction is shown below (source here):

Your composites are probably something closer to the target on the top right (potentially). They seem to be measuring the same thing, but there appears to be some noise in the estimation, which can be any number of factors (noted in Jeremy's answer).