Let's get critical: Aug 2014 Site Self-Evaluation

Question

We all love Cognitive Sciences Stack Exchange, but there is a whole world of people out there who need answers to their questions and don't even know that this site exists. When they arrive from Google, what will their first impression be? Let's try to look at this site through the eyes of someone who's never seen it before, and see how we stack up against the rest of the 'Net.

The Site Self-Evaluation review queue is open and populated with 10 questions that were asked and answered in the last quarter. Run a few Google searches to see how easy they are to find and compare the answers we have with the information available on other sites.

Rating the questions is only a part of the puzzle, though. Do you see a pattern of questions that should have been closed but are not? Questions or answers that could use an edit? Anything that's going really well? Post an answer below to share your thoughts and discuss these questions and the site's health with your fellow users!

Community · Accepted Answer · 2017-04-13 12:56:15Z

Final results overall

Central tendency of net scores: mean = 1.5, median = 3
- Distribution: SD = 4.7, skewness = -.4, kurtosis = 2.1, Shapiro-Wilk W = .93, p = .43
- Frequencies: excellent = 35, satisfactory = 32, needs improvement = 20
Most consensus:
Are there any proven ways to increase concentration and focus? = needs improvement×6 + 1E
Least consensus:
Is learning and memory formation mostly comprised of new synapses forming, or already-existent synapses strengthening? = 2×E, 3×S, 4×NI

Comparison to five previous site self-evaluations:

Net score data

             _______________Month_/_Year________________
Question_#__|__8/14_____2/14_____11/13_____8/13_____5/13_____5/12
         1  |   7        5         3        7        6        7
         2  |   7        3         3        6        4        7
         3  |   4        1         2        6        4        7
         4  |   4        1         2        6        3        7
         5  |   4        0         0        6        3        6
         6  |   2        0         0        5        3        5
         7  |   0        0        -1        3        2        4
         8  |  -2       -4        -1        2        2       -2
         9  |  -4       -4        -1        1        1       -3
        10  |  -7       -5        -4       -2        1       -3

Descriptive statistics

Month/Year:   8/14   2/14   11/13   8/13   5/13   5/12
   Medians:    3      0       0      5.5    3      5.5
     Means:    1.5    -.3      .3    4      2.9    3.5
        SD:    4.7    3.2     2.2    2.9    1.5    4.4

Inferential statistics

Kruskal–Wallis χ² = 13.6, df = 5, p = .02; self-evaluations' net scores are distributed differently. However, no post-hoc comparisons (Dwass–Steel–Chritchlow–Fligner test – Steel.Dwass() in RcmdrPlugin.EZR package; see also this reference) achieve p < .05; only the difference between 2/14 and 8/13 came close (t = 2.74, p = .067). These net scores still correlate negatively with order of site evaluation (Kendall's τ = -.25, which converts to r = -.39 using this method; z_τ = 2.6, p = .01), indicating a downward trend. The linear model plotted above using the Theil–Sen single median estimator (in the mblm package) is score_i = .75×time_i - .5 + ε_i; its standardized β_time = .36, not .39.

This month's set has the greatest variance so far, which suggests ratings may be heteroskedastic across time. However, a Brown–Forsythe test (levene.test() in the lawstat package) can't reject the null of homogeneous absolute deviations from the median (statistic = 1.7, p = .15).

Discussion

Despite the overall trend since the first evaluation, scores are up since our last evaluation. All pairwise differences are insignificant when comparing ratings aggregated to the question level, but the omnibus test is still picking up overall differences in ranks of scores across evaluations.

I'm still ignoring dependencies across time (e.g., I participated in the last evaluation too) and the hierarchical structure of individual ratings nested within questions, which I'm comparing across evaluation periods. Again, the May '12 scores may not be comparable due to method variance.