Best practices for sample size in disaggregation of data on educational outcomes

Question

The California community college where I teach has a prescribed system for collecting and analyzing data on students' success and then disaggregating the data by race. As a department, we are required to use these numbers to calculate an "achievement gap," defined as the highest success rate minus the lowest success rate. This tends to lead to statistically silly results, since many of the racial categories are very small. For example, we may have one student in our department in a given year who clicked on "other non-white" to describe their ethnicity, so then that racial category is assigned a success rate of either 0% or 100%, depending on whether that particular student passed the class they took. I have raised this issue with the relevant faculty committee but was rebuffed, apparently because any proposal to change these statistical practices was perceived by some members of the committee as bad for promoting racially equitable outcomes. Disaggregation in general is required by our accrediting body, but there are no specific requirements, e.g., we could disaggregate according to categories that have nothing to do with race, or we could use race but not define an achievement gap in this way. Their requirement is:

The institution disaggregates and analyzes learning outcomes and achievement for subpopulations of students. When the institution identifies performance gaps, it implements strategies, which may include allocation or reallocation of human, fiscal and other resources, to mitigate those gaps and evaluates the efficacy of those strategies. (2014 ACCJC Standards, Standard I.B.6)

Googling shows that the national K-12 establishment seems to have considered these issues very carefully, including consideration of privacy issues that arise when a racial group is very small, so that individuals can be identified. The 2015 Every Student Succeeds Act (ESSA) makes a specific requirements that:

the statistical rigor that informed the selection of the minimum n-size should be documented and how this minimum number is statistically sound should be described.

However, I'm not turning up much information relating to higher education. I came across some state legislation in my state that in its original version would have required statistical significance in both health and educational statistics, but the educational part was stripped out during debate by the Senate before the bill was passed into law.

Are there laws or recognized best practices that apply at the college level? The whole thing seems to me like a pretty trivial exercise in freshman statistics, but it seems that mathematical facts and competence, in and of themselves, do not count for much unless stated by some authoritative body. I would think that academic publications on this topic would also command at least some respect.

[EDIT] After some discussions in comments with Buffy, I think the following clarification may be helpful. I do not advocate the use of these statistics for these purposes, so I'm not asking for a magic prescription as to how to use them so that they will be valid indicators of something. We are currently being forced to use them for a particular purpose, as part of a required program review process that occurs at intervals of a few years. These statistics are used by administrators for making decisions that affect us. For example, if we want money to be spent on lab apparatus or a new tenure-track position, we are expected to explain why that spending will improve our students' success, according to various measures, including the "achievement gap" described above. What I'm hoping for is basically suggestions for harm reduction, such as publications that I might be able to point to in order to show that the practices being imposed on us are not in line with best practices. It may be that the entire exercise is so totally flawed as to be unusable for any serious purpose. If so, then it would be great to have an answer making the case that that is so, but in order to be helpful, such an answer should point to something like a peer-reviewed paper, procedures at a specific institution that the answerer considers to be a model, or guidelines by a national organization. Our accrediting body requires us to do something with data like this, so I don't think it's going to fly if I tell my colleagues that we need to lobby for change at our accrediting body because anonymous-internet-person makes a good argument.

Answer 1

Basing a proportion on one observation is silly. If your funding body needs a reference for that, then they are a lost cause. It is unlikely any argument based on that will reach them, so I would (initially) forget about that angle.

One possible alternative would be a requirement that the results are anonymous, i.e. you cannot recover the results from individual students from the numbers you report. Statistical agencies often use that requirement not to report results from small groups. This style of argument may fit the bureaucratic mindset of your funding agency better.

Answer 2

I think you are missing the purpose of gathering such data. While statistics has quite valid procedures and rules for measuring the probability that a sample accurately reflects characteristics of a population, that isn't the purpose here. In particular, your samples aren't random, most likely, so even the interpretation with larger samples becomes problematic.

Even more problematic is the question of determination of the population itself. If it is just the local students, then you aren't in the world of statistics at all, as you are querying the entire population.

However, the purpose of gathering such numbers is to provide warnings when things seem out of balance leading to possible policy steps to correct them. All such things are fraught, of course, but they are really just triggers that might generate action. On the flip side, they can be "soothing tones" that suggest all is well when perhaps it isn't.

In the best case, "bad" numbers lead to discussions and further "research" which may lead to corrective action if it is deemed necessary. But scientific "accuracy" isn't necessary unless it is being built in to automatic actions applied without thought.

However, with such small numbers, it is unlikely that one should try to publish "scientific-ish" papers that suggest something has been "proven" when it really hasn't.