Queue growth
Review queues lose effectiveness if posts aren't regularly (and
accurately) cleared. A queue such as close votes on Stack
Overflow
that never seems to get to 0 fails to provide time-sensitive feedback
that aids learning. So if we raise the bar for late answers which
increases the number of tasks going into the queue, we risk making the
queue less useful for people answering questions late.
As I write, the Late Answer queue on Stack Overflow is empty. In the
last 24 hours the number (non-audit) review tasks of each type added
to the queues:
Review Tasks Type
------------ -----------------
2467 Triage
2428 First Post
2333 Close Votes
1846 Suggested Edit
883 Low Quality Posts
619 Late Answer
594 Helper
141 Reopen Vote These two factors suggest that there's room to add more tasks to the Late Answer queue. Assuming we
raised the reputation limit to 100, we'd add about 300 tasks:
Rep Bin Tasks
------- -----
10 417
20 160
30 85
40 76
50 44
60 41
70 37
80 21
90 20
100 20
You might expect the number of tasks associated with the 1 to 10
reputation bin would equal the number of late answer tasks in the last
day. The discrepancy stems from reputation earned since answers were
added to the queue. But the takeaway is that raising the reputation
limit to 100 would add half again as much work to the queue. That
doesn't seem particularly onerous.
Not surprisingly, Stack Overflow would see the most added Late Answer
reviews by far. Other large sites, such as Ubuntu, Server Fault, Super
User and Mathematics, would get 10 or so extra reviews a day.
Justifying 100 rather than 50 reputation
Discovering answers as comments certainly would suggest 50 as a good
limit for this queue. But this queue is as much about granting
exposure to potentially upvote-worthy answers as it is about finding
problems. Especially on large sites, it's pretty easy for a good
answer to a very old question to be lost in the noise. As a strategy
for new users answering
old questions that already have good, but not excellent, answers seems
productive if not that few people will notice those answers. At 10
reputation, a knowledgeable new user has one shot at earning upvotes
from excellent answers to aged questions.
A potential complication
The one concern I have is that answers are dequeued once the author
has reached 100 reputation. If we raise the maximum reputation for
going into the queue, we should also raise the minimum for coming out
of it. Unfortunately, I don't have an easy way to gather numbers to
set an appropriate level. One of the reasons I'm only looking back a
day in the numbers I showed above is that it minimizes the effect of
gained reputation. It's possible, but messy to estimate reputation at
the time a user answered a question. So while my instinct would be to
raise the deque minimum to 200, but I can't really predict if that's
appropriate or not.
Proposal
Raising the bar to 100 would require changing two variables at once.
Since that would make it harder to evaluate each individual change,
I'd like to try raising the reputation for getting enqueued to the
Late Answer reviews to 50 and reconsider upping it to 100 at some
point in the future.