Well, here it is, three years later... Three years of complaints, bug reports, stalled discussions and abortive attempts to make that number more accurate. And I have to say...
...this is done!
Which is to say, we lost the review count. It is no more. No matter how we approached it, calculating an accurate count of reviews - or even just an accurate timestamp for the newest task - for the current user on every page load was simply not something that could happen without a complete redesign of review.
Upon that sad realization, we revisited the original rationale for having a counter for review:
I was wondering if there wasn't something that could be done to help direct everyone possible to review (or at least be aware of it - there was a comment at one point of a 10k user who wasn't aware of anything in the /review link after clicking it as a new user and not seeing anything in it that could be done by him)
Note that the specific suggestion there was never implemented (too expensive) - we punted and threw the total number of pending reviews up for users with 10K+ rep, thus fulfilling the prophecy contained in that same request:
This should be something that you can get to 0 every day. Making it something that is an unmanageable number makes it something people will ignore, and that's not what this is suggesting.
...as well as generating endless confusion as to what the number even meant.
But there was a small side benefit to it, annoying though it was: if you were the only active reviewer on a site, you couldn't ignore the fact that your peers weren't pulling their weight - if no one else was reviewing, the number would just keep going up.
These were the base requirements then:
- Notify people that their assistance was needed in review
- Don't display a number that doesn't correspond to anything the reviewer can actually do
- Indicate when the site is unhealthy due to a lack of active reviewers
Now... There are some previous use-cases that aren't in those requirements. For example: if you happen to be a moderator and the only reviewer on a small site, the number worked perfectly for you and its replacement is not going to work as well. This was never really a goal for review though.
So with those requirements in mind, we replaced the number with a simple indicator, triggered based on one or more queues exceeding a per-queue (and per-site) threshold. When chosen correctly, those thresholds should represent a level at which there are an insufficient number of active reviewers - thus fulfilling requirements #1 and #3. The thresholds should also be sufficiently high that discrepancies between the cached task counts and what the current reviewer can actually review don't matter - thus fulfilling requirement #2.
Needless to say, this isn't the conceptually-elegant, trivially-scalable system we wanted: it's likely going to take a lot of tweaking of those thresholds to make it work effectively. But it has a distinct advantage over all those other ideas we've been batting around for the past 3 years: it's actually been implemented and hasn't brought any site to a grinding halt.
