51

With the rollout of the new top bar, you’ve probably noticed the new review indicator:

Reviews needed indicator or My sock's view of the reviews needed indicator, which has less available queues

It’s possible there will be a different review icon by the time you read this. But this post is just about the red circle on the icon. You’ll see the indicator is turned on:

  1. if any review queue has a red dot (explained below) and
  2. if it’s been at least an hour since you clicked on the review icon.

So the idea is that when you see the indicator light up, it’s a reminder that there are reviews to be done if you haven’t looked recently. The goal is to draw reviewers’ attention to queues that can use some help.

Each queue may be in one of three states:

  1. No dot (empty queue)
  2. Grey dot (at least one but fewer than N review tasks)
  3. Red dot (N or more review tasks)

Each review queue has its own threshold (N). The default values are:

Queue Default
Close 20
Reopen 5
Low Quality 4
Suggested Edits 3
First Posts 10
Late Answers 6
Help (Stack Overflow) 150
Triage (Stack Overflow) 100

We call more than N tasks the “danger zone”, but the thresholds aren’t tied to any particular system limitation. Instead they are based on the 90th percentile of hourly samples of pending tasks. To put it another way, we should expect each queue to be in the “danger zone” roughly 10 percent of the time. Obviously, each site has its own level of activity so we can (and do) adjust the thresholds from time to time. If we do that job right, it shouldn't matter what the exact numbers are.

How we adjust the thresholds (for now)

First, we group the queues based on privilege level:

For each group, we calculate the median and 90th percentile active reviewers per hour for the last two weeks. Then we calculate the median and 90th percentile pending reviews for each queue. The idea is to estimate how many people can work on each queue and how much work is available for them to do.

Then we run some checks of potential problems:

  1. over-warning

    If the threshold is less than the 90th percentile of pending reviews tasks, that queue will likely trip the indicator more often than we'd like. In this case, raise the threshold. (We have a little fudge factor here to avoid making small changes each time we run the check. We've noticed that review queues tend toward the threshold we set. There's a ratchet effect in play here.)

  2. never-warning

    If the threshold is more than twice the 90th percentile of pending reviews tasks, the queue may never trip the indicator. That doesn't mean these queues won't be cleared since people might do these review tasks without prompting from the top bar indicator or while they are doing other reviews. But we'd like all of the review queues to get attention from time to time.

  3. reviewers warned of same tasks twice

    Especially on sites with a small number of active reviewers, there's a chance tasks you'd acted on will still be in the queue after an hour when the indicator can turn on again. Ideally, we'd keep track of how many reviews each user has outstanding and only show the indicator if they have something to do. But in that case we might as well show everyone the number of review tasks they might be able to do, which brings us back to the performance/accuracy problem that got us here. So we fudge it by checking if the median number of reviewers is less than the median number of tasks needed to complete the reviews. In that case, we raise the threshold.

  4. colliding reviewers

    The opposite problem is when there are many potential reviewers getting notified about too few reviews. If the threshold is too low (we check if it's less than the 90th percentile of reviewers in an hour), there's a chance reviewers will visit the queue only to be locked out of the tasks. In this case, we raise the threshold.

Over the next few weeks, Shog9 and I will be looking a the queries he wrote to see if any queues across the network need their thresholds adjusted. Assuming we've hit upon appropriate criteria, we'll look into automating these adjustments.

Caveats

As always, this is complicated by caching. You might see the site review indicator light up and then not see any of the individual queues in the red. In that case, it’s quite likely one or more of the queues just dipped under the threshold (and might pass it again soon).

Finally, all active tasks are counted for each queue when determining if a queue is in “danger”. Thus, there may be times when the indicator will light up but you’ll be unable to do anything about it—for example, if you’ve already done all the reviews you’re allowed to do for the current day in that queue, or if all the items to be reviewed are your own work.

Improve this question
18
  • 1
    I think you're wrong with this statement if it’s been at least an hour (30 minutes on Stack Overflow) since you clicked on the review icon. tested and verified
    – Optimus Prime
    Commented Oct 18, 2017 at 4:35
  • 2
    @OptimusPrime: There may very well be a bug. Can you describe your test in more detail? Perhaps in a separate meta post?
    – Jon Ericson
    Commented Oct 18, 2017 at 4:37
  • 2
    @OptimusPrime: I just tested the indicator using a timer. It did indeed remain off for the entire half hour and turned back on when I reloaded the page after the timer went off.
    – Jon Ericson
    Commented Oct 18, 2017 at 5:46
  • 5
    I think it's better to split this into question and answer, as it's not really an announcement, but rather a faq-like post. (Also, it doesn't look like you seek feedback, right?)
    – Shadow Wizard
    Commented Oct 18, 2017 at 6:30
  • 3
    Flagged for including the answer in the question :-P
    – Rand al'Thor
    Commented Oct 18, 2017 at 8:43
  • "all active tasks are counted for each queue when determining if a queue is in “danger”." - So are all tasks only taken into account for the red dot or for grey dots too?
    – Christian Rau
    Commented Oct 18, 2017 at 11:07
  • Is it technically feasible just to calculate the 90th percentile for every site and use that, rather than grouping them? The default numbers are unlikely to be reached at all for small beta sites like the site I moderate, where the 90th percentile of close votes is... 3. Is there a reason (simplicity?) that groups were chosen instead of per-site values based on the 90th percentile, or would this be a viable option to fix the issue?
    – Aurora0001
    Commented Oct 18, 2017 at 14:08
  • 1
    The 90th percentile thing was just a tool for picking ballpark numbers, @Aurora0001 - it makes for a terrible threshold if you keep updating it. Obvious example is Stack Overflow: when we first rolled this out, all the queues were at record-high numbers of tasks with the exception of First Posts / Late Answers - so the initial thresholds were unworkably low for those queues (tons of reviewer collisions) and nearly useless for the rest. I went back and adjusted them based on a combination of available reviewers and practical considerations (suggested edits take priority over new questions).
    – Shog9
    Commented Oct 18, 2017 at 15:43
  • That's a good point, @Shog9. Just wondering whether there's a more automated way of determining sane thresholds that are actually useful, as even Catija's proposal for a "small" category would still not really fit sites like IoT (I don't remember ever seeing more than 5 review items). With over 170 sites, I don't think it's possible for a human to individually decide on reasonable values, and the current ones mean that the feature just won't be seen at all on some sites, and too much on others, hence my suggestion that the thresholds be generated algorithmically (easier said than done, though)
    – Aurora0001
    Commented Oct 18, 2017 at 16:03
  • 3
    @Aurora0001 which is why I added the timer function... a single review item wouldn't immediately be brought to the site's attention, so if it were handled by users (or mods) who were looking anyway, then it would be... but if it wasn't cleared in a day or so, it would start lighting up the alert even if it was alone in the queue.
    – Catija
    Commented Oct 18, 2017 at 16:09
  • 1
    My not-very-clever solution is to simply wait for this to operate for a week or two and then revise the thresholds based on how folks actually act on each site, @Aurora0001. 'course, we really need that bug Jon mentioned above to get fixed, since right now this is essentially non-existent for most reviewers.
    – Shog9
    Commented Oct 18, 2017 at 17:07
  • I find the new red dot on the top bar annoying when I'm using the site for its main purpose. When asking or answering a programming question, or reading through old answers, I want to focus on that particular programming-related task. Red alert dots are an unwelcome distraction. I do look at the review queues pretty often, but in between other tasks. A red dot suggests something urgent. For routine reviewing tasks, if we need an indicator on the top bar at all, perhaps a grey dot would be more appropriate (saving red dots for absolute emergencies, that really do need urgent attention).
    – Rapunzel Van Winkle
    Commented Oct 19, 2017 at 0:06
  • 1
    @JonEricson here's what I told yesterday. i.sstatic.net/ZbAOc.gif
    – Optimus Prime
    Commented Oct 19, 2017 at 9:43
  • 1
    @OptimusPrime: That's not the intended behavior. It happens because we calculate the indicator based on all queues and show it to everyone who has access to any of them. It's possible to fix, but we need to make sure we don't slow down the page load to do this calculation. We're investigating.
    – Jon Ericson
    Commented Oct 19, 2017 at 16:44
  • 1
    @Shog9 despite the buggy behavior, this is very bad UX. And the bug can be seen every single day on UX.SE (I assume other sites as well), so it's far from being non-existent, would be great to have it fixed for once, or not have it at all
    – Devin
    Commented Jan 12, 2019 at 19:56

4 Answers 4

Reset to default
21

Don't show the indicator when that user can't do anything about it.

The indicator is intended to draw attention to the review icon, but showing it when a user can't do anything trains the user to ignore the indicator and the review icon. That's the exact opposite of what's desired from showing such an indicator. The indicator should never be shown when it's not possible for that user to actually review posts in the queue(s) which are "in danger".

While doing this might be more expensive with respect to database access, it's important if we are not going to train users to ignore this notification.

Improve this answer
6
  • 3
    I agree. We're working to fix the most egregious problem, which is that the icon lights up for everyone including people with too little reputation to handle all queues. There are a couple more ideas that we'd like to test for individuals, but this is a very difficult problem to solve without killing page loading times. (That said, I think there's some value in getting the red signal, seeing there are no queues in the red and going on to one of the grey icon'ed queues instead. But finding none of the queues you can handle with tasks after getting the signal is pretty bad.)
    – Jon Ericson
    Commented Oct 27, 2017 at 23:06
  • 1
    @JonEricson Seeing that this is now 1,5 years old, is there any progress on improving the behaviour of the red dot. Not showing it when someone is review banned, not showing it when the queue in danger is already maxed out by the reviewer in question, etc. When I see the red dot, and there is no queue with a red dot I simply get frustrated and click away.
    – Luuklag
    Commented Apr 3, 2019 at 8:35
  • @Luuklag: We are reviewing the settings. (I have a meeting that will touch on it today, actually.) It would help to know what site(s) you see this on. Unfortunately, this will never be perfect. Because of the complexity of the review system, there will always be situations where the indicator is misleading. The goal now is to reduce the odds this will happen with statistics.
    – Jon Ericson
    Commented Apr 3, 2019 at 15:43
  • 1
    @JonEricson We were seeing an issue on Stack Overflow earlier today. Personally, I saw the red dot for a couple of different queues which were both at 0 reviews available at the time the red dot was shown. Actually, at the time, all queues other than the CV and reopen queue were at zero reviews available at that time. We discussed it a bit in chat.
    – Makyen
    Commented Apr 3, 2019 at 16:38
  • @Makyen: Stack Overflow is an entirely different animal because of volume. We can make more confident predictions than on other sites, so I'd expect fewer problems on the whole. It sounds like there was a bug earlier today. If you still see a problem later in the week, we might need to run the numbers to see what we can adjust.
    – Jon Ericson
    Commented Apr 3, 2019 at 16:43
  • @JonEricson for example have a look at this MSO Q of mine, and the linked Qs: meta.stackoverflow.com/questions/373788/…
    – Luuklag
    Commented Apr 3, 2019 at 20:30
17

Thanks so much for sharing this information! I'm glad to have it so that we know what to expect. My understanding is that these are (for now) pretty rough numbers so my concern (particularly with the smallest number for the close vote queue being 50) is that many of these queues will never light up red.

I use (nearly exclusively) what I think are generally considered "small" sites... they're the sites that fall into the "default" range on this list but none of them ever have had 50 questions in the close queue that I've seen.

Using your query on the sites I do use, I got the following numbers for the close queue:

  • Cooking: 4
  • Movies & TV: 15
  • Interpersonal Skills: 3
  • English Language Learners: 10
  • Arts & Crafts: 0

That's a lot lower than 50. And, I'll be honest... I consider M&TV to have pretty high volumes of questions in their close queue and I still wasn't expecting it to be set quite so high.

So, the question is - is it better to start customizing these on a per-site basis to reflect their actual usage or is it better to add additional levels so that all sites are on some sort of level with other sites but there are more than four of them or do we add something additional, particularly for really tiny sites, that will help them find their way to the queues?

My concern with per-site customization is that it's difficult to document - your chart in the question is clear and easy for everyone to understand. If each site gets to set their own limits, this becomes tedious to document. At some point we may decide that it's necessary but, until then, I think we should try to make it simple.

My proposal:

Add a fifth group... call it "Small", maybe? This is set much lower than the others - 10 for close votes and 3 for everything else. This means that ELL and M&TV will turn on occasionally but the others won't... which is OK, because there's another thing you add that will only affect these little sites even though you can implement it across the network.

Set a timer.

If something has sat in the review queue for more than X days... say one or two... that red dot lights up even if there's only one thing in the queue because clearly, no one is looking for it and we don't want to let these age away on our tiny sites just because they're rare and users don't notice the queues.

There are certainly other options I've been considering but I think that they'd likely be more complex to implement. This proposal only requires a fifth group and a timer.

My favorite/preferred enhancement to the idea above would be:

Light the indicator when the total number of reviews meets N1, in addition to when a specific queue does. This would particularly help small sites where they may have one or two items in each queue but never get to that magic number for all of them. So, say N1 is 12 for small sites, if a site gets:

Movies & TV Review queue satus
(That's - 7 close votes, 2 first posts, 2 low quality posts, 1 reopen vote)

The red dot lights up in the top bar (though the queues stay grey) despite none of them meeting the red dot limits. I recommend this combined N1 should be something just slightly higher than the close vote queue, scaled e.g.: 10250, 230, 115, 55, 12.

This would help sites that are really close to the upper end of their thresholds but don't quite get there regularly. Coupled with the timer function mentioned earlier, this should help bring more visibility to the queues without (hopefully) sending out the bat signal constantly and annoying people.

Improve this answer
19
  • Take that query with a big grain of salt. I threw it together as a rough gauge of review activity for use on internal data; public data loses timestamps, so the numbers are closer to pending tasks per day - on any site and queue that doesn't normally carry a large backlog of tasks the numbers are useless (imagine a site that always handles suggested edits within 2 hours of their creation; you're essentially just counting edits that come in near UTC midnight).
    – Shog9
    Commented Oct 18, 2017 at 16:01
  • That seems even worse, though... 15 close items per day on M&TV and you have to get up to 50 just to light up the alert - it'd take ~3 days to ever be lit up! Am I misunderstanding something? Users don't even get 50 review tasks per day...
    – Catija
    Commented Oct 18, 2017 at 16:08
  • 15 is the carry-over - 90% of days had <= 15 tasks remaining. Based on that, it's unlikely that the button would ever light up for the close vote queue unless/until the volume of questions to be closed spiked or the volume of reviews dropped. Which is ok; the close vote queue shouldn't normally be in "danger". Contrast with, say, suggested edits, which hold up progress on the site until reviewed - the "danger" threshold is much lower (and probably could be 1 on most sites - these initial thresholds are very conservative).
    – Shog9
    Commented Oct 18, 2017 at 16:15
  • 2
    I don't understand why the red light is only for "danger"? Why are we trying to de-emphasize review? Leaving bad questions open is problematic, particularly for little sites, so not alerting users to their presence until there are 50 of them... that seems... excessive. Even the 10 I have proposed here seems to be a bit much on pretty much every site I use.
    – Catija
    Commented Oct 18, 2017 at 16:23
  • 50 may be the wrong number, @Catija. That's about a week of insufficient reviewing on M&TV at current volume, and in practice some number of votes would've aged away in that time - we'll take a closer look at these thresholds once the bugs are fixed. That said, the close queue is one of the most demoralizing queues to work because the number doesn't go down unless you're the final reviewer; it's not the queue I want to nag people about unless folks are simply ignoring it. See: meta.stackexchange.com/questions/238472/…
    – Shog9
    Commented Oct 18, 2017 at 17:02
  • 2
    I'm not saying that 10 is the right number, either... personally, I would rather see the red dot than not see it, though. I think that getting some data on how much people are reviewing compared to the older system would be useful... but it seems to me like you think I'm only focusing on the close queue here... and I'm not. The only queue I'm not recommending lowering in this answer is the suggested edits because it's already at 3. I'm concerned that your (plural) preferences don't align with what I have experienced or what I want... and I don't think my concerns are mine alone.
    – Catija
    Commented Oct 18, 2017 at 18:37
  • It's trivial to make the red dot appear more readily, @Catija. It's not trivial to convince folks who've gotten used to ignoring it because it [appeared too often | didn't lead to reviews they could perform] that they should consider it important. If I have to go back in a week and drastically lower all of these thresholds, that's not a big problem. For sites where I have to go back in a week and raise them, we've probably lost reviewers.
    – Shog9
    Commented Oct 18, 2017 at 19:20
  • 1
    I respect those concerns but I also don't think that overdoing it will permanently lose the reviewers (especially after just a week)... if they are raised and they become less common, then people will eventually start noticing the infrequency and likely start paying attention again... but this is all conjecture and I don't know whether you all want to turn it into a big social science experiment to see what the Goldilocks zone of bat signals for review is... and I'm guessing that it depends on the person... some people never do reviews, regardless.
    – Catija
    Commented Oct 18, 2017 at 19:44
  • Yeah, I don't particularly want to do that. Especially since on many sites there are only a tiny handful of active reviewers to begin with; I'd rather err on the side of getting more reviewers involved than feeding existing reviewers' addition to an indicator they've loved to hate. Also... as useful as the indicator can be, it's fundamentally an annoyance - a well-intentioned annoyance, but still annoying. As a matter of principle, we should try to keep it as unobtrusive as possible while still allowing the site to function.
    – Shog9
    Commented Oct 18, 2017 at 19:51
  • 3
    That seems like it would require a major change in mentality, though... I only look at reviews when there's an alert... Maybe that's crappy of me, but I simply don't... I'm guessing that a lot of other users are like that, too... Yes, I think annoyance is a concern but you still need the alerts to pop up before the reviews start to expire. Even if you only add the timer without lowering the queue triggers, that would be a big improvement.
    – Catija
    Commented Oct 18, 2017 at 19:57
  • That's a really easy workflow to encourage, so it doesn't surprise me that it's also the workflow we did encourage. But... It's a terrible workflow. It's the "lose half a day of writing because you got an email" workflow, the "you were gonna read that paper but then some game popped up a notification and you spent the hour playing it" workflow. It steals time and attention and eventually goodwill, and... for what? Are you more efficient doing 5 reviews scattered throughout the day, or 5 reviews back to back? Which is more fun? Unless something is on fire, why not pick your own workflow?
    – Shog9
    Commented Oct 18, 2017 at 20:04
  • 1
    Moderator flags are immediate and constant..there's not even a "give me an hour break from this big orange flag alert to actually add content to this site I'm supposed to be using in addition to moderating" so is this something that is going to change - making stuff spool (chat flags, inbox notifications, achievements)? If it's really that bad of a workflow, you're using it a lot and I'd sort of prefer it be more consistent rather than me having to remember to check one queue while having another constantly drawing my attention.
    – Catija
    Commented Oct 18, 2017 at 20:30
  • Yeah, I'd like to. It's the big reason why we don't do cross-site flag notifications. One of the ideas we've been discussing for a while is an "off duty" mode for moderators wherein you can temporarily be, well, not a moderator (in that you wouldn't see the UI, not that you would lose the status). For exactly this reason. There are also delays on some flags.
    – Shog9
    Commented Oct 18, 2017 at 20:36
  • That's a noble goal to change that workflow fundamentally. But together with giving people the power to do reviews at their own pace, are you still encouraging them to actually do them? I know such a fundamental workflow change needs to start somewhere, but I'd rather have people be reminded to do something (be that just occasionally) than not at all, especially when there's a ton of things to reivew despite the site-agnostic system thinking 20 close votes expiring tomorrow is a piece of cake. I don't want to read that important e-mail a week later either, when it's lost relevance.
    – Christian Rau
    Commented Oct 25, 2017 at 17:09
  • Replace the suggested threshold of "3" by "1" and simply deal with the two extra items to clear the queue.
    – Cœur
    Commented Oct 29, 2017 at 2:30
14

What I am missing so far is an indicator for when there are any items to review.

At the moment the only time the overall review indicator on the icon lights up is when there is at least one red dot in a queue, but as Catija already explained in her answer this is rarely ever the case on small sites. And even if none of the queues are in utmost danger, I'd still like to know when there's something to review (i.e. a queue has a grey dot). This was a very significant feature from the previous system that has gone lost and I can't really imagine people to review more now that there isn't any kind of alarm for it anymore.

  • The easiest and most straight-forward option would be to just light up a grey dot on the review icon when there are items in any of the queues (i.e. any queue has a grey dot).
  • Maybe that's considered too much and would be on most of the times, but this can be mitigated by a timer similar to the one already done for the red dot.
  • Alternatively the threshold could be a little higher than one queue being grey. But the point I'm trying to make is that this threshold should be significantly smaller than the current red-only one.
  • If grey is considered inappropriate, you could as well pick something else, like yellow, or maybe even a multi-layered system with yellow-orange-red (green might be confusing). But something as elaborate isn't even necessary, on the bottom line

just show us when there's anything to review.

Improve this answer
7
  • Maybe it would be better to go the other direction: Have the icon stay more-or-less as it is, except have it fade out when there's no (or very few) items to review; this would better indicate "there's really no reason to click this right now".
    – goldPseudo
    Commented Oct 18, 2017 at 15:18
  • 1
    Based on Shog's comments here, this seems like something that they're very hesitant to do because of how the review system works... part of the reason I went with the "solutions" I did is an attempt to find a middle ground. I do think that some notification for smaller numbers of tasks would be useful, particularly considering that even the sites I thought were "bigger" are apparently tiny.
    – Catija
    Commented Oct 18, 2017 at 16:02
  • 1
    First, keep in mind that there's a very serious bug affecting this right now (the only "small" site where you could ever see the indicator would be Movies & TV, regardless of trigger). Kinda wanted that fixed before this got posted, but there was too much confusion over the intended mechanics to wait. Beyond that... The previous system didn't work the way you think it did. Most people would only see a number if there were suggested edits - there was no indicator for non-mod-tools users for any other queue. And there was a threshold for even the handful of users who could see the full count.
    – Shog9
    Commented Oct 18, 2017 at 16:08
  • Independent of if the previous system worked or not, it is nevertheless useful to be reminded that there's something to be reviewed before the review queues basically explode to trigger the red icon. I am, however, not exactly aware of the bug you're speaking about, so am not sure how it affects my proposal.
    – Christian Rau
    Commented Oct 18, 2017 at 16:12
  • Catija's answer is focused on the close queue, which tends to have a backlog purely due to the number of reviews it takes to process a single task; on quite a few sites, your suggestion would result in the top icon always having a grey dot due to the presence of the close queue (as I type this, 136 sites have 1 or more items in that queue). If you aren't a moderator, it's entirely possible to review every task in the close queue without altering the number of pending tasks at all; you almost certainly don't want the close queue driving an indicator outside of exceptional circumstances.
    – Shog9
    Commented Oct 18, 2017 at 16:24
  • That's what the timer could be used for, or a threshold >1 (but still <<big). That being said, if there's always something to review, then that's important to know, since it means the community is backlogging.
    – Christian Rau
    Commented Oct 18, 2017 at 16:33
  • 1
    +1 to this; as a regular reviewer on one site, I want to know if there's anything in the queue at any time, regardless of whether I've clicked on the review button in the past hour.
    – Jeff Schaller
    Commented Dec 19, 2017 at 2:36
5

Isn't the color of the grey dots too light? Sometimes, I barely can see them.

Current color:

enter image description here

It would be better something a little bit less lighter:

enter image description here

Improve this answer
1
  • 2
    That would be confusing. Dark color means urgent, no matter if dark red or dark gray.
    – Shadow Wizard
    Commented Oct 25, 2017 at 22:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .